So you did everything right. You defined your product and acceptance criteria well. You focused on the task at hand. You followed good processes for design and development. Now you have hit code complete. How do you get from here to a released product? How do you manage your resources and schedule?
Everyone has their own names for various phases. So that you understand what I am referring to, let's start with some definitions (the following definitions explain what you should have at the end of each phase):
- Development: design, design reviews, coding, code reviews, test code development and test code reviews are complete. The features basically work and the tests have been run but bugs may exist. You have met the Alpha entry acceptance criteria.
- Alpha: All criteria for General availability are met. Ready for field test. Beta documentation is ready.
- Beta: All criteria are met and the product has been field tested.
At the end of Beta, the product is ready for General Availability. Some companies have a Limited Availability phase before General Availability to augment their field test. Many customers will not field test Beta. A Limited Availability Release provides a production product with expectations that the product needs more field exposure.
The aforementioned phases occur regardless of what you plan or what you call them. Some people release final product to the field that I would call Alpha. The product still must go through the rest of the phases to achieve the stability expect in a released product.
Do not enter one phase until you meet the acceptance criteria for the previous phase. If you do, you will just be fooling yourself and slip later on. It is near impossible to make up time during these integration phases.
How do you size the time required for Alpha and Beta? A number of factors should be considered including:
- The number of features added to the product
- The amount of foundational change in a release
- The length of a QA cycle
- Efficiency of the team at addressing issues
- How complete the product is from the previous phase
This paragraph provides some guidelines for scheduling Alpha and Beta. If you have a product that can be robustly tested within a 7 day period and you introduce 10-30 moderate-sized features, then the Alpha time would be from 4-6 weeks and the Beta time would be from 6-8 weeks. For a larger product of 30-100 features, an Alpha cycle of 12 weeks and a Beta cycle of 24 weeks is more appropriate. Clearly the above factors will modify those times.
From here it is a matter of managing the test cycles, field test cycles and the bug list.
You must make sure that your test team has viable test candidates. Try to not sequentialize test (i.e. don't delay doing one test because another did not work). Make sure the test information is clear and concise. Report progress to the whole team. You should consider any bug stopping the test team from doing their testing, as the highest priority bug to fix.
Someone must drive field test. Start it as early as possible (even during Alpha). Make sure you are using your own products where possible. There is no substitute for real world testing. If you cannot accomplish this, you may have to do a limited availability release (see above). Make sure there is enough time to get feedback from the field test and incorporate changes. Send someone to customer sites to get them up and running quickly. Define what you need them to accomplish. Select your Field test sites well. Avoid using pre-sales customers as field test sites.
The final thing is the bug list. You need to agree upfront about what your criteria is. Do not wait until the end to negotiate this. Do not use Alpha or Beta time to fix bugs from the previous releases as they should have been scheduled during the Development phase. All releases have bugs. When you classify bugs, the only piece of information that is important is when the bug needs to be fixed by:
- Development complete
- Alpha Complete
- Beta Complete
- First Patch or Bugfix Release
Your product team including product management, engineering and support should help determine when or if a bug must be fixed.
You need to track the trend for your bugs. The three critical pieces of information are:
- How many bugs can your team fix per day
- How many total bugs are in the system
- How many new bugs your team is finding each day
With this information, you can determine whether you will make the schedule. You can also set goals about where you need to be during each week of Alpha or Beta. The product team can determine the cost/benefit trade-off for fixing each bug. If you cannot make the schedule, then you should ask your team to work nights and weekends to help try and get back on schedule. This is typical and should be expected for the integration phases of a release. If the trend continues, you will need to adjust the schedule to accommodate improving quality and you should examine your development practices so that you can do better on the next release. If you are experiencing a lot of bugs or as you get close to a milestone, conduct daily meetings with your direct reports or full staff as needed.
Once your release is stable enough to reach Beta, you need restrict the type of bugs you allow into a release. You must assume that any bugfix is as likely to introduce a new problem as it is to fix an existing one. You must also provide more restrictions and scrutiny on changes you allow your developers to check-in. More rigorous code reviews and management sign-off is common.
Finally when you reach a point when you have your golden bits that you wish to release, you must have rules about what to do between the time you ok that release and when it gets into customers hands. You will always find one more thing you can fix. So create a rule that says any bug found after the release is signed-off must go into the first patch or bugfix release (unless you would have stopped shipment for the bug in an already released product).
It is easy to slip releases in Alpha and Beta phases. There are always exceptions to the above process but you should remain focused, watch things closely and adapt.
More later ...
