Showing posts with label releases. Show all posts
Showing posts with label releases. Show all posts

4/27/2007

Planning

People ask me about Agile and other project planning methods in high tech today. What I tell them is that all tools can be used well or poorly. It takes leadership, forethought, common sense and perseverance to plan and follow through on a project.

Many managers do incomplete jobs at planning, or they do not manage the plan once it is developed, or they do not handle change appropriately. Each of these mistakes will affect predictability. I have seen some organizations that embrace some of these techniques as a facade for "it will get done when it gets done". The problem is that your CFO, VP of Sales and customers cannot plan appropriately in that kind of environment. Your company needs predictability in order to succeed.

I am a lot less interested in the form you use to plan than the functionality. All of these techniques can be applied using almost any method. Here are some basics:

  • Do not commit to a date until you have preliminary specifications and detailed schedules
  • Make sure every stakeholder physically signs off at each phase
  • Adopt a train model
  • Don't make top-down dates
  • Complex projects require very detailed planning that will change
  • Manage each slip with urgency and importance
  • Deal with major changes as if you were starting anew

I have talked many times about the proposal and definition phases of a project. You cannot have predictability without specifications. Those specifications must be good enough to ensure you hare heading in the correct architectural and product direction. They must also be good enough to create schedules. The schedules must prove that you can make the target date. You must work with the information you have at the time and understand that it will change. The schedules should be manageable (< 2 week tasks, measurable deliverables, etc.).

Don't assume that verbal or incidental communication is enough to get buy-in from your company. Get a signoff sheet for each phase in writing. Make sure development, QA, docs, architects, sales(customers), support, and the CEO are all in the loop. It is awful to find out after you have spent money on developing something that you have done the wrong thing.

Feature driven releases often slip. If a major feature slips and the release slips, then there is pressure to add more features since it will be a while until the next release. Adding late features can cause more integration work and potentially more slip or worse yet cause quality problems. Separate features out from releases. Make releases time based with a shorter cycle (3 months). With a time based release, the features that are ready can go out. Big features can be developed independently. You can respond to customer needs more quickly. The train model requires adequate automated testing to be successful in order to reduce the overhead of more frequent releases.

Top down dates don't work. Schedules must be bottoms up. Managers must help there teams make aggressive but accomplishable schedules. Managers must help make trade-offs around function, quality and resources and have substantive business discussion to achieve the right balance. Don't commit until everyone on your team can look you in the eye and commit. This is a pay-me-now or pay-me-later issue. It is more than a burn-out issue. It is again about predictability.

Complex projects need detailed schedules. Even if they go out two years. Schedule re-scheduling events for long projects. If you use Agile, you can translate detailed schedules into sprints but you still need the homework upfront. You need the exercises to get predictability. The schedule is really a step-by-step design. It makes your developers think. It allows your team to identify many dependencies and resource issues upfront. It will change -- get over it. There is no free lunch. You cannot have predictability with coarse grain tasks. You cannot have predictability without having your best guess at the details upfront.

You need to manage the plan. Most slips start from day one of the schedule. Require your team to have at most two tasks open per person at one time as they accumulate risk (this may require scaffolding). Require that each slip have a mitigation plan on a weekly basis. Do not shove all delays into a bucket to be dealt with later. Your team needs to adjust quickly to issues.

Finally, if you have major changes including features additions or architectural surprises, you need to go back into a planning phase. Never assume you can just add it on. Do the resource planning, specifications and schedules. Do new signoffs. Anything less will affect predictability and cause slips.

We all face these issues. True leadership in planning is the foundation for success for your team.

More later ...

3/29/2007

What are good goals for testing?

Sadly, software testing has not become a prestigious endeavor in our industry. In ASIC design, for example, the test verification folks often are paid the most and lead the teams. In software, test development and test execution are often looked at as stepping stones to product development. As a result, test development has developed a stigma and is often done as an afterthought and often not pursued by the best engineers.

Without reasonable tests and test infrastructure, your company will spend a lot of time and effort doing manual testing at each patch or release. You may not be able to respond to customer requirements in a competitive fashion. The worst result may be inflicting poor software on your customers and having them find your bugs while you lose your reputation.

So how can you set good goals and get better results in your testing effort? Here are the basics:

  • Make sure everyone feels responsible for quality
  • Define detailed acceptance criteria that drives what you test
  • Invest in infrastructure for testing
  • Drive toward specific goals around automation and responsiveness

I have talked about many of these issues both in the book and in other blogs but I will summarize them here as a reminder.

Everyone needs to feel responsible for quality. From the developer who should produce automated exhaustive functional unit tests to the project or release owner who should drive quality across many functional areas, everyone should feel responsible. Break down the organizational walls. Treat quality like performance or reliability and bring to bear the resources you need to make it happen. Don't differentiate between a development specification and a test specification -- make them one document. Send a message that there is only one thing and that is a quality product for your customers. Send a message that there is only one team and everyone must work together to achieve quality.

Don't let acceptance of a project or product only be that it contains no critical bugs. That measure is a second order measure. Define the parametrics for the product around performance, durability, interoperability, compatibility, usability, etc.. Decide on those things together with all of your developers and product management. Use those criteria along with exhaustive functional unit tests to drive the test portion of your specifications. Make sure you identify real customer scenarios as a basis for the tests. Finally, measure, measure, measure the results in both your schedule as the tests are completed and in your integration as more of the tests are deployed. Do not hide these details behind test points. Expose them in highly visible presentations. Make sure everyone can see what is important to the team with respect to determining progress and whether the project or product is done.

I hate the word framework. It is used to denote a lot of things and does not communicate what work is actually being done. Testing frameworks fall into this grouping for me. So what infrastructure needs to be in place for people to actually develop automated tests? Here is a list of the basics:

  • Tools to setup and knockdown a test. These may include tools that can reboot or reinitialize test platforms and/or test drivers automatically. They may include setting up a configuration or data set. Clearly how fast they work is a factor.
  • A programmatic interface to the test platform. This may be through a CLI, scripting language or API. It must work remotely. It must control as much of the product as possible in order to test as much as possible. Avoid doing this through the GUI as test vehicles that go through the GUI are often hard to maintain.
  • A standard output format. Define what you need for summary and detailed information. If every test produces different output formats or worse undecipherable output formats, you will spend an enormous effort in trying to figure out what is working and what is not.

Obviously there are always more items that you may need for a particular product or environment but your list should always include the items listed above.

The last thing you should do is set goals around automation and responsiveness. How much effort should it take to test a patch or a release? How much elapsed time should it take to test a patch or a release? By its nature, the last question will affect the answer to the first question (i.e. with a limited amounted of time you will need to automate more and it will reduce the effort). So here are my rules of thumb for elapsed time:

  • For a patch you need to be able to broadly but not deeply test your whole product within 12 hours.
  • For a release you need to be able to test your whole product broadly and deeply within 7 days (excluding duration tests for obvious reasons).

The patch level test is often also used on nightly builds during a development cycle.

Obviously your goals should be to reduce the effort for both kinds of test cycles as low as possible. However, there will always be a manual component to verify that GUIs look right.

If you follow the guidelines above you can set better goals around testing and achieve better results for your team and your company.

More later ...

1/15/2007

How late should you allow changes in a release?

This entry is about the very end of a release. I have talked about the whole end-game in a previous posting but this period warrants more discussion. There are always more bugs to fix. Sometimes this pressure comes from engineering trying to perfect a release and sometimes from product management trying to help out one more customers or avoid a service call. Either way I suggest you avoid a lot of changes down the end of a release.

There should always be a quiet period before a release where only bugs that would stop shipment and cause an immediate patch of a production release should get fixed. Clearly the length of this period will vary with the size of release or the maturity of a company. But the period should not be less time than a full run of testing (which should be set to take 7 or less days).

In this period, any bugfix is as likely to cause more problems as fix problems. People like to minimize the impact of changes in this period but my experience does not support that position. I have seen slips due to innocuous changes.

If you have not met your bug goals or you have not completed testing do not enter this period because you will need to fix bugs that are not appropriate for this period. Do not assume you can fix a lot of bugs and keep the release stable.

So what can you do to manage this period:

  • Show leadership
  • Plan a patch release shortly after your release as a vehicle to handle critical non-showstopper bugs.
  • Get involved in the bug reviews yourself
  • Do not start new testing that has not been performed before in the cycle

Managing this period well will improve your company's and group's predictability which will in turn enhance your chances for success.

More later ...

3/01/2006

How do you handle the end-game of a release?

So you did everything right. You defined your product and acceptance criteria well. You focused on the task at hand. You followed good processes for design and development. Now you have hit code complete. How do you get from here to a released product? How do you manage your resources and schedule?

Everyone has their own names for various phases. So that you understand what I am referring to, let's start with some definitions (the following definitions explain what you should have at the end of each phase):

  • Development: design, design reviews, coding, code reviews, test code development and test code reviews are complete. The features basically work and the tests have been run but bugs may exist. You have met the Alpha entry acceptance criteria.
  • Alpha: All criteria for General availability are met. Ready for field test. Beta documentation is ready.
  • Beta: All criteria are met and the product has been field tested.

At the end of Beta, the product is ready for General Availability. Some companies have a Limited Availability phase before General Availability to augment their field test. Many customers will not field test Beta. A Limited Availability Release provides a production product with expectations that the product needs more field exposure.

The aforementioned phases occur regardless of what you plan or what you call them. Some people release final product to the field that I would call Alpha. The product still must go through the rest of the phases to achieve the stability expect in a released product.

Do not enter one phase until you meet the acceptance criteria for the previous phase. If you do, you will just be fooling yourself and slip later on. It is near impossible to make up time during these integration phases.

How do you size the time required for Alpha and Beta? A number of factors should be considered including:

  • The number of features added to the product
  • The amount of foundational change in a release
  • The length of a QA cycle
  • Efficiency of the team at addressing issues
  • How complete the product is from the previous phase

This paragraph provides some guidelines for scheduling Alpha and Beta. If you have a product that can be robustly tested within a 7 day period and you introduce 10-30 moderate-sized features, then the Alpha time would be from 4-6 weeks and the Beta time would be from 6-8 weeks. For a larger product of 30-100 features, an Alpha cycle of 12 weeks and a Beta cycle of 24 weeks is more appropriate. Clearly the above factors will modify those times.

From here it is a matter of managing the test cycles, field test cycles and the bug list.

You must make sure that your test team has viable test candidates. Try to not sequentialize test (i.e. don't delay doing one test because another did not work). Make sure the test information is clear and concise. Report progress to the whole team. You should consider any bug stopping the test team from doing their testing, as the highest priority bug to fix.

Someone must drive field test. Start it as early as possible (even during Alpha). Make sure you are using your own products where possible. There is no substitute for real world testing. If you cannot accomplish this, you may have to do a limited availability release (see above). Make sure there is enough time to get feedback from the field test and incorporate changes. Send someone to customer sites to get them up and running quickly. Define what you need them to accomplish. Select your Field test sites well. Avoid using pre-sales customers as field test sites.

The final thing is the bug list. You need to agree upfront about what your criteria is. Do not wait until the end to negotiate this. Do not use Alpha or Beta time to fix bugs from the previous releases as they should have been scheduled during the Development phase. All releases have bugs. When you classify bugs, the only piece of information that is important is when the bug needs to be fixed by:

  • Development complete
  • Alpha Complete
  • Beta Complete
  • First Patch or Bugfix Release

Your product team including product management, engineering and support should help determine when or if a bug must be fixed.

You need to track the trend for your bugs. The three critical pieces of information are:

  • How many bugs can your team fix per day
  • How many total bugs are in the system
  • How many new bugs your team is finding each day

With this information, you can determine whether you will make the schedule. You can also set goals about where you need to be during each week of Alpha or Beta. The product team can determine the cost/benefit trade-off for fixing each bug. If you cannot make the schedule, then you should ask your team to work nights and weekends to help try and get back on schedule. This is typical and should be expected for the integration phases of a release. If the trend continues, you will need to adjust the schedule to accommodate improving quality and you should examine your development practices so that you can do better on the next release. If you are experiencing a lot of bugs or as you get close to a milestone, conduct daily meetings with your direct reports or full staff as needed.

Once your release is stable enough to reach Beta, you need restrict the type of bugs you allow into a release. You must assume that any bugfix is as likely to introduce a new problem as it is to fix an existing one. You must also provide more restrictions and scrutiny on changes you allow your developers to check-in. More rigorous code reviews and management sign-off is common.

Finally when you reach a point when you have your golden bits that you wish to release, you must have rules about what to do between the time you ok that release and when it gets into customers hands. You will always find one more thing you can fix. So create a rule that says any bug found after the release is signed-off must go into the first patch or bugfix release (unless you would have stopped shipment for the bug in an already released product).

It is easy to slip releases in Alpha and Beta phases. There are always exceptions to the above process but you should remain focused, watch things closely and adapt.

More later ...