How should you measure engineering?

One of the common questions I get asked is how should you measure engineering. It is a loaded question. Usually there are personal biases and long painful experience that color the question. However, the sentiment of the question is correct. If we do not measure ourselves we cannot get better.

The common industry term for these measure are Key Performance Indicators (KPIs). As a general guideline, I like these to be simple and easy to measure.

The following three questions are important to the business, can give you an idea about how we are doing and have answers that are reasonably easy to measure:

• How productive is engineering?
• How good is the quality?
• How quickly can we respond to customer problems?

The first two are solely engineering measures and the third is a joint measure with support.

You can easily measure engineering productivity by measuring the number of features that engineering releases to customers per quarter. In order to create this measure, you need to have a common denomination or coin of the realm to normalize the size of projects. You can classify your projects into small, medium, large and extra large where you can size those project sizes and provide some basic equivalences. For example:

• Small - less than 1 development week effort == 1/3 medium project
• Medium - less than 1 month of development effort
• Large - less than 3 months of development effort == 3 medium projects
• Extra Large - greater than 3 months of development == 9 medium projects

You could clearly make the measurements exact but this method also provides a normalization that both reflects the impact of larger projects by virtue of rounding up and removes anomalies by capping the translation. This measure will have a direct effect on the company's ability to compete and to manage costs.

You can measure quality by how many bugs need to be fixed in patches per quarter and in the next release per quarter. The idea here is that it does not matter what the company marks as the priority or severity of a bug. Instead if the company requires the bug get fixed in a patch or release, then for whatever reason engineering had to expend resources to fix the bug. Quality then becomes about addressing issues earlier and removing the need for the company to spend resources after software has been released. This measure will have a direct impact on customer satisfaction and engineering productivity.

Finally, we all know that there will be critical problems that we cannot foresee that the company must address. It is important that the company reduce the time for customer relief for critical bugs and measure it at least quarterly. The fix could be a work around developed solely by customer support or a complex engineering based fix and therefore this is a joint measure. Efforts like customer training, robust error-logging, and code assertions can help reduce time-to-relief. This measure will have a direct impact on customer satisfaction.

If you don't measure you cannot get better.

More later ...