Featured Article II

A Roadmap for Balanced Productivity Metrics: Are We There Yet?

The Balanced Approach

Philosophy

IT outsourcing continues to be a popular course of action for streamlining businesses.  These arrangements always include measurements of performance and productivity improvement.  The questions that we should be asking are:

·     “Why isn’t it just as important to measure performance and productivity related to in-house IT?”

·     “Why don’t we develop in-sourcing contracts, as an alternative to out-sourcing?”

Whether IT is inhouse or outsourced, performance and productivity are important. 

You know how the kids in the back seat are always asking: “Are we there yet?”  Most companies make outsourcing decisions without even knowing their current IT capability – and are often surprised when they find out that they were better off prior to outsourcing.  Similarly, many companies embark on software process improvement programs without knowing their existing level of performance and without any metrics in place to measure productivity improvement.  It’s like the old saying “If you don’t know where you are, a map won’t help.”  This applies to measuring progress as well. It is very difficult to know if we’re “there yet,” if we don’t know where we started and exactly where we still need to go.

Definition of BPM

Balanced Productivity Metrics (BPM) incorporates the use of both quantitative and qualitative data for measuring performance and productivity improvement.  The primary goals of a Software Engineering Institute (SEI) Capability Maturity Model (CMM) based process improvement program are to improve productivity, improve quality, and reduce risk via consistent processes.  BPM focuses on the SEI CMM core measures (size, time, effort, and defects), as well as other data collected to measure process improvement.  A multi-dimensional approach is needed to measure productivity improvement in a way that also provides an understanding of the environmental factors (and other significant factors) that influence project productivity.

The BPM approach is used to measure productivity with regard to software development and maintenance.  BPM is based on the principle that the management of productivity improvement should focus on achieving a balance of time (schedule), cost (effort), and quality (defect rate) [TCQ] ¹ improvement.  It is consistent with a balanced scorecard philosophy, since these fundamental metrics components cannot be observed independently with regard to providing a valid productivity assessment.  Balancing TCQ is much like filling a balloon.  If it is filled or compressed too much, it bursts.  If it doesn’t burst and it is compressed in one area, it expands in another.  Likewise, if a project’s schedule (time) is compressed too much, the project will either fail or the project effort (cost), project related defects and production defects (quality) will be disproportionably increased, which increases maintenance effort (cost).  The reverse is also true.  Project schedules can be increased too much, which does reduce effort and defects, but may not provide the desired business value.  Therefore, a balanced measurement approach is needed.  

To quote Karl Wiegers:

A risk with any metrics activity is dysfunctional measurement, in which participants alter their behavior to optimize something that is being measured, rather than focusing on the real organizational goal. For example, if we are measuring output productivity but not quality, expect some developers to change their coding style to expand the volume of material they produce, or to code quickly without regard for bugs. I can write code very fast if it doesn’t actually have to run correctly. The balanced set of measurements helps prevent dysfunctional behavior by monitoring the group’s performance in several complementary aspects of their work that lead to project success.”²

The two primary BPM reporting components for providing a balanced set of metrics, are the:

·     Productivity Metrics Report (PMR), and the

·     Process Performance Report (PPR).

PMR Quantitative Measures

The quantitative measures associated with the Productivity Metrics Report (PMR) are used to assess productivity improvement.  The initial assessment period is considered the baseline year.  Improvement is normally expected at the end of the second year over the baseline year.  Every year thereafter, improvement is expected over the previous year. 

The PMR uses a combination of metrics for calculating the BPM Score, which measures success in achieving productivity improvement goals.  The project metrics are calculated for each software enhancement and new development project, and then averaged for each productivity metrics component.  The maintenance metric is calculated with regard to the yearly hours required to provide ongoing support for the total inventory of supported applications.  The metrics components to be included in calculating the productivity score are shown in Table 1.

Table 1 – BPM Components and Calculations

PPR Qualitative Measures

The qualitative measures associated with the Process Performance Report (PPR), when analyzed along with the quantitative measures, provide insight into the overall process productivity and direction toward achieving process improvement goals.  Looking at the BPM Score – which is a letter grade or number – only provides the current status or contractual measure.  Unless there is an underlying analysis behind that score, there is little information to use for identifying the next steps.  The PPR provides information that can be used to identify problems and areas that have improved, for managing process productivity improvement activities.  This report provides the following analyses, trending, and comparisons to previous period:

·      Demographic data analysis related to organization profiles, project types, size, complexity, and languages

·      Statistical analysis of metrics categories against previous performance trends

·      Analysis of delivery performance, including requirements volatility

·      Significant factors impacting project results

·      Analysis of customer satisfaction survey results in relation to project data

·      Analysis of environmental factors related to tools/methodology, technical difficulty, and personnel

Process Overview

Administration

The BPM process involves joint responsibilities.  The client performance metrics manager (client PMM) manages the process and the database and serves as team lead for a productivity metrics team.  The client PMM will prepare the PPR and the IT metrics group will prepare the PMR, which includes calculating the BPM Score.  Change management with regard to this process will be the responsibility of the productivity metrics team.  (These roles may vary depending on organizational structure.  Additional roles are identified in Table 2.)

Frequency

The PMR should be calculated annually for productivity scoring related to annual software process improvement goals, or contractual agreements in the case of outsourcing.  For purposes of monitoring progress and enabling the parties to promptly address any service or process deficiencies, quarterly BPM Reports containing cumulative data should be created and reviewed.

If, at any time the BPM Score is to be calculated, and one or more metrics or component metrics cannot be calculated, either because the relevant data is not available, or the applicable scoring scale has not been adopted or is not yet in effect, then the BPM Score should be calculated disregarding such metrics.  The weights of the other metrics or component metrics within a given metric shall be increased proportionately such that other metrics or component metrics maintain their respective relative weights.

Data Collection and Dictionary

Data consistency is the key to having a good decision support database; therefore, to insure consistency, the definitions must be documented.  Also, when building an internal historical project repository, the project data points must be categorized.  This will provide stratification such that apples-to-apples project comparisons can be made based upon project type, environment, user organization, development organization, language, etc. The data collection requirements should also be defined and documented.  The collection of software project data during the project is designed to track the status of projects, enable predictive analysis, and facilitate the collection of project closeout data required for the BPM reports.  Upon completion of the project, the data should be summarized in a detailed data collection form.

Tools/Database

BPM project data should be stored in a metrics database.  The project-level information in this database can be used for planning new projects and evaluating risks.  At a minimum, this can be done by using data from similar historical projects for providing an analogy to the project being planned.  This is associated with the CMM Level 2 Software Project Planning key process area and supports the software project estimation and risk management processes. (For more information on this, see my December 2000 ITMS article, “Saving the World One Project at a Time: Planning by the Numbers.”)⁸

Process Steps

The BPM process-flow schematic is illustrated in Figure 1.  The BPM schematic description and responsibilities are provided in Table 2.  The BPM process begins with collecting baseline project data and the development of a productivity baseline.  The schematic illustrates the next steps after the baseline has been established. 

Figure 1 – BPM Process Flow Schematic

 Table 2 – BPM Process Schematic Description

Critical Success Factors

The following factors are critical to the success of a Balanced Productivity Metrics Program:

Scoring

Metrics Component Scores

Each BPM Metric Component will be given a letter score determined in accordance with the scoring scale shown in Table 3, assuming a 5% yearly improvement for each component is desired.  This letter grade will be converted to a numerical score using the scale also shown in Table 3.  (The percentages and letter grades can be adjusted based upon the yearly improvement desired for each TCQ category.)           

Table 3 (T3) – Metrics Category Component Grading

Metrics Component Weighting and Balanced Scoring

Business value should determine the weightings with regard to Project TCQ and Maintenance, since this is a business decision as to which component has the highest priority, or if they are all equally important.  Of course this will vary from project to project, depending on the circumstances, therefore the BPM weightings should reflect the overall philosophy for the organization.  Component/Category weightings should be established once per year, at the beginning of each annual measurement cycle.  If everything were equally weighted, the total BPM composite numerical score would be calculated as shown it Table 4, and the final composite BPM grade would be calculated as shown in Table 5.  Examples of BPM Score Calculations are provided in Tables 6, 7, and 8.

 Table 4 (T4) – Numerical Component Weighting and Composite Calculation

 Table 5 (T5) – Final BPM Composite Grading

 Table 6 – Example 1: BPM Score Calculation

 Table 7 – Example 2: BPM Score Calculation

Table 8 – Example 3: BPM Score Calculation

Conclusion

The measurement of productivity and performance is just as important for inhouse IT as it is for an outsourced IT environment.  A balanced approach, as provided by the BPM process, is needed in order to ensure that all aspects of productivity are assessed and that the correct behavior with regard to performance is achieved.  Some outsourcing vendors will argue that only one measure should be used, such as effort per unit of output, but to most clients, schedule and quality are equally important.  Therefore, measures should be in place such that schedule or quality cannot be sacrificed, and to prevent dysfunctional measurement behavior from occurring.  It is beneficial to IT that all factors be measured, due to the clients’ ever changing TCQ priorities.  In addition to balancing the TCQ metrics and quantitatively tracking projects, it is also important to understand the relationship between the quantitative and qualitative measures that influence productivity.  When we trend the quantitative and qualitative data and compare the results to previous internal benchmarks, our corporate knowledge will be increased such that we can make intelligent business decisions. The BPM process can help us to understand the factors that influence our productivity, know where we are on the map, effectively predict where we are headed, and recognize when we get there.

References

1.   Null, Christopher. (2001, November). “Mess with The Bull, Get the Horns.” Smart Business.

2.   Mayes, Jim. (2001, August). “A Road Map for Balanced Productivity Metrics: Are We There Yet?” IT Metrics Strategies. Cutter Consortium.

3.   Paulk, Weber, Garcia, Chrissis, Bush. (1993). Key Practices of the Capability Maturity Model, Version 1.1. CMU/SEI-93-TR-25, ESC-TR-93-178.

4.   Wiegers, Karl. (1999, July). “A Software Metrics Primer.” Software Development

5.   Mayes, Jim. (2000, March). “Achieving Business Objectives: Balancing Time, Cost, and Quality.” IT Metrics Strategies. Cutter Consortium.

6.   Lombardi, Vince, former NFL coach. Green Bay Packers.

7.   Wheeler, Donald, (PhD.) Alabama School of Mathematics and Sciences

8.   Mayes, Jim. (2001, August). “Saving the World, One Project at a Time: Planning by the Numbers.” IT Metrics Strategies. Cutter Consortium.

9.   Baumert and McKinney. (1992) Software Measures and the Capability Maturity Model. CMU/SEI-92-TR-25, Technical Report. 

10.  Florac, Park, and Carleton. (1997) Practical Software Measurement: Measuring for Process Management and Improvement. CMU/SEI-97-HB-003.

Mayes Consulting, LLC

Quantitative Software Engineering  

Copyright (c) 2001 Mayes Consulting

1605 Kinsmon Lane
Marietta, GA  30062
770-649-8599 or 404-754-2707
jimmayes@bellsouth.net