Measuring software delivery performance is crucial, especially when trying to improve various DevOps tenets like improving business culture, which subsequently influences delivering exceptional products and services. Take for instance Agile software development practices, which advocate adaptive planning, evolutionary development, early delivery, and continual improvement, which is necessary because of how the scope of the project can change during development.
To understand the important nature of measuring software delivery performance, we first need to define valid and reliable means of measurement.
Embracing Organizational Outcomes
Embracing DevOps requires organizations to focus on universal outcomes, which drives organizations, leaders, teams and developers to work toward a common goal. Moreover, measuring outcomes, not output, helps direct focus onto the practices and resources that helped achieve the common goal.
Productivity as a Measurement
Many leaders today measure their DevOps team’s productivity (e.g., completed stories; bugs, etc.) using a point system, which is documented by stakeholders at the end of a sprint cycle. This form of measurement is a capacity planning tool, which will log how long it will take the DevOps team to complete the work planned.
Consider the following: DevOps teams may embellish their numbers by completing unplanned work. While completing many stories sounds great, if the work is not contributing to the organizational goal it may discourage DevOps cross-team building and collaboration. It may even impact the reliability of the measure due to over-inflated estimates.
It is essential that we manage productivity by minimizing high levels of utilization, which hinders performance due to overutilization, which increases lead times. Set sprint capacity carefully; capacity planning should consider the variation in work hours by team members as well as any PTO. Azure Boards provides comprehensive tooling to set DevOps teams capacity for sprints, track capacity when performing multiple activities, add or remove DevOps team members for any given sprint and track multiple teams.
Measuring continuous integration and delivery…
Delivery lead time is measured from the time a request is made to its completion. The main factors included in this measure are designing the software and continuously integrating/delivering features (New software/Software updates). Subject matter experts measure continuous delivery lead times from code committed in Microsoft Azure repos service-to-software running in development or production environments. The goal is to minimize delivery lead times. Why? Because it accelerates user feedback. There are many advantages to this approach, like detecting defects early to mitigate the effect. This is typically addressed via a hotfix.
Deployment frequency is our second measure. It has many similarities to cycle size, which refers to our release pipeline commit sizes. An AzureDevOps release may include multiple source control commits. Anecdotal evidence from a variety of industry displays frequent multiple deployments per hour throughout the day. However, this delivery frequency method may vary due to business practices. But, by decreasing our cycle sizes by streamlining each commit to release we can release each commit into production, drastically reduce risk and overhead costs.
Time to restore is our third measure, where we gauge how rapidly we can restore service. Failure is Inevitable, there are a multitude of incidents that can arise, such as, outages, misconfigured services, etc.
Lastly, we measure the change fail rate, which is the percentage of software updates that result in impairment or service outage, which will require a software patch or rollback.
