Measuring Developer Productivity has been the holy grail for managers ever since developers, managers, and the software field have existed. There are many "solutions" out there that use wrong or inconclusive metrics as a proxy for developer productivity.
Why is Productivity difficult to measure?
As much as we would like to have neat, well-defined ways of measuring something, this doesn't work quite as well for knowledge work. In software construction, for instance, the amount of hours spent doesn't necessarily translate into more output or better outcomes. In the software industry, we have a plethora of excellent tools for planning, development, deployment, and monitoring - and all of which provide good data. Sadly, too many companies focus solely on the data, equating that with productivity.
On the surface, almost everyone agrees that productivity ≠ no. of commits or lines of code written or no. of defects found or no. of hours worked. However, time and again, many teams fall back to these metrics because they are easily available. But we must remember that most of these are based on activity, not outcome.
Measuring outcome is often quite complex. Most software engineering teams are far removed from the business outcomes and it is difficult to trace the business impact to a certain piece of work (say, a commit or a feature).
There is one thing that has become increasingly clear over the last several years though: the faster a code change reaches the users, the faster the customers realize its incremental value. Doesn't matter how much work the engineering team is putting in, unless it is benefitting the users of the product it cannot be counted as productive output.
Metrics without context are dangerous
One thing that is consistently missing in most productivity measurement activities today is context. Without the right context, just numbers are meaningless and can often be misleading. It is tempting to think that more commits, more lines of code, or fewer defects mean better quality or productivity but we all know that is almost isn't quite true. The only time metrics like these should set off alarm bells is when their value is zero! Then you know for sure that things are not right somewhere. Every other value needs more context.
Let us take the example of one such metric and see how it can be used to evaluate the loss of productivity. Efficient engineering teams track the Lead Time to Changes (LTC) aka Commit-To-Deploy Time - the amount of time it takes for a certain code commit to reach the production (i.e., to the users). Most CI/CD tools provide this metric or it can be easily derived. However, for business leadership, a lot of its meaning doesn't get translated. What does it mean if the LTC has gone up by say, 20% in the last quarter? Should the business invest in improving this metric at all?
An interesting way to look at this would be to see the impact it has on developer productivity. A consistently increasing LTC could point to increased complexity or inefficient processes. It could point to the two most dangerous enemies of developer productivity - incessant Context Switching and spiraling Technical Debt.
Context Switching is not just a productivity killer, with studies showing that it could lead to increased stress and frustration. Now imagine that the average LTC in an organization is high (say > 3 days). If it takes a Pull Request (PR) more than 2 days to get a review comment, the author, who has most likely moved on to some other task, has to switch back to this. The author can, of course, get back to it after they finish the current task but that will only increase the lead time. In any case, it takes the author anywhere between 15-60 minutes for the author to switch the context (and an equal amount to switch back later). If the build fails, the author has to drop everything and jump back to identify and fix the problem. How long would that take? Apart from the context switching, if there is no investment in logging and observability, the developer will have to spend far more time debugging.
In any non-trivial software product, there is bound to be some Technical Debt. While not all types of Technical Debt are bad (we will leave that for another article), it does have a direct impact on developer productivity. Technical Debt slows down the developers as they have to think about a dozen different things while modifying a piece of code, it needs a deep review, it takes longer for new team members to be productive, and it could result in production issues.
Insights, not just metrics
To summarize, a steadily growing LTC is a lagging indicator of things that need to be fixed under the surface. This loss of productivity may be hard to measure accurately, but the trends are easier to spot with metrics used with the right amount of context. Focusing on just decreasing the LTC metric while disregarding the underlying problems may have pernicious effects including burnout, poor quality, and dissatisfied users.
At Praximax, we are considering all these complexities while building our product to highlight areas that engineering and business leaders can act upon. Relying on mere numbers that are easy to measure may not paint the full picture. Context is all-important.
Stay tuned...
Comments