“What gets measured, gets managed” – Peter Drucker
Most of the places I have worked, spend a lot of time measuring multiple dimensions of software for the entire lifecycle of the product. Many try to see what is the “best practice” in the industry and put a wrapper on them to make their own metrics. I have been thinking for a while on what makes sense, and is there a silver bullet like the Grand Unified Theory.
While at it, it looks to me that we are trying to contain ourselves in one direction in as chaotic as a place of Software Product Development. Organizations tend to copy “what worked for them” without even caring for the constraints and conditions that the specific metric worked for “them”.
Then, what to measure? In my view organizations which have understood what and how much to measure a product's usefulness, have survived and thrived. When they have multiple products, they have different lenses for each of them as well. Some products will need a lot of statistics and data, while some may need only a few!
And, how do you arrive at what to measure? For me the closest guide is the 7th principle from Agile Manifesto “Working software is the primary measure of progress”. If you can define what is working software for you, it becomes easier to measure on the progress.
Again, each one in their role, sees the “working software” in a different way. Hence it becomes very important to have a holistic view of who is the customer, who is the builder, who is the sponsor and who manages the process. Let us try to think from each perspective.
Note: These thoughts for building a product or service should only serve as a starting point and none of them are given in any order of preference. “The best architectures, requirements, and designs emerge from self-organizing teams” and so do the metrics and measurements.
Business/Product Owner/Product Manager
This role is primarily responsible for understanding the market and synthesize the data. They have “Hypothesis” around what makes the organization profitable. When all they are working with is a hypothesis and not much of clarity, they will be interested in forward looking metrics. The following could be the measures they need:
1. Cost of Delay: Cost of Delay is "a way of communicating the impact of time on the outcomes we hope to achieve." More formally, it is the total expected value with respect to time. Cost of Delay combines an understanding of value with how that value leaks away over time.
It is the answer to the question: "What would it cost us if this was delayed by 1 month?" Or, alternatively, "what would it be worth to us if we could get this 1 month earlier?"
More on Cost of Delay, here
2. Product Market Fit : Product/market fit is the degree to which a product satisfies a strong market demand. Product/market fit is the first step to building a successful venture in which the company meets early adopters, gathers feedback, and gauges interest in its product(s).
More on Product Market Fit, here
3. Riskiest Assumption Test (RAT): A Riskiest Assumption Test is explicit. There is no need to build more than what’s required to test your largest unknown. No expectation of perfect code or design. No danger it will prematurely become a product
More on RAT, here
4. Minimum Viable Product (MVP) : The “version" of a new product that allows a team to collect the maximum amount of validated learning about customers with the least effort. It is a product with just enough features to satisfy early customers, and to provide feedback for future product development.
More on MVP, here
5. Minimum Marketable Product (MMP) : MMP is based on the idea that less is more: It describes the product with the smallest possible feature set that addresses the needs of the initial users (innovators and early adopters), and can hence be marketed and/or sold. The MMP is a tool to reduce time-to-market: It can be launched more quickly than a fat, feature-rich one.
More on MMP, here
6. Cycle Time : Cycle time is the total time from the beginning to the end of your process, as defined by you and your customer. Cycle time includes process time, during which a unit is acted upon to bring it closer to an output, and delay time, during which a unit of work is spent waiting to take the next action.
More on Cycle Time, here
The “Product” group should have a mechanism to understand and keep an eye on the metrics that are important to Sponsors/Investors as well, this will make sure the loop is closed and any suggestion to “Pivot” is indicated by them.
Sponsor/Investor
Interestingly, this group has interest in the same metrics that are used by Business/Product Managers before the product is launched. I am inclined to use thoughts from “lean start-up” as they are more “actionable” and not “vanity metrics”. During and after the product launch, the following are the measurement/information they will be most interested in:
1. Employee Satisfaction: The happier and motivated employees make sure they keep the customer happy and engaged. Employee satisfaction is the terminology used to describe whether employees are happy and contented and fulfilling their desires and needs at work. The employee satisfaction is a factor in employee motivation, employee goal achievement, and positive employee morale in the workplace.
2. Viral coefficient : A viral coefficient is a number which tells you how many customers each your present customers are bringing to you on an average. So this means if your viral coefficient is 2, then each of your current customers is bringing in 2 more customers to your business. This metric calculates the exponential referral cycle - sometimes called virality - that accelerates company growth. Virality is the inherent incentive for customers to refer friends or colleagues to your company.
More on Viral Coefficient, here
3. Sunk Costs : In economics and business decision-making, a sunk cost is a cost that has already been incurred and cannot be recovered. Sunk costs (also known as retrospective costs) are sometimes contrasted with prospective costs, which are future costs that may be incurred or changed if an action is taken.
More on Sunk Costs, here
4. Viral Cycle Time : Viral cycle time is the time it takes for one such cycle to complete. In other words, viral cycle time is the amount of time it takes for a user to invite another user.
More on Viral Cycle Time, here
5. Net Promoter Score : Net Promoter or Net Promoter Score (NPS) is a management tool that can be used to gauge the loyalty of a firm's customer relationships. It serves as an alternative to traditional customer satisfaction research and claims to be correlated with revenue growth.
The Net Promoter Score is calculated based on responses to a single question: How likely is it that you would recommend our company/product/service to a friend or colleague? The scoring for this answer is most often based on a 0 to 10 scale.
Those who have responded with a score of 9 to 10 are called Promoters. Those who have responded with a score of 0 to 6 are labelled Detractors. Responses of 7 and 8 are labelled Passives, and their behavior falls in the middle of Promoters and Detractors. The Net Promoter Score is calculated by subtracting the percentage of customers who are Detractors from the percentage of customers who are Promoters. For purposes of calculating a Net Promoter Score, Passives count towards the total number of respondents, thus decreasing the percentage of detractors and promoters and pushing the net score towards 0.
More on NPS, here
6. Customer Happiness Index (CHI) : Instead of using NPS, the organization could also create a Customer Happiness Index of its own including the parameters it wants to measure. This approach has a benefit, as the organization has to “talk to the customer” and can understand the parameters that are necessary for its context.
7. % Paying Customers : Understanding the demography of how many people are using the paid product/services in the overall customer base helps us to understand the market better.
8. Conversion Rate : The gist of the funnel is that you start with people who are suspects, who then convert to prospects then convert leads which convert into sales. The percent conversion at each step is calculated. The overall conversion rate of a sales funnel is determined by taking the number of sales divided by number of suspects times 100.
More on Conversion Rate, here
9. Customer Acquisition Cost (CAC): CAC can be calculated by simply dividing all the costs spent on acquiring more customers (marketing expenses) by the number of customers acquired in the period the money was spent. For example, if a company spent $100 on marketing in a year and acquired 100 customers in the same year, their CAC is $1.00.
More on CAC, here
10. Customer Stickiness : Customer stickiness is the increased chance to utilize the same product or service that was bought in the last time period
More on Customer Stickiness, here
Developer/Builder
Sadly, we try to measure everything here. While the measures and metrics here give you an indication of “are we building it right?”, it doesn’t guarantee “are we building the right thing”. It is vital for a developers to know the metric they are getting along with having a view of all other metrics we talked about till now. The metric we use here, should always supplement the metrics that we talked about.
This group is technology oriented and hence, the metrics are more technical in nature as well.
Some of the metric that could help understanding “building it right” are:
1. Health of CI/CD pipeline : Are the builds happening right? Are they fast enough for the teams to experiment often?
2. Number of Green builds per day/week : Discipline within the team, so as to get code checked in daily, and builds are made often. The team can work in small batches as they know that the last logical piece of code they have submitted is working.
3. % of green builds : To continue, not just the number of builds are important, but also how many of them were good. A good developer doesn’t want others to be stuck because of him/her.
4. Unit Test Coverage : Adherence to Agile Test Pyramid the only way to be in the fast zone of Product Development. The Unit Tests should comprise ~80% of all tests and an essential part of building the quality-in.
5. Code Coverage : Code coverage is a term to describe which application code is exercised when the application is running. Measuring code coverage is a technique for understanding exactly which application code is being exercised.
6. Non Functional Test Coverage : Its essential to understand the non-functional requirements, called often as “ility tests” (usability, accessibility, reliability etc.) as they help in understanding what the customers or market don’t explicitly ask for.
7. Benchmarking of NFR’s : Benchmarking is an important part of understanding where we can reach. It could be against customer wishes or against industry standards. When we understand the current measures, we can aim/plan or negotiate the future measures.
8. Defects found per unit of Unit/Functional/Integration Test : While the number of defects might not indicate anything substantial, they do slow us down in building. If we can find a pattern or the magnitude, planning for releases becomes easy and market commitments live their date.
9. Number of defects per feature from Production/UAT/Customer : Feature is what customers pay for in most cases. Understanding the important features and making sure they are built with quality makes developers and builders of the system to concentrate more on where it matters.
10. Defects that can be lived with : Not all defects kill you! Pay attention to the most important ones and spend as less time as possible with the other ones. When delivering often is the necessity for a system/product/service, perfection is not always an answer and asked for as well.
11. Rework : Unknowingly we spend a lot of time on rework, which can well be utilized for the “work”. Practices like Split-testing, Pair Work (E.g. : Dev-Test, Test-Documentation etc. working in pair), having separate environments, testing and fixing with Developer can help the teams avoid the cost of late defect fixes.
Customer/User
The person, for whom we are building a solution is also a part of the system. The customers may/may not understand any of the numbers, measurements and metrics. They want their problems to be solved in a simplistic way, without learning too many things in the process. We should not be adding more stress to them.
The organizations/entrepreneurs who understand the above statement and work towards that will succeed and thrive!!!
Summary
We may or may not be able to demarcate different roles as I have done here in the real world. Care has to be taken to understand “what”, “when” and “how much” to measure depending on the circumstances a product/service works. Most of the measures talked about links us back to the customer and it is thoroughly essential that we take that view from the first step we take towards building a product or service.