As we head into Q4 and the end of the year, everyone is scrambling to review their OKRs. People in all industries and functions are asking themselves: What do the metrics say? Did I hit my targets? How can we manipulate the data to tell the right story?
(That last one sends shivers down my spine.)
But the point is that one metric - one solitary number - may not be enough to tell the whole story. Often one static number doesn't include enough context to enable a full understanding of what's going on. And that metric may not have garnered the right results, either. Or if it did, it may have been at the expense of more important pieces.
Goodhart's Law says that when a measure becomes a target, it ceases to be a good measure.
Have you heard of Goodhart's Law? I've recently grown intrigued by a series of laws, one of which is Goodhart's Law. Goodhart's Law states: "When a measure becomes a target, it ceases to be a good measure" (BuiltIn). I think BuiltIn does a nice job of explaining the law: "[W]hen we set one specific goal, people will tend to optimize for that objective regardless of the consequences. This leads to problems when we neglect other equally important aspects of a situation." In other words, when you optimize for X, you may unintentionally yield Y and/or neglect Z, either (or both) of which might be worse than achieving X.
Here's a graphic from SketchPlanations to illustrate an example of Goodhart's Law in action:
In the graphic, we see two different scenarios play out. Let's examine each one.
Scenario 1
In Scenario 1 (at left), Boss tells Employee to increase the number of nails made. Maybe that's their company goal for the year - increase volume. So Employee proudly shows Boss his work - he produced thousands and thousands of tiny nails. Boss is taken aback. 'What are these? They're too tiny - they're unusable. And they're nothing like the nails we're known for - our customers won't want to buy these. And we don't have the ability to create new packaging to fit more of these tiny nails either! What happened?!' Employee argues, 'You only told me to increase the number of nails made, so I optimized for that metric.' Employee isn't wrong - just misguided. Boss should have been more specific - and strategic - when setting that goal.
Scenario 2
Then, we see Scenario 2 (at right): Boss tells Employee: 'Okay, fine - clearly that metric didn't work, so let's shift. Let's now make the goal to increase the weight of nails made.' In Boss's mind, Employee will increase the nail size back to normal size and maybe also increase the volume. Employee takes on the challenge and, optimizing his processes and efforts to increase the weight of the nails he works on, he proudly shows Boss that he achieved the goal: He created a few giant, heavy nails. He points to three gigantic nails, each of which is the size of a human body and weighs even more. Boss is irritated beyond belief! 'How are we supposed to sell only three nails? And who will buy nails that big - what purpose would they even serve? How could we beat our competitors by producing products that no one wants?' Employee argues that Boss didn't say anything about customer empathy or competitive analysis or sales strategies - Boss only said to increase the weight of nails produced, and that's what he did. Thus, the measure that became a target was not a good measure of success.
In both of these scenarios, other considerations are lost. Usability, user experience, customer empathy, competitive analysis, sales strategies, value proposition, packaging, marketing, internal processes, long-term impact - all of the interdependent, supporting yet necessary pieces of the puzzle were forgotten. As a result, they will suffer.
One Metric can Miss the Mark
Sometimes, companies fall prey to Goodhart's Law by focusing only on one metric and encouraging all teams to organize toward that one lone target. The executives think they are aligning teams and motivating everyone with a common goal. But the individuals know better, because they experience the other elements of work that fall by the wayside when only that one metric is evaluated.
This can be especially detrimental in companies where the culture is already toxic. For example, let's examine a [fake] company where sales and marketing teams are judged against a metric that has become a lone target against which all efforts are evaluated, which then causes riffs in the organization because of miscommunication, misalignment, and a lack of understanding of the big picture.
Company T decides that they need to make a certain revenue amount in a year, and they announce that everyone in the company is responsible for producing outcomes that contribute to that revenue. However, all deals must go through the sales team, so no other team can cinch a direct sale - they can only provide sales with opportunities. One year, the sales team at Company T falls short of their revenue goal, but the sales reps feel buoyed by the fact that the executives are holding everyone in the company responsible for generating revenue - not just their team. So they don't feel as responsible as they might if their efforts held more weight than the other teams' efforts. When asked why they fell short of their revenue goal, they blame marketing for not giving them enough leads.
(Right here, Company T should talk to the failing team about accountability and taking responsibility of their own actions, and should talk to their leader about innovative strategies and motivation practices and other things they CAN control. But I digress.)
Company T's executives turn to the marketing department. How much revenue did you generate? They ask. Marketing can't provide one static number, because they haven't set their programs up to work that way. They were never asked to, and they never had the resources to do so. But the executives still blame marketing for the company falling short of its revenue goals. From that point on, executives start looking more closely at the marketing department as being the reason revenue isn't high enough. Marketing employees start to feel demotivated and frustrated, because their lead gen efforts actually do provide leads to sales - the sales team is simply not converting those leads into revenue. They are doing their part, but they have an added dependency on another team (sales) later in the process, so their success metric isn't evaluating only their success - it's evaluating the work of another team, which they cannot control.
Therefore, we see Goodhart's Law at play here, amplified in a poor culture: Marketing might look unsuccessful if the company is focused on one metric as a target that it is not in their control. And when the metric became the target, other issues fell by the wayside, such as upholding accountability or searching the full process for roadblocks. Beyond just this law, we see a misalignment of metrics here, as well as blindness to the full process and other, longer term or nonlinear impacts teams can have. The executives were missing the part of the process where marketing gave leads to sales but sales did not convert them. Identifying this would have changed their behavior and, thus, the outcome. If the marketing department were measured on a different metric - say, the number of leads they gave to sales - they would have exceeded expectations. But that one measurement was not a good fit for this team, particularly in the way they were organized, so the metric becoming a target did not serve marketing or showcase their positive impact in ways beyond that one measure. It made the executives blind to everything else they were doing right that should have been celebrated. The metric becoming a target also hurt the sales team because by only stressing the importance of hitting the target, sales leaders may have forgotten to instill in their teams the importance of accountability, respect, cross-team communication, long-term relationship-building, creative problem-solving, curiosity, or innovation. They could have brainstormed new ways to find new prospects or devised creative new ways to work together with other teams to hit their goal, instead of pointing fingers, finding a scapegoat, or only letting that one target matter - even if it wasn't hit.
Don't forget - if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid. Don't judge people based on the wrong metrics. And don't judge people based on a singular metric that doesn't tell the whole story.
If you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid. (~Einstein? Maybe? Maybe not? Doesn't matter - the sentiment still stands.)
One Metric can Cause Misaligned Outcomes
Targets don't necessarily have to be wrong in order to produce the wrong outcomes. As we saw in Scenarios 1 & 2 with the nails, it wasn't incorrect or non-strategic for Boss to want to increase the number of nails made or even the weight of nails made. But when you only focus on that one metric, you miss the way that metric is obtained. You miss the process.
Don't forget about the process whereby that metric might be achieved. For example, I recently went to a beautiful garden space and met the owner. He explained that predator animals had crossed into his gardens and his neighbors' yards and attacked a few of their chickens. He and his neighbor agreed they wanted to fix the predator problem. The garden owner wanted to put up a fence; the neighbor wanted to shoot the predators. Same problem, same metric (fewer predators, or fewer attacked chickens), but two different outcomes - one okay and totally feasible (putting up a fence), one not great (shooting the predators).
The same thing happens in workplaces all the time. You may tell your team to increase output (in one measurement, such as more blog posts published, or more meetings booked, or more trainings completed) but not have a conversation around why that's important or how to do it right or what the process looks like, so they may end up sacrificing other things you care about, like quality or relationships or integrity, in order to hit that one goal.
Don't forget about the process whereby that metric might be achieved.
Be aware of the intended outcomes that could arise by focusing too deeply on one metric.
Solutions to Mitigate Goodhart's Law
So what do we do? We need goals. We need OKRs. We need ways to know whether things are working. We need to prove our value to our company executives or clients. How do we mitigate the risk of Goodhart's Law while remaining data-driven and goal-oriented?
Here are some possible solutions:
Take a holistic view. Even if you want to have one overarching goal for your company, understand the big picture and how every department fits into it. Understand how pulling one lever affects the others, and be sure that everyone is also aware of these strategic inner workings of your company. Also give consideration to intangibles such as culture, processes, systems, tools, relationships, people operations, and management trainings. All of these factors contribute greatly to every single metric and should be given due attention. Combine qualitative and quantitative measures for a more comprehensive understanding of what's going on at macro and micro levels.
Personalize metrics. Instead of asking the entire company to contribute to one metric (e.g., # of widgets sold), communicate the outcome as an overarching company goal, but then break down how each department will contribute to that goal in their own unique ways. Provide departmental metrics based on what each department can control and the unique ways they provide value.
Communicate more effectively. In Scenario 1 & 2 above, if Boss had given clearer instructions, the wrong size or quantity of nails would not have been produced. Engage in dialogue at the start of a project and at the goal-setting phase to ensure everyone is aware of the preliminary strategy and the intended outcomes.
Uncover the context. What do different metrics mean? What is the full story? Look at all of the work going into making one metric happen, and understand why that works or doesn't work. Explore all of the other avenues being covered and consider other metrics that could overlap or play a part in painting a picture of success as well. Context is everything, especially in data. Don't let one metric cloud your judgment. Dig deeper.
Diversify your measurements. One number probably isn't going to be enough to tell a complete story. Instead, devise a set of measurements to constitute your assessment foundation. What are the impacts of each? What inputs are required to generate the intended outcome of each? Don't spread yourself too thin with too many goals, but have a few solid metrics to round out your view. (Here's a funny anecdote: If you have an iPhone and set up your thumbprint to unlock your home screen, you couldn't just place your thumbprint on the home button once and be done with it - you had to place your thumbprint in many different angles around the button so as to give Apple the full picture of your thumbprint and to allow for the various ways you might press it in the future. Okay, maybe it's a little bit of a stretch, but it aligns with the idea of diversifying your measurements and understanding that you need a little more data than just one number to understand success.)
Evaluate processes. Look at your full journey and understand where dependencies lie and where projects are handed off so you can determine who should be responsible for which parts of the journey, helping you determine who should be responsible for which metrics based on what they can control. For example, if your systems are set up such that all of your revenue opportunities must go through the sales department, you cannot hold any other department or function fully responsible for revenue, because of the dependency on sales and the fact that other teams cannot control the sales team's actions. No one team should be evaluated based on work that is partially done by another team; effective metrics only measure what the party in consideration can control. Other teams' goals and OKRs must end with their handoff or the way they influence those deals, but they cannot be stretched to the end of the line when another team is responsible for some key steps - especially if that final-touch team is not performing adequately. In following our marketing & sales example, instead of saying marketing is responsible for the revenue the sales team closes, find the point in the process where the marketing team hands something over to the sales team and is no longer in sole control of their work. That is the last possible metric you can measure for the marketing team. You have to look at the full journey and identify that chasm (in this example, occurring where sales is not working the leads marketing is giving them, thus not closing revenue deals) and make the OKR the last step BEFORE that chasm (marketing handing off leads, in this example). It may not look like marketing is influencing as much revenue, but if marketing is only in control of generating leads, then measure how many leads they are generating and handing over to sales. Otherwise the OKR is not fair, will not paint the appropriate picture, will not empower the right teams in the right ways, will not reinforce the right behaviors, and will lead to burnout for teams who are doing their parts but being penalized for teams on whom dependencies lie not doing theirs.
Check in regularly. If you only talk to your team once when you tell them the metric you want them to achieve and then once at the end when the project is done, you are missing out on everything that happens in the middle. Schedule regular check-ins at various milestones so you can receive updates from your team, help them work through any struggles, clear any roadblocks for them, support them, and guide them toward the right goal. Checking in regularly will also ensure they are not doing the wrong thing or that other, worse outcomes are not being generated as a result.
Broaden your definition of success. On that note, if you are using one number to define success, you are missing a LOT. I mean a LOT. Think about all of the work it takes to create one one-sheet or facilitate one workshop - all of the preparation, research, drafts, edits, feedback loops, audience empathy, competitive analysis, resource review, strategic alignment, vetting for brand voice or takeaways or engagement - one singular output incorporates dozens of elements of input. Ignoring all of that input is naive and, in a word, disrespectful. Respect everyone's efforts and acknowledge that it takes a village. One number ignores all of the work that went into that number. Map out the full journey for one process and you'll get an idea pretty quickly of all of the wonders your team works on a daily basis. Celebrate and appreciate them, and don't whittle down all of those conversations and thought processes and brainstorms and collaborations and editing sessions and execution blocks and revisions and personal growth tracks and feedback integrations and compromises and wins into one singular number.
Reflect:
What's working for you? Where do you run into issues? How can you adjust your metrics to enable the right behaviors? Which solution above is most feasible for you? Which one might be the most impactful as you explore ways to devise the appropriate metrics for the appropriate teams and goals? Let me know in the comments!
Sources:
コメント