The Unintended Consequences of Metrics

Among economic development circles, we have heard the phrase “unintended consequences” a lot over that past few years. My favorite recent topic that is subject to unintended consequences is metrics. Matt Hamilton, Anne Swift and I, all of Carnegie Mellon, are researching the impact of technology-based economic development programs on the regional economy. So far, this project reveled to us the seemingly endless number of cities and regions asking the inevitable question “what should we track to know if we’re on track?”

The answer is complicated beyond imagination because for benchmarking, trend analysis, and the like, no two regions face the same supply, demand, and political economy conditions. Even with a sophisticated multivariate regression, we currently lack the economic understanding (theory and evidence) to make meaningful comparisons.

But the grander issue, the “big picture” if you will, is: what happens when we start tracking data… and publishing it? On a regular basis? That’s where unintended consequences come in.

We all recall the famed Frederick Taylor studies, the grandfather of efficiency and operations management. Taylor, it is recounted, would time workers with a stopwatch as they performed various tasks, only to later realize that workers changed their practices when they knew they were being timed.

This happens everyday in economic development. For example, everyone’s favorite yardstick, patent counts, is an oft used metric for regional innovation output… or capacity, depending on the consulting report. And, I’ve seen several consulting reports by the same company that – at least pick one for consistency, but, it can’t be equally both. Anyway, if you start measuring patents, then the investment strategy of a region quickly turns to favor patent-intensive industries, like pharma, biotech, and chem. Twenty years of economic research by Wes Cohen, Dick Nelson, John Walsh, Sid Winter and others has shown that these industries focus on patents far more than semiconductors, software, and other areas. As a result, measuring patents only measures the mix of biotech and chem industries in a region and rarely represents actual performance, thereby veiling deeper economic trends.

Patent counts are perhaps the most obvious, but represent just one of many metrics that fall into this trap. Recognizing this problem, econ dev gurus propose a “multi-tiered” or “multi-layered” measurement approach drawing from a basket of measures. Now we take one bad measure, and multiply it by ten. We’re only increasing our “measurement error” in statistical terms.

The bottom line: focusing on econ dev metrics without nuanced understanding of what popular measures are really capturing dangerously misguides policy and business leaders. Dangerous is a heavy-handed word to use, but I mean it. And, this concern is compounded when one considers the temptation to report metrics on a short-term (annual or quarterly) basis when we all know economic development is a long-term investment.