Sabermetrics in tech biz
October 2, 2004
As Joe pointed out recently, running your business like the Oakland A’s is the latest fad. Only in America could a book about a non-winning baseball front-office guy become a business bestseller.
I actually read the book — a former CEO of mine gave all of us copies — and I find it unbelievable that the main local takeaway from the whole Sabermetrics approach is measuring engineering goodness via simple statistics. That is just totally screwed up. Engineers already have the ultimate metric: either our shit runs or it don’t. Believe me, engineering teams have an exquisitely calibrated sense of who is creating value and who isn’t. We watch each other’s work habits, checkins, bug fixes, releases, and rollbacks with a cold, beady eye that owes nothing to personal feelings. That means, oddly enough, that techniques evolved to measure people with “soft skills” — primarily 360-degree feedback — are likely to be much more effective with engineers than with any other group, because even our “subjective” measurements are actually more like well-filtered objective measurements.
Where a fresh statistical approach could really add value, I think, is with the people who don’t face the hard objective tests every day: PM, marketing, bizdev. I have no idea how to evaluate these people — and yet as an engineer, you know that they can make your life a lot better and create success, or they can make your life a living hell and crater the company. Take bizdev, for instance: my sense is that they get overvalued for things like revenue per unit (e.g. per-click pricing) and not devalued enough for the cost of the deals they make — so you can end up with a lot of deals that look great on paper but actually hurt the company. That seems a lot closer to a Sabermetrics stat like on-base percentage than anything engineers do. Remember, engineering is itself a resource to other groups in the company — so it seems a lot less important to measure individual engineer productivity than to measure the return on investment per engineering hour used by the other groups. If you’re really worried that engineers are slacking, which is risible in Silicon Valley, just fire the bottom 10% every year as measured by 360-degree feedback.
Actually… the ideal role to be measured by Sabermetrics-like statistical analysis is probably CEO. People are worried that engineers might be overvalued? Well, how about CEOs? What statistic tells you whether they are creating value or pissing it away? How do you know if a glamourous one is worth the extra cash and stock s/he can command over a no-name one? Especially in an early-stage startup, everything you do has a pretty big opportunity cost… and who makes those decisions except ultimately the CEO? And remember that measuring the CEO gives you a lot of bang for the buck… measuring individual junior engineers probably does not except over a large data set. So let’s turn the tables: if you were designing a metric for CEO goodness, what would it be?