Sabermetrics in tech biz

October 2, 2004

As Joe pointed out recently, running your business like the Oakland A’s is the latest fad. Only in America could a book about a non-winning baseball front-office guy become a business bestseller.

I actually read the book — a former CEO of mine gave all of us copies — and I find it unbelievable that the main local takeaway from the whole Sabermetrics approach is measuring engineering goodness via simple statistics. That is just totally screwed up. Engineers already have the ultimate metric: either our shit runs or it don’t. Believe me, engineering teams have an exquisitely calibrated sense of who is creating value and who isn’t. We watch each other’s work habits, checkins, bug fixes, releases, and rollbacks with a cold, beady eye that owes nothing to personal feelings. That means, oddly enough, that techniques evolved to measure people with “soft skills” — primarily 360-degree feedback — are likely to be much more effective with engineers than with any other group, because even our “subjective” measurements are actually more like well-filtered objective measurements.

Where a fresh statistical approach could really add value, I think, is with the people who don’t face the hard objective tests every day: PM, marketing, bizdev. I have no idea how to evaluate these people — and yet as an engineer, you know that they can make your life a lot better and create success, or they can make your life a living hell and crater the company. Take bizdev, for instance: my sense is that they get overvalued for things like revenue per unit (e.g. per-click pricing) and not devalued enough for the cost of the deals they make — so you can end up with a lot of deals that look great on paper but actually hurt the company. That seems a lot closer to a Sabermetrics stat like on-base percentage than anything engineers do. Remember, engineering is itself a resource to other groups in the company — so it seems a lot less important to measure individual engineer productivity than to measure the return on investment per engineering hour used by the other groups. If you’re really worried that engineers are slacking, which is risible in Silicon Valley, just fire the bottom 10% every year as measured by 360-degree feedback.

Actually… the ideal role to be measured by Sabermetrics-like statistical analysis is probably CEO. People are worried that engineers might be overvalued? Well, how about CEOs? What statistic tells you whether they are creating value or pissing it away? How do you know if a glamourous one is worth the extra cash and stock s/he can command over a no-name one? Especially in an early-stage startup, everything you do has a pretty big opportunity cost… and who makes those decisions except ultimately the CEO? And remember that measuring the CEO gives you a lot of bang for the buck… measuring individual junior engineers probably does not except over a large data set. So let’s turn the tables: if you were designing a metric for CEO goodness, what would it be?

5 Responses to “Sabermetrics in tech biz”

  1. Dylan Says:

    Joyce,

    Not sure why you would call Billy Beane a “non-winning baseball front office guy”… he managed to take a team with one of the smallest budgets in baseball to the playoffs four straight years. It surprises me that there are so many more Giants fans than A’s fans when the A’s have many more World Series titles than the Giants.

    Regarding using sabermetrics to measure engineers, that isn’t really what the book is about. The book is about buying low and selling high, and accepts the fact that in an unfair market, BB can’t afford superstars (top notch engineers) as they all get signed by a few teams with a lot more money to spend.

    Statistics are part of what BB uses to determine what is undervalued in the market. There are of course other factors that go into play in evaluating talent. However, something is also interesting about the book is that rather than relying on generally accepted statistics, they looked for ones that actually describe player value. An analog to our field would be that batting average in baseball is analogous to the number of bug fixes by a developer. Both stats look impressive on the surface, but they don’t really measure performance. So I don’t fundamentally believe that stats don’t help in evaluating an engineering team, but I don’t believe that anyone has come up with stats that come even close to what can be gleaned by a simple qualitative analysis. Relying on bad stats is worse than having no stats at all.

  2. Troutgirl Says:

    Hey Dylan! I am not a Giants fan, in fact I think following the NL is like incest… I just think, in the immortal words of Nelly, “two is not a winna and three nobody remembers”. ๐Ÿ™‚

    The thing that baffles me, especially in the Silicon Valley context, is that the book is about making do. It’s not about winning the World Series, is it? The A’s haven’t really come that close since 1989. It’s about _doing pretty well, very cheaply_. Is that what Silicon Valley is about? I would submit that it is not.

    In baseball the margins are relatively narrow — a great player hits 4 out of 10 times, a good player hits 3 out of 10 times, but they basically all have to hit. Is that what programming is like? In my experience, there are pretty vast differences between the expected value you can get out of engineers — everything from “build a company or a company-sized line of business out of this engineer’s ideas” to “does job competently, adds value slightly beyond salary”. Do three of the latter add up to one of the former? I don’t think so.

    I do think we agree that easy stats like bug fixes don’t tell you much… but I _still_ think there’s more value in trying to Sabermetricize non-engineers. ๐Ÿ™‚

  3. Dylan Says:

    Yeah, I didn’t mean to insult you by implying that you were a Giants fan… more of a reflection of my few months spent so far in the Bay Area. ๐Ÿ™‚

    I think that the A’s believe that as long as they make the playoffs, they have a shot. Each year they’ve made it, at times throughout the season they’ve been dominant enough to win it all. What has prevented them from winning it all is a combination of pitching injuries during the playoffs, and bad luck.

    I think that people who interpret the book as a way to make do are really missing the point. The point is how to win with less. So rather than looking for a way to make do with cheaper developers, people should be looking at how to find the right developers to get the job done.

    In baseball, there’s a clear goal, one objective (ignoring rebuilding, etc.). In software development (and business in general), there are different goals, different business lifecycles, etc., and as a result, different types of people are needed for different situations, cultures, business plans, etc., making sabermetrics that much more difficult.

    CEOs of public companies are already held accountable to sabermetrics: their quarterly earningsresults, a case which I would argue (as do the founders of Google, and Warren Buffett) is measuring the wrong stats and results, a reflection of our instant gratification culture.

  4. Marty Morrow Says:

    For the first few years of startup, Revenue growth seems to be what most boards are looking for. However, once you hit a meaningful threshhold, you need to be able to prove that you can make a profit. You’re right about engineers being very effective measurers of each other’s performance and they are often baffled about the promotion process. Probably because the promotion process doesn’t measure as well…

  5. Silicon valley engineer Says:

    I agree with your analysis of engineers evaluating performance. Although, I think it’s hard to evaluate other jobs just because most of the time, their lack of a deliverable or the inability to specifically define their role and responsibilities leads most engineers to view everybody else a incompetent, lazy, or failing to produce.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: