Sabermetrics in tech biz

October 2, 2004

As Joe pointed out recently, running your business like the Oakland A’s is the latest fad. Only in America could a book about a non-winning baseball front-office guy become a business bestseller.

I actually read the book — a former CEO of mine gave all of us copies — and I find it unbelievable that the main local takeaway from the whole Sabermetrics approach is measuring engineering goodness via simple statistics. That is just totally screwed up. Engineers already have the ultimate metric: either our shit runs or it don’t. Believe me, engineering teams have an exquisitely calibrated sense of who is creating value and who isn’t. We watch each other’s work habits, checkins, bug fixes, releases, and rollbacks with a cold, beady eye that owes nothing to personal feelings. That means, oddly enough, that techniques evolved to measure people with “soft skills” — primarily 360-degree feedback — are likely to be much more effective with engineers than with any other group, because even our “subjective” measurements are actually more like well-filtered objective measurements.

Where a fresh statistical approach could really add value, I think, is with the people who don’t face the hard objective tests every day: PM, marketing, bizdev. I have no idea how to evaluate these people — and yet as an engineer, you know that they can make your life a lot better and create success, or they can make your life a living hell and crater the company. Take bizdev, for instance: my sense is that they get overvalued for things like revenue per unit (e.g. per-click pricing) and not devalued enough for the cost of the deals they make — so you can end up with a lot of deals that look great on paper but actually hurt the company. That seems a lot closer to a Sabermetrics stat like on-base percentage than anything engineers do. Remember, engineering is itself a resource to other groups in the company — so it seems a lot less important to measure individual engineer productivity than to measure the return on investment per engineering hour used by the other groups. If you’re really worried that engineers are slacking, which is risible in Silicon Valley, just fire the bottom 10% every year as measured by 360-degree feedback.

Actually… the ideal role to be measured by Sabermetrics-like statistical analysis is probably CEO. People are worried that engineers might be overvalued? Well, how about CEOs? What statistic tells you whether they are creating value or pissing it away? How do you know if a glamourous one is worth the extra cash and stock s/he can command over a no-name one? Especially in an early-stage startup, everything you do has a pretty big opportunity cost… and who makes those decisions except ultimately the CEO? And remember that measuring the CEO gives you a lot of bang for the buck… measuring individual junior engineers probably does not except over a large data set. So let’s turn the tables: if you were designing a metric for CEO goodness, what would it be?

No news is OK news

October 2, 2004

People seem to wonder, so let me mention that I have not taken a job. I spent 30 days straight running around meeting some of the most amazing people in the business, and I’m amazed by the energy revving up in the Valley right now… but I’m cashing no one’s checks yet. I wish I could mention a lot of the coolness I saw, but I feel like practically every thought in my head is covered by someone’s NDA at the moment.

Any spare time not spent on interviews, I visited the family and tried to deal with the car situation. As expected, the repair on the transmission of the Stealthmobile cost more than the car was worth… so I donated it to the Apache Software Foundation (or tried to, the donating charity might have screwed up). It’s kind of humiliating to buy a car without having a job, but luckily Tim promised to share his creditworthiness with me in solvency or in unemployment as long as we both shall live.

I’m thinking I could use a little break now. This has been a pretty hectic 9 months — lots of highs, lots of lows. I have some little projects to keep me busy for a few weeks, and then we’ll see where it’s at.

Everyone else is talking about the Google Labs Aptitude Test, but I’m wondering something a lot more simple-minded: now that CS students typically learn high-level languages first (or only), where are programming interviews going?

A lot of the classic programming interview questions are actually sort of stupid in themselves, but supposedly test whether you have a reasonable CS background and/or have memorized one of the cram books available. Exhibit A is reversing a singly-linked list, a thing that very few people actually have to do in life but which still shows up in programming interviews (generally, in my experience, those given by people who haven’t actually programmed since the 1980’s).

In C, there are some little subtleties in the implementation, like remembering to pass a pointer to the head pointer inside a function. But in Java, anyone who does not do it this way:

import java.util.*;
List mylist = new LinkedList();

is a moron. This is the Java way to accomplish the stated task. Any other way is artificial, wasteful, and distasteful. But what does it prove about a candidate?

Anyway, bad hiring managers could at least get away with stuff like this while C/C++ were still the standard in CS education. But now that Java and C# seem to have completed their takeover, and may even be giving way to Python and Ruby, it will be interesting to see whether the dinosaur managers have to rethink their interview tactics.