Better Than We Were

Better Than We Were

There was a point in my career when I helped decide where Blockbuster would invest roughly $200 million each year opening new stores.

That's real money.

So the conversations weren't about statistics.

They were about responsibility.

Over the course of a week or so, a group of very smart people found themselves wrestling with a model structure that made all of us a little uncomfortable.

Economists.

Statisticians.

Engineers.

Operations researchers.

Some accomplished academics.

Everyone in the room understood exactly why we were uneasy.

And that was a good thing.

We had started experimenting with what would probably be called model stacking today.

One model estimated the probability that a prospective location would become a "stud" or a "dog."

Those probabilities then became inputs into additional models estimating first-, second-, and third-year revenue.

Along the way there were latent class models, Gibbs sampling routines written in GAUSS, and enough matrix algebra to make your head hurt.

None of that was particularly common in the late 1990s.

What surprised us wasn't the complexity.

It was the consistency.

The models just kept making better investment recommendations than the business was making on its own.

That should have settled the debate.

It didn't.

The discussion wasn't whether the models worked.

The discussion was whether we understood why they worked as well as they did.

Those are two very different questions.

The academics in the room were asking exactly the questions they should have been asking.

We all were.

Eventually I said something that has stayed with me for almost thirty years.

We know this isn't completely kosher.

We also know it's right far more often than we are today without it.

Let’s adopt it.

We watch it closely.

And if the results start to tell us we're wrong...

we'll change course.

Looking back, I don't think that was a statement about confidence.

I think it was a statement about humility.

We weren't choosing between a perfect model and an imperfect one.

We were choosing between an imperfect model...

...and imperfect judgment.

Those aren't the same decision.

Sometimes organizations compare every new idea against perfection.

More times than not, the better comparison is much simpler.

Is it better than what we're doing now?

For Lack of a Conclusion…

I don't remember much of the code anymore.

I remember the conversations.

I remember the questions.

I remember how seriously everyone took the possibility that we might be fooling ourselves.

Looking back, I think that's exactly why the decision held up.

We didn't trust the models because they were “sophisticated.”

We trusted them because they had earned the opportunity to improve an important decision.

And we remained willing to change our minds if the evidence changed.

I'm not so sure those are the same thing.

What do you think?