A Perfectly Horrible Model

A Perfectly Horrible Model

Years ago, during my Blockbuster days, we developed a series of models to help forecast first-year revenue for new stores.

The methodology was pretty sophisticated. We used several different models and an algorithm that evaluated the level of agreement among them.

But one of the core models was a multivariate OLS regression.

And that model produced something one of our executives remembered from a statistics class he'd taken in B-school.

R-squared.

Our R-squared was consistently somewhere around .70 to .75.

He wasn't impressed.

“It needs to be .90 or higher.”

We had some version of that conversation more than once.

Finally, I told him:

“I can make it 1.00 if that’s what you need.”

Bingo!

I went back to my office and reran the OLS model with one important change.

I just removed the intercept.

R² = 1.00.

Whoop! There it is!

There was just one problem.

It was now a perfectly horrible model.

I was able to give him what he was asking for.

And at the same time, demonstrated why what he'd asked for wasn't actually what he wanted.

He wasn't fixated on R-squared because it was necessarily the most important measure of our methodology.

It was the measure he understood – or at least he thought he did.

That's a pretty easy trap to fall into.

When we're faced with something complicated, we naturally gravitate toward the parts we understand.

Familiar feels more comfortable.

More tangible.

More relevant.

More important.

But...

Familiarity can masquerade as importance.

So what?

There's a famous observation from statistician George Box (not to be confused with Al Gore’s Lock Box) that I've repeated a version of throughout my career:

“All models are wrong, but some are useful.”

I usually put it a little differently:

Models, by definition, are wrong. But some can be really useful.

Models are just an attempt to explain some version of reality.

But they can’t contain everything.

Not possible.

If they did, we'd just call them reality.

 (Or AI).

The question isn't whether a model perfectly describes the world.

The question is whether it's useful for the decision we're trying to make.

And our methodology was pretty darn useful.

It was making the correct out-of-sample business recommendation roughly 90% of the time.

The methodology we'd been using before that?

About two-thirds of the time.

So...

Which would you rather have?

A sexier model with prettier statistics?

Or a methodology that actually makes you more money?

I'll take door number two.

The metric isn't the objective

Don't get me wrong.

Metrics (statistics) matter – I rely on a ton of them daily.

We need ways to evaluate performance, compare alternatives, identify problems, and understand whether something is working.

Metrics are just constructs.

They're representations of something we actually care about.

But… they aren't the thing itself.

And things unravel when we forget that.

Because once a metric becomes the objective, we can become remarkably creative at improving the metric without improving the thing it was supposed to measure.

And sometimes… we can even make the thing worse.

That's not just an analytics problem.

Organizations do this everywhere.

They improve customer-service metrics while frustrating customers and the ops team.

They hit productivity targets producing things nobody needs.

They hit short-term revenue targets at the expense long-term value.

And yes...

We can improve model statistics while making the model less useful.

Never let an artificial measure of success become more important than actual success.

Useful beats perfect

There's a temptation in analytical work to keep improving things simply because we can.

Another variable.

Another specification.

Another transformation.

Another metric.

Another decimal point.

Maybe another model.

Sometimes that's exactly what the work requires.

But sometimes we're just making the machinery more impressive for ourselves.

At some point, someone needs to bring us back to why we built the thing in the first place.

What decision is this supposed to help us make?

What does “better” actually mean?

And perhaps most importantly...

So what?

Because analytical sophistication isn't the objective.

Neither is statistical perfection – (which doesn’t exist by the way).

The objective is a better decision.

A model doesn't need to be perfect.

By definition, it can't be.

It just needs to be useful.

For lack of a conclusion...

The next time someone shows you an impressive metric, don't immediately ask whether it can be made even better.

First ask what it's supposed to represent.

Then ask something more important:

If this number improves, does what we actually care about improve with it?

If yes...

Great. Keep going.

If not...

You may be milking mice.

And if someone insists that your R-squared simply has to be 1.00...

Well...

I know a trick.