Does This Make Sense?

Does This Make Sense?

I love statistics.

I’ve spent a good portion of my career using them to understand customers, businesses, markets and what happens when we try something new.

But statistics have an important limitation.

Statistics don’t know what is reasonable.

They’ll happily estimate the relationship between just about anything you give them.

Watermelons sold.

White Volkswagens in the parking lot.

Wednesdays between 10:00 and 11:00 in the morning.

Give me enough data, enough variables and enough patience, and eventually I can probably find something statistically interesting.

Great.

But does it make sense?

This should have worked

Years ago, I was helping evaluate a new promotion.

It was a type of promotion I’d helped test and measure in other organizations many times before.

And I’d never seen one quite like it not work.

But this time?

Nothing.

Our analysis was telling us the promotion wasn’t lifting the business.

Well.. I guess that’s possible.

Just because something had worked elsewhere didn’t mean it had to work here.

My initial reaction however was pretty immediate:

That can’t be right. What’s going on?

My first thought was that I had a data problem.

Nope.

Then I wondered whether we had a bad set of control stores.

We tried different controls.

Nope.

But something else kept bothering me.

No matter which control group I used, I kept seeing a massive residual error pattern during the pre-period.

Why?

At that point, I wasn’t trying to prove the promotion worked.

I was trying to understand what I wasn’t capturing.

Eventually, we found it.

There had been a significant event in the test market during our pre-period that hadn’t occurred anywhere else.

The statistics weren’t wrong.

The data weren’t wrong.

And the controls weren’t necessarily wrong.

The real world was messy.

The statistics couldn’t know that.

There’s something else going on here

I think “That can’t be right” is a perfectly human reaction.

Sometimes it’s even an informed one.

I had experience with this type of promotion.

I had a plausible hypothesis for why it should produce an effect.

And now I had evidence telling me something very different.

Those things didn’t reconcile.

But that also didn’t give me permission to dismiss the result.

Prior experience isn’t proof.

And a plausible hypothesis can still be wrong.

But it did give me a reason to say:

There’s something going on here that I’m missing.

Maybe it was the data.

Maybe it was the controls.

Maybe something about this market was different.

Maybe the promotion really didn’t work.

I didn’t know.

So I got curious.

When experience, theory and evidence don’t reconcile, don’t simply choose your favorite. Figure out why.

Surprise is information

A surprising result isn’t necessarily a bad result.

Sometimes reality surprises us.

So when a result seems far-fetched, don’t automatically reject it.

But don’t blindly accept it either.

Check the data.

Check the design.

Check the implementation.

Check the model.

Look for confounds.

Look for real-world messiness.

Try to reproduce the result.

Maybe you’ll find a problem.

Maybe the result will survive every challenge you throw at it.

And when it does, you may have learned something considerably more interesting than what you originally set out to measure.

Surprise isn’t a reason to reject evidence.

It’s a reason to get curious.

Relationships need explanations

This is one reason I’ve never been particularly interested in blindly mining data for statistically significant relationships.

Exploration has value.

Unexpected patterns can point us toward things we didn’t know to look for.

But finding a relationship isn’t the same thing as explaining one.

Discovery can generate a hypothesis. It doesn’t automatically validate one.

I want some theory.

Not necessarily an elaborate academic theory.

Just a reasonable explanation for why we believe changing one thing might affect another.

What’s the mechanism?

What’s the expected direction?

When should the effect appear?

How large could it reasonably be?

If our explanation is right, what else should we observe?

That gives us something considerably more useful than a statistically interesting relationship.

It gives us something we can test.

Analysts still have to think

One of the easiest traps for an analyst is falling in love with the analysis.

The model is sophisticated.

The methodology is defensible.

The statistics are beautiful.

And there it is:

p < .01

But the person across the table who has spent twenty years operating the business may be thinking:

That doesn’t make any sense.

Don’t dismiss that reaction.

Maybe they know something you don’t.

And their intuition shouldn’t automatically override the evidence either.

Maybe the evidence is showing them something they don’t know.

The analyst brings analytical expertise.

The operator brings business expertise.

Put them together.

Ask questions.

Challenge assumptions.

Look for explanations.

Scrutiny isn’t criticism.

It’s how we figure out what the evidence really means.

Believable evidence

Ultimately, I’m not trying to produce statistically significant results.

I’m trying to produce evidence we have good reason to believe.

Statistical rigor is essential.

But it isn’t sufficient.

Does the direction make sense?

Does the magnitude make sense?

Does the timing make sense?

Is there a plausible mechanism?

Are there other explanations we haven’t considered?

Statistics are part of that assessment.

They aren’t a substitute for it.

There’s a difference between:

The statistics look great.

and:

This result is believable.

For lack of a conclusion…

The next time you get an exciting – or troubling – analytical result, make yourself uncomfortable.

Forget the p-value for a minute.

Forget the confidence interval.

Look at the result.

Think about what actually happened.

Think about what you expected to happen – and why.

Then ask:

Does this make sense?

If it doesn’t, don’t throw it away.

But don’t blindly accept it either.

Investigate it.

Maybe you’ll find a data problem.

Maybe you’ll discover some real-world messiness.

Maybe your hypothesis was wrong.

Or maybe you’ve discovered something genuinely important.

Because statistics can tell us many things.

Statistics don’t know what is reasonable.

We still have to use our heads.