Compared to What?

Compared to What?

Sales increased 8%.

Great.

Compared to what?

Last year?

Last month?

Plan?

Similar stores?

Stores where we didn’t do whatever it is we’re trying to measure?

Those aren’t the same comparisons.

And each comparison will undoubtedly produce a different answer.

If we’re trying to understand whether something we did caused an outcome, the comparison we really want is usually the one we can never actually observe:

What would have happened if we hadn’t done it?

That’s the counterfactual.

And it’s one of the fundamental challenges of measuring almost anything.

The thing that never happened

Suppose we introduce a new promotion in 50 stores and sales increase 5%.

Did the promotion work?

Maybe.

But what if sales would have increased 8% without it?

Or suppose sales declined 3%.

Did the promotion fail?

Maybe.

But what if sales would have declined 10% otherwise?

Knowing what happened isn’t enough.

We need some credible estimate of what would have happened otherwise.

That’s why we use control groups, experiments, historical comparisons and statistical models.

They’re different ways of trying to estimate something we can never directly observe.

A control group isn’t the counterfactual. It’s our attempt to approximate it.

And the quality of that approximation depends on more than selecting the right stores or running the right model.

We also need to understand what was happening on both sides of the comparison.

Reality doesn’t owe us a clean test

This gets particularly interesting in multi-unit businesses.

A store isn’t a pristine laboratory.

It has employees. Customers. Competitors. Weather. Construction. Local events. Marketing activity. Economic conditions.

And occasionally, a hurricane.

All while our test is running.

Some of that messiness is outside our control.

A competitor opens across the street.

State tax refunds are earlier and larger than normal.

Road construction changes traffic patterns.

A major employer or anchor tenant closes.

But plenty of the messiness comes from inside the organization too.

Operations launches a quarter-end contest.

IT introduces new technology.

Field Marketing runs a local promotion.

HR tests a new-hire incentive.

Loss Prevention installs a new video system.

Nobody is doing anything wrong.

Everybody is trying to improve the business.

The problem comes when those activities affect the KPIs we’re trying to measure and nobody running the test knows they're happening.

We don’t need paralysis.

We just need more visibility.

What changed?

Where?

When?

How?

Maybe we exclude some stores. Maybe we measure something separately. Maybe the overlap doesn’t matter at all.

But at least we’re making that decision with our eyes open.

We can’t account for what we don’t know happened.

And that isn't only a problem while the test is running.

The past was messy too

Comparisons need memory.

We often treat historical data as though the past were some clean, fixed baseline.

Sales are up 6% versus last year.

Okay.

Compared to what?

Last year we ran a promotion this week that we didn’t anniversary.

Or maybe we did anniversary it, but changed the offer.

Maybe store hours were different.

Maybe we changed media agencies.

Maybe we reorganized operations.

Maybe something unusual happened in the market that everybody knew about at the time.

But that was last year.

People change jobs.

Agencies change.

Responsibilities change.

Memories fade.

Eventually:

Hey, wait... didn’t we have a promotion last year?

is about the best institutional memory we have left.

Meanwhile, the data are still sitting there and “results” are being reported.

Businesses have become extraordinarily good at preserving what happened.

Sales. Transactions. Labor. Traffic. Margin. Customer satisfaction.

Store by store.

Day by day.

Sometimes minute by minute.

But we’re often considerably less disciplined about preserving what changed around those numbers.

At the time, everybody knew.

Two years later, maybe nobody does.

The numbers survived.

The context didn’t.

And without that context, a mathematically correct comparison can still tell an incomplete story.

A good comparison requires memory.

Not perfect memory.

We can’t document everything that might affect every KPI. And there will always be things we simply don’t know.

The objective isn’t omniscience.

It’s preserving enough of the history surrounding our evidence to understand how much confidence the comparison deserves.

For lack of a conclusion…

Testing in the real world is messy.

That's not an excuse for poor measurement.

It's the reason good measurement requires more than math.

We need thoughtful experimental design.

We need visibility into what else is changing.

We need organizational memory.

And we need enough humility to acknowledge what we may not know.

Reality doesn’t owe us a clean test.

But we can get considerably better at understanding the reality in which our tests take place.

So the next time someone tells you sales increased 8%, customer satisfaction improved four points, or transactions declined 3%, don’t decide immediately whether that’s good or bad.

First understand what you're comparing.

Then ask one more question.

Compared to what?