The Wrong Question

The Wrong Question

Sometimes the problem isn't that you have the wrong answer.

It's that you're answering the wrong question.

That sounds obvious… right?

However, my experience suggests it’s not all that obvious.

We spend enormous amounts of time trying to get better answers.

Better data.

Better models.

Better analysis.

More precision.

More confidence.

All useful.

But none of those things can rescue an answer to a question we didn't really need to ask.

Did it sell?

Let’s assume we launch a new product.

A few weeks later someone asks:

Did it sell?

Yes.

Great.

But put something new on a shelf and, surprisingly enough, somebody will probably buy it.

What did we really want to know?

Did it sell better than its nearest substitute?

Did it cannibalize something else?

Who bought it?

Were the sales net incremental?

Did it create new demand – or simply pull future purchases forward?

Five perfectly reasonable questions.

But which one (or all) were we trying to answer?

And depending on why we introduced the product in the first place, none of them may actually be the right one.

The question changes what we see

Consider a low-velocity product sitting on a shelf.

Looking only at its sales, removing it might seem obvious – make room for something that moves better.

But what if that product is disproportionately purchased by some of your most valuable customers?

And what if those customers routinely buy a lot of other things when they're there to purchase this low-velocity item?

Suddenly:

How well does this product sell?

may not be nearly as interesting as:

Who buys it?

or… what will happen if we remove it?

Same product.

Same customers.

Same transactions.

Different questions. Different insight.

The question determines what we're capable of seeing.

Don't lose the question

A good test usually starts with a question.

What are we trying to understand?

From that question comes a hypothesis.

What do we believe will happen – and why?

From the hypothesis comes the objective of the test.

What specifically do we need to learn to evaluate that belief?

These aren't the same things.

The hypothesis should help us investigate the question.

The test objective should give us a way to evaluate the hypothesis.

And our measurement should tell us something meaningful about the objective.

Question → Hypothesis → Objective → Measurement.

Sounds straightforward.

Until it isn't.

Sometimes we start with one question and somehow end up measuring something else.

We want to know whether a new product creates incremental value.

We hypothesize that it will generate incremental purchases without materially cannibalizing existing products.

Then we measure its sales.

It sold.

Success!

Maybe.

Go back to the beginning.

What did we actually want to know?

Sophisticated analysis can still answer the wrong question

This is where things get dangerous.

We can collect extraordinarily good data.

Build a sophisticated model.

Run diagnostics.

Estimate the effect precisely.

Produce a statistically defensible answer.

To… the… wrong… question.

In fact, sophistication can make the problem harder to notice.

The methodology is sound.

The numbers are precise.

The analysis survives scrutiny.

Everyone starts discussing the answer.

And somewhere underneath all of it is a question nobody stopped to challenge.

What are we actually trying to understand?

Sometimes we have the question handed to us

We answer questions because they're the questions we've always answered.

The metric is on the dashboard.

The report has always included it.

The industry benchmarks it.

Leadership asks for it.

Someone asked us to analyze it.

So we do.

Over time, we can become exceptionally good at answering questions nobody remembers choosing.

That doesn't necessarily make them bad questions.

But occasionally it's worth asking:

Why are we asking this?

And perhaps more importantly:

What decision(s) are we trying to make?

What would you do differently?

There's another question I've found useful:

If we get this answer or that answer, what would we do differently?

Sometimes the answer is obvious.

If this works, we scale it.

If it doesn't, we stop.

If customers value this, we invest more.

Great.

But sometimes we ask what we'd do differently if we knew the answer and realize:

Probably nothing.

That's interesting.

Maybe we're satisfying curiosity – there is some value in this.

Maybe we're measuring something because we've always measured it.

Maybe the question mattered once but doesn't anymore.

Or maybe we've never even connected the analysis to a decision.

Perhaps the next thing we need isn't a better answer.

Maybe it's a better question.

Are we still answering it?

Even when we start with the right question, we can lose it.

The analysis evolves.

New data become available.

Someone asks for another cut.

We discover something interesting.

The methodology becomes more sophisticated.

None of that is necessarily bad.

But every once in a while, it might be worth working backward.

Here's the result.

What objective did it address?

What hypothesis did that objective evaluate?

What question was that hypothesis supposed to help us understand?

Can we travel all the way back?

If not:

Are we still answering the question we started with?

Or did we answer a different one exceptionally well?

For lack of a conclusion…

We spend a lot of time looking for better answers.

Maybe every once in a while we should spend a little more time with the question.

What are we actually trying to understand?