What a year of A/B tests taught us

Most tests are inconclusive — and that’s the point. Lessons from twelve months of being wrong politely.

The shape of a year

Of roughly forty experiments, six moved the needle, eight broke something, and the rest taught us we didn’t have a strong opinion in the first place.

What we stopped testing

• Button color. Always.
• Copy variants without a hypothesis.
• Anything we couldn’t afford to lose for two weeks.

If you can’t describe the loss case, you don’t have a test — you have a coin flip with extra steps.

What we test now

Structural changes: who sees what, in what order, with what default. Tests at that level move enough to be worth the wait.

A short rule of thumb