Treacherous Promise of A/B Testing

Recently, I have been seeing a trend in product management to prove every feature right by A/B testing. PMs say “Great idea, let’s build it and measure if it works. Let’s A/B test everything like booking.com.” A/B testing is great for measuring the impact but has a significant flaw – it’s very expensive, as it requires building production-ready code. So PMs use their judgment to prioritise ideas and focus on increasing their teams velocity to deliver more experiments. But we are not a real user and our judgmens are often biased – 90% or A/B tests don’t bring positive results and over half of the delivered features don’t make impact. That’s extremely wasteful.

At the same time, I see organisations shielding PMs more and more from the actual users. PMs are busy, so users are “researched” by UX researchers, then results are worked on by UX designers. And after the PM team filters the results, the user research reaches the development team.

I want to remind PMs that “ideas” that we have are only hypotheses and are our bets and that there are several ways of verifying them. Product discovery is an exciting journey but requires focusing PMs to think back to users and their problems. Making real impact requires shifting delivering features to verifying ideas early. And I’d like to inspire them to embark on that journey again.