Balancing Heuristics and Data

“I love data and here at Skyscanner we have loads of it. With 3 million users a day we can measure really small differences in almost any metric we care to measure. Data is great because it objective, it’s available to anyone in the company and it allows any idea to be challenged. It’s no surprise that given the increasing ease with which software companies can collect data that there has been a surge towards ‘data driven’ companies.

Data has other benefits over just reporting and measuring change. Used correctly it can help us find new use cases or problems in our product. This has been vital in the past for finding holes in our funnel and problems in different mobile platforms as well as bugs. This data can also be fantastic for prioritising different opportunities.

While data makes decisions more objective and tradeoffs more precise there is a dark side. Too much data can lead to analysis paralysis. Lack of data can build barriers against taking action and in the worst case bad data or data interpreted incorrectly can lead to bad decisions that defy common sense.

If we look at some of the most innovative current companies they bias very heavily towards how they will make the user feel rather than KPIs. How can this be? Surely more data driven companies with all of their experiments and data should be able to squash this opinionated approach. In this presentation I will highlight by example some of the dangers of relying only on numbers and argue that a balance of heuristics and data will outperform an over-reliance on either one”