Data is Killing Our Intuition

I recently went to see The Batman, and as usual, I checked the Rotten Tomatoes score—85%, by the way— before walking into the theater. After the film finished, I realized that my expectations had been anchored based on the score alone. I enjoyed the movie but was that my judgment or someone else’s?

I have helped companies use data to make better decisions for the past six years. I have spent thousands of hours collecting it, storing it in all kinds of data warehouses, visualizing it in BI tools, futilely tracking down discrepancies, pulling my hair out in Excel, and so forth. 

I have seen how companies and individuals use data for good and how it can act like cement, holding them in place. I have seen great ideas get shut down due to the lack of data, and bad ideas get adopted due to faulty data. 

After all of this time and effort, I have concluded that data is killing our intuition. 

It’s affecting us all. We won’t enjoy a movie unless a score tells us we should. We don’t know what “good” sleep is unless we can track it and get a high rating. We can’t even choose a restaurant unless we read enough reviews.

Our quest for data-backed ideas has displaced our ability to use our intuition. 

Let’s look at how we got here and how we can get back into harmony.

How We Got Here

Three major trends have led us to where we are today.

  1. dramatic improvements in technology, 
  2. the overreliance on reason, and 
  3. the ever-increasing pool of choices.

Reason One: Dramatic Improvements in Technology

The first trend is all about technology. Computers, smartphones, AI, and other developments have made data usage easy. It’s now trivial to store billions of data points and run any analysis on it. Our lives are spent mostly in the digital world, leaving a trail that can be easily tracked.

Let’s take an example that I have heard multiple times. We swear that phones are listening to us because of how creepily accurate ads can be. If there’s a big brother, he seems to be obsessed with helping us find the perfect kind of shoes. 

I have helped companies build the very systems that allow them to personalize their marketing down to your secret preferences.

Most companies use their data for good, but there are always a few bad actors. Facebook and the 2020 US presidential election come to mind as recent examples. Technology has made data more accessible than ever, for good and bad. 

Reason Two: Overreliance on Reason

The second trend is the conquest of reason over religion. Long gone are days where superstition and gods rule our lives. Francis Bacon was one of the first people to shift our world towards the “study of nature itself through experience and experiment” aka the scientific method. 

Our adoption of this way of thinking has been nothing short of miraculous. In just a few hundred years, we have cured fatal diseases like polio, converted deadly viruses like HIV into manageable conditions, and in recent times, created a COVID-19 vaccine in 12 months instead of 10 years.

Science has given us better and longer lives, but we over-rely on it. Look at COVID-19 and how inconsistent governmental restrictions have been. If we are all “following the science,” why do we keep insisting on strategies that weren’t effective at controlling the virus? Case in point, border restrictions. 

Multiple studies have shown that locking down borders doesn’t prevent the spread of COVID-19 enough to justify the economic damage. Yet, every country imposed these restrictions at some point in time. We rely on science to such a degree that we accept un-scientific statements as indisputable facts.  

Science may give us a logical answer, but we have to use our intuition to weigh it against the reality of our world.

Reason Three: Ever-Increasing Choices 

The third reason is the ever-increasing pool of choices. We are now surrounded by world-class food from all over the world. We can buy products from the smallest cities and have them shipped for free to our doorsteps. Whenever I look for a product on Amazon, I feel overwhelmed by all the options available. How can we sort through hundreds of nearly identical options?

The sheer amount of choices has created companies simply for helping you choose, such as Yelp, TripAdvisor, and RottenTomatoes. Even Netflix has now released an option called “Play Something” to help us get over our hesitation to make a choice. 

How to Get Back to Balance

What’s the solution to getting back into balance? Re-learning to make decisions without data. The solution isn’t complex, but it can feel uncomfortable, especially in a world ruled by numbers.

Robert Greene talks about learning to “quiet the anxiety we feel whenever we are confronted with anything that seems complex or chaotic”. Life can be complicated. We have more demands than ever, and sometimes, there are just too many options. Data can be helpful, but it’s not a panacea. 

When I work with clients, I always tell them that we will explore how data can help them and harm them. Having a “data expert” tell you that you don’t always have to use data to make decisions can be a relief. Many executives and companies are working themselves out, trying to justify every hunch or decision with numbers. 

Along the way, he may reference data to validate assumptions or questions. If there’s no data, he weighs the upside and downside of making the wrong decision and then decides. He still makes mistakes, but he’s learned to adjust while flying instead of waiting for clear skies.

By helping clients understand the difference between data-driven and intuition-driven decisions, they can start to gauge when to use either approach. If someone has 30 years of experience in a field, the last thing I want to do is tell them to throw it out of the window. 

Think about the areas where you use data in your personal life to make decisions. Commit to watching a movie and deciding what you thought about it. You can then compare it with critics and see how your judgment differs. Choose a restaurant by simply walking around instead of looking at reviews. If you use data to measure your health, try merely thinking about whether you feel better, happier, or stronger.

The point is to become less dogmatic about our beliefs. We need to remember that we don’t know as much as we think. I love a quote by Will Durant saying that “our knowledge is a receding mirage in an expanding desert of ignorance.” Data has been a marvelous human invention, but there’s more than one way to think about our world. 

Conclusion

Next time you hear someone talk about being data-driven, think about the potential downsides that they may run into. Our love of data begins with hope, grows with confirmation bias, and dies when faced with uncertainty. 

My default is no longer to look at the data. I know too well how easy it can be to manipulate numbers or ask the wrong question. I wanted to develop my intuition and judgment, which means walking on a different path.

Featured image by Pierre-Etienne Vilbert

One more thing before you go!

I send a weekly newsletter called the Growth Needle. It's short, sweet, and full of interesting—and dare I say provocative—ideas. It's the best way to access my latest thinking and share your own thoughts. The next edition will be out on Tuesday, and you can receive it by subscribing below.

Want to see the goods before signing up? No problem! Here are a few of the recent editions: