Data-Driven Design is An Oxymoron That Forgot About the User

Data-driven design seems like a noble goal but I think it is misleading it. In this article, I’ll make an argument for why a user-driven design is better and how data can support this process. You already know how data can help you but I’m here to help you answer why this isn’t happening in real life and what you can do about it.

User-Driven Design, Not Data-Driven

I understand the desire to use data to improve your design. This helps companies go from something that looks great to something that performs well AND looks great. Data can help back up your design decisions and choices.

However, I think “data-driven design” is an oxymoron. You should be focusing on a user-driven design that happens to be supported by data in some situations. This is different from what typically happens in the real world.

I see companies that take anecdotes, convert them into a hastily designed A/B test and then hope it works. This isn’t data-driven or user-driven. It seems to be more ego-driven based on who has the highest rank. If the CEO has an idea, that will get the fast track to the top of the list.

Data can help combat this process but you need to be aware that it can also lead you down this path where every user is a data point and the human element is gone. Behind every data point, there’s an actual person with real issues and needs. Trying to get data to fully capture this person is incredibly difficult but we can do our best.

I also find it interesting how some of the most memorable campaigns or companies seem to be focused on telling the stories of individuals. Nike is represented by specific athletes, charity campaigns focus on individual stories and even banks are trying to convert their complex services by highlighting the individual.

Your designs can also do the same. They can be designed with the individual in mind while using data to back up your choices and results. This is the best of both worlds and what we will explore in the remainder of this article.

If you know anything about data-driven design, then you’ll know the basic concepts:

  • Use data to analyze your user base
  • Make sure to also capture qualitative data
  • Run experiments and measure the results

I’m not going to cover these ideas because they have been beaten to death in 1000 other articles. Instead, let’s start by talking about why teams have a strong bias towards quantitative or qualitative data and why this bias makes it harder to achieve the “balance” that everyone is recommending. 

Why Teams Are Biased Towards Quantitative or Qualitative Data

It’s common sense that you should have a nice balance of quantitative and qualitative data. The qualitative helps explain the data points while the quantitative helps keep the feedback in check. Even after becoming common knowledge, I still see teams that have a strong bias towards one of these data types.

It took me a while to figure out but I realize that the bias comes from what internal capability of the company has. Teams that have a strong technical ability to capture quantitative data love to use this data to answer everything. Any question can be easily converted into a chart of some kind.

Teams that don’t have a strong technical background or have limited technical resources, will gravitate towards qualitative data like surveys and NPS scores. It’s much easier (relatively speaking) to capture surveys than to track what users are doing with your product.

Fixing this bias isn’t just about making the choice. You need to work across teams and figure out where the limitations are. If you’re missing technical resources, you need to find a way to get these resources provided. If you have too many technical resources, you need to find a way to prove the value of qualitative data by showing all the insights that you’re missing by simply looking at data points.

Besides this bias, there’s one more thing that companies need to achieve the user-driven design framework I talked about earlier: process.

The Most Important Thing Isn’t the Data

The right process is the most important element of the user-driven design framework I mentioned earlier. You need to get most of the steps right, otherwise, you will only get some of the value.

The process at a high level looks like:

  1. Determine what area to focus on or improve
  2. Collect the qualitative and quantitative data
  3. Analyze the data to find insights and patterns
  4. Design your experiments and their measurement 
  5. Analyze experiment results
  6. Roll out wins and/or go back to step 1

It’s a circular process where you’re learning as you go along. It’s not enough to just have data, you need to do something with it. You also need a way to measure your experiments which could be A/B testing or another format. 

Beyond the process, there are also a few best practices that you should keep in mind when going through the transition into user-driven design.

Best Practices for User-Driven Design

How do the people fit into all this?

You should think about all the people that will be part of your process and what role you expect of them. They should also be aware of their role to avoid confusion or conflicts. This is a process that is heavily dependent on people and will fail if you don’t get the right people on board.

Do you have the technical resources to support this kind of design process?

Back to my section on qualitative vs quantitative, technical resources are important. They will help you collect the right data and will also help you roll out and measure your experiments. Tools are getting easier than ever but you still need guidance from your technical team. Maintain a good relationship with them, you will always need them.

Can you determine the biggest opportunities?

Step 1 in the process seems straightforward but it’s one of the areas where I see teams stumble consistently. Biggest opportunities aren’t just what is obvious but what the data and customers are telling you. Ideally, you should have a way to rank ideas based on their potential impact, your confidence in their improvement and how much effort is required to deploy them.

I love data as much as the next person but I understand it’s limitations and the role it plays. Data at it’s best is a supporting character in the movie. This doesn’t mean it can be ignored but you shouldn’t make it the entire focus. The user-driven design framework is meant to give data on the appropriate importance while also focusing on other important elements. 

One more thing before you go! Do you know how to get more insights out of your data? 

All companies are sitting on a goldmine of data that they haven't fully explored. It's not about technology or capturing more data. The key is to learn how to make the most of your current data and convert it into actionable insights. This is the main idea behind my first book, The Data Miage: Why Companies Fail to Actually Use Their Data

I'm excited to announce the release of the book through all major retailers. If you're interested, you can download the first chapter for free using the form below. You'll learn what the best data-driven companies do differently and how to make sure you're playing the right data game.