Opinions expressed by businessman the collaborators are theirs.
Some people say they give the customers what they want, but that’s not my approach. Our job is to figure out what they want before they want it. – Steve Jobs
Understanding customer preferences is critical to driving business growth, but customers typically don’t know what they want. As a business, it is beneficial to better understand your customers and deduce what they want from you. This is where experiments help!
Startups with limited historical data typically gain a better understanding of customer needs by running business experiments on a feature rather than running analytics on historical data. You can make experimentation the key to your startup’s growth if you can establish the right strategy, approach, and methodologies. First let’s talk about what experimentation is.
Related: Experimentation and A/B Testing: A Must-Have Ecommerce Growth Strategy
What is experimentation?
Experimentation, or what is sometimes known as A/B testing, is a method where a business hypothesis is tested practically on consumers. Often, an organization may not have historical data to analyze business decisions. Likewise, they might be looking at some decisions where historical data is unavailable, such as trying a new pricing strategy they’ve never tried before. In these scenarios, let’s do an experiment.
We follow a methodology where we take a small sample of the entire population that will be affected by the decision and introduce a new characteristic to them. We compare the observed results of this test group with a control group that was not introduced to the feature and understand whether this particular feature can be beneficial for consumers and the company. Using this specific methodology, we evaluate which of our hypotheses will be most valuable and implement the same. That’s experimentation in a nutshell for you.
This three-part article series will cover what experimentation is, why you should embrace it, elements of a successful experiment, common reasons for experiment failure, and some behavioral biases that affect results of the experiment.
Related: Transform your business by encouraging experimentation and change
Why experiment?
The two obvious reasons for conducting experiments are hypothesis testing and causality testing:
Hypothesis testing:
Humans usually make decisions based on feelings and intuitions. Data analytics is the antithesis of supporting data-driven decision making. But not all data is created equal. You will find yourself in situations where you believe that specific changes to the feature can increase your primary metric (such as growth or revenue). The hypothesis may seem reasonable to you and your colleagues, but success is not guaranteed since you have no supporting data. In this situation, experimentation is that friend who can provide you with a data-driven answer that can validate (or invalidate) your hypothesis.
To demonstrate causation:
Correlation versus causation is a hot topic in data analysis. Two or more variables are considered related in a statistical context if the values of one variable increase or decrease as the value of one variable changes. This change can have two cases:
Correlation is a statistical measure (expressed as a value between -1 and 1) that describes the magnitude and direction of a relationship between two or more variables. However, a correlation between variables does not automatically mean that the change in one variable is the cause of the difference in the value of the other variable.
causation indicates that the change in one variable results from the changes in another variable, that is to say, that there is a cause-effect relationship between the two variables.
Theoretically, the difference between correlation and causation is easy to identify. However, in practice it is not easy. Randomized experiments help differentiate between these two realities to find truly causal effects. Randomized experiments are the norm in the real world to understand whether a specific change can create a difference in outcome. For example, a randomized controlled clinical trial establishing the effectiveness of a pill helps confirm that the effect is the result of the intervention and not something else.
They are not as resource-intensive as real-world experiments:
Digital experiments are not resource intensive compared to offline experiments. It doesn’t need any additional funding or arrangements needed for real-world experiments. You don’t need to recruit participants or tell users that they are part of an experiment! So how exactly is it different from data analytics?
Related: To be successful, you must continuously evolve your business
What makes experiments different from analytics?
The data source for analysis is the fundamental element that differentiates experimentation from analysis. There are typically two ways to obtain data for quantitative analysis:
historical: Historical data includes data stored by the company in its data warehouses about what has happened in the past, which helps to understand how users behave on your platform. Historical data helps perform various analyses, including user behavior and identifying customer segments.
experimental: Experiments help you validate business hypotheses, as a new change idea will not have essential historical data to validate the change. Experiments could be conducted to observe user responses to an application change or feature addition and compare this to the behavior of the control group.
Experimentation can be your friend and a business enabler, which is a widely discussed and commonly used process, but one that is usually not executed without fallacies. The next post in this series will cover the key elements that define a successful experiment and the four common reasons for experiment failures.