How to use Experimentation as a Growth Accelerator

“Experimentation is the least arrogant method of gaining knowledge. The experimenter humbly asks a question of nature.” – Isaac Asimov

One thing that a company should learn to do, that can accelerate its growth many folds is running effective experiments. It is a widely discussed and used process yet widely misunderstood by start-ups.

For companies in their growth stage, experiments have become a part of everyday activities, with different teams, such as Marketing, Product, and Technology, running experiments in siloes. As the same customer is subjected to experiments by the various groups, we don’t get concrete results. It is hard to evaluate the efficacy of each experiment and what the results are. 

The above is just one of the reasons for inconclusive experiments. However, you can make experimentation the key to growth for your start-up. If a company can establish the right experimentation strategy, it will be able to grow in an accelerated manner by recognizing potent growth drivers.

What is Experimentation?

To understand what experimentation is and how it’s different from Analytics, let’s go through how we capture data for an organization. Typically, there are two ways to get data, historical and experimental.

Let’s understand each way with an example of a freemium dating app. In a freemium app, growth is one of the key metrics, a function of the number of free and paid users. Now, assume that 1% of the customers are paid users. If we were to figure out who among the free users looks most similar to paid users and use various methods to persuade them for an upsell, we could drive growth. We can use historical data to assess the behavioral patterns and attributes of paid and free users, and this method is called look-alike modeling.

Let’s take another case from the same freemium app example to understand experimentation. Let’s assume that a free user can view only five profiles unless they convert to a paid subscription, and then they can view more profiles. Once they have viewed five profiles and want to view the sixth one, they come across a paid subscription nudge; this clause is an ‘upsell hook‘. Let’s assume that the company’s upsell hook is a free user purchasing an annual plan. But the company also has an option for a monthly plan. A product manager might wonder, “What is the best-paid plan to present for this upsell hook?”. The product manager may hypothesize and suggest a monthly plan over an annual one, considering the higher user investments. Since this idea is new and there is no historical data available to support the hypothesis, an experiment can help capture sufficient data to determine which plan is better for the company and its growth.

Why Experiment?

There are two significant reasons why companies should try experimentation.

1. Hypothesis Testing – We have already looked at the case of someone who has a strong hypothesis. In the above case, the product manager hypothesized that “if we change the hook from annual to monthly, we will increase our trial rate by 10%.” The hypothesis sounds reasonable because people would not want to invest a lot, and experimentation is required to collect backing data to implement the change
2. To Prove Causality – Let’s continue with the dating app example to understand the second reason. Suppose a model helps predict the chances of an upsell hook based on user activity. For instance, the model suggests that a free user connects their Spotify playlist to their profile between day ten and day thirty; they have a 90% chance of subscribing to a paid plan. In this case, you establish a high correlation between the event and the subscription. Still, you want to institute a causality between event and subscription to make a core function change within your user behavior, like in this case, to insert the playlist connection. Running an experiment helps confirm the relationship involved and understand if people who are likely to be paid subscribers are doing this event or not.

We now understand that the primary reasons for experimentation are hypothesis testing and establishing causation. For almost everything else, analytics would be sufficient.

What are the other reasons for experimentation failure?

Knowing how to optimize the user experience and the journey flows through a series of experiments can improve customer experience and clarify the organization’s business path. So, growth-stage companies should experiment often. But, the challenge lies in doing it the right way, as experiments often are not designed and executed well. Several considerations are required to derive conclusive results from experiments.

This post only mentioned one aspect of why experiments fail: the same customer involved in experiments from multiple departments. However, there are several other reasons why experiments fail, which we will cover further in the next post.

Until then, happy experimentation!