2. When the risk of not launching is high:

You are working at an established company selling breakfast foods. A rival company’s new cereal product is starting to dominate the cereal part of the breakfast foods market, and your team decides to run new marketing around its cereals. You are running an experiment to see whether this marketing campaign would regain control of the cereal part of the breakfast food market. In this case, the cost of being wrong is low because the cereal market is a small part of the breakfast foods market.

Based on these examples, we would say it is relatively more risky for a start-up to put all its bets on a moon-shot project, compared to a more established company. Risk should be based on your key metrics. If your experiment is trying to impact customer churn, for example, you want to think through how much of your customer base you are willing to risk based on the decision you will make based on the experiment.

From a statistics (frequentist) perspective, we can describe these two risks from making the wrong decision as False Positive Errors and False Negative Errors, respectively.

- A False Positive Error is when you conclude not to launch treatment, but you should have.
- A False Negative Error is when you conclude to launch but you should not have.

How big False Positive or False Negative errors are varies from one context to another (see examples above). But once we know how big the risks are, we need to decide how likely we are willing to make these wrong decisions

Based on these examples, we would say it is relatively more risky for a start-up to put all its bets on a moon-shot project, compared to a more established company. Risk should be based on your key metrics. If your experiment is trying to impact customer churn, for example, you want to think through how much of your customer base you are willing to risk based on the decision you will make based on the experiment.

From a statistics (frequentist) perspective, we can describe these two risks from making the wrong decision as False Positive Errors and False Negative Errors, respectively. A False Positive Error is when you conclude not to launch treatment, but you should have. A False Negative Error is when you conclude to launch but you should not have. How big False Positive or False Negative errors are varies from one context to another (see examples above). But once we know how big the risks are, we need to decide how likely we are willing to make these wrong decisions

Yes, there is always a chance that you make a wrong decision because noise in your data may have you draw the wrong conclusion. Deciding how likely you are to make these wrong decisions in the experiment reflects how risk averse you are. The higher the rate, the more risk-loving you are.

The lower these errors are for the experiment, the higher confidence we will need from our experiment to have us decide to launch. There are established rules of thumb for False Positive and False Negative Error rates:

• False Negative Rates are 20%: this means that 20% of the time when treatment and control are different, we will incorrectly conclude that they are the same
  
  

• False Positive Rates are 5%: this means that 5% of the time when the treatment and control are no different, we will incorrectly conclude from the experiment they are different.

Today we showed how to translate the business risks of making decisions into statistical concepts: the False Positive Rate and False Negative Rate. Determining these rates when we design our experiment forces us to think through how much risk we are willing to take on when we make the decision.

These risks are unavoidable because our data are noisy. But there are strategies we can use to remove some noise from our data before we thinking through the risk. Our next blog post will cover how to reduce the noise in your data and make more precise decisions.