Incrementality testing involves running experiments, such as holdout testing, A/B testing, or geo-lift tests, to find the impact of your marketing efforts on sales. For example, let’s say you are running a marketing campaign for your app and want to test its incremental lift on installs.
Group A acts as your control group. It’s the benchmark for installs. These users have not been exposed to your ads. Group B is made up of those who are exposed to your ads. Group A had 100 installs, while Group B had 120 installs. With this information, you can calculate two key insights: incremental lift and incrementality.
Incremental Lift = [Test conversion rate - Control conversion rate]/Control conversion rate * 100
Incrementality = [Test conversion rate - Control conversion rate]/Test conversion rate * 100
Limitations of incrementality testing
While Incrementality Testing is great for insights on the causality of marketing activities, it has the following gaps:
Slow and costly: Incrementality tests can take weeks to run and are expensive. It is not feasible to run these tests for every channel or campaign
Opportunity cost: Let’s say we run a test to establish causality by exposing a new ad to an audience. By withholding the ad from the test group for a certain period, the brand forgoes potential revenue that could have been generated from that group's exposure to the ad.