Why Deliveroo is putting incrementality testing back on the menu

For Deliveroo, as an app-first business investing heavily in performance media, we wanted to understand how much of our growth was genuinely being driven by Google’s Universal App Campaigns (UAC) for Installs – and not activity that would have happened anyway. But our incrementality measurement methodology needed to evolve.

The limits of geo-tests

Like many digital-first businesses, we historically relied on geo-based incrementality methods: splitting regions into exposed and control groups to calculate lift. Rigorous in theory, but in practice, increasingly unworkable and impractical.

There were a couple of reasons for this. One: operational complexity. Identifying a ‘clean’ testing window – with no overlapping geo-targeted campaigns that could distort results – was almost impossible. Two: scalability. Only our largest markets could support statistically valid tests. As a result, our last geo-test was three years old, yet still being used to inform budget allocations in a post-Covid market that looked nothing like today’s.

It became clear that a one-size-fits-all incrementality estimate was unlikely to reflect the actual impact in each market, especially for those markets that were newer requiring more proof points to inform budget planning. Relying on outdated, infrequent, or no testing at all risked misallocating spend at a time when agility mattered most.

Putting new methods to the test

So what did we change? Deliveroo explored Google’s user-based Conversion Lift Studies (CLS) as a more agile alternative. By randomizing at the user level, CLS promised scalable insights without the heavy operational lift of geo-tests. But we weren’t willing to adopt it blindly. While user-based CLS has faced industry discussion and some debate over methodology, we wanted to validate its reliability in our own context.

In practice, this meant setting up a multi-cell validation experiment to see whether CLS results aligned closely enough with our in-house geo test – a step necessary before trusting CLS as a scalable alternative.

Here’s how we set it up: We focused on the UK, our most mature market, where we had previously run a geo-test. We split regions with 55% receiving UAC ads (treatment group) and 45% held back (control group). Within that same geo-test treatment group, we ran a user-level split test on Android campaigns in partnership with Google, with 90% of users in treatment and 10% in control.

We also built in safeguards to ensure the results would be robust. For the geo-test, we conducted a power analysis to make sure the test could detect meaningful effects. For the user-based test, Google performed feasibility checks to confirm the design was statistically sound. Meanwhile, our team monitored the media mix across YouTube, Search and Content to ensure spend reflected normal business conditions, rather than an artificial test setup.

This “nested design” wasn’t just about measuring incrementality – it was about testing the testers. By validating the reliability of both methodologies, we could build confidence in the results and scale our approach with certainty.

The outcome: alignment unlocks scale

After the test concluded, we analyzed the results. Despite differences in methodology (CLS being user-randomized, the geo-test being region-randomized), the results from both methods closely matched and the similarity in outcomes built confidence in CLS as a viable alternative to our internal geo-testing.

The study showed Google’s App Campaigns drove a statistically significant uplift in our primary metric, new customers. With a further statistically significant result seen on secondary metrics such as first opens of the app and existing customers placing an order.

From one-off diagnostic to continuous input

This validation meant we could finally leverage CLS as a viable alternative to our internal geo testing approach allowing us to shift incrementality from being a once-every-few-years diagnostic into a repeatable, scalable input for our Marketing teams.

We are excited about the upcoming methodology and product developments to Google’s Conversion Lift further lowering barriers to testing and allowing us to increase testing rigor further globally. Why does that matter? Because incrementality isn’t static. It varies by time, by market, by category, and by improvements you make to media setup and efficiency.

Today, we’ve expanded user-based incrementality studies to multiple markets, while continuing to validate annually with a dual geo/CLS approach.

Learnings for marketers

For us, it’s about a new philosophy: validate quickly, test often, make incrementality central and ensure every pound of spend delivers maximum impact. That’s the Deliveroo story but it also reflects some wider truths and learnings for any senior marketer grappling with measurement:

1. Play the long game: the impact of marketing extends far beyond a single campaign
   We’ve seen first-hand that the effect of marketing often continues well beyond the initial campaign window. When measurement happens too rarely or focuses only on short-term effects, it misses the full picture. Embracing a longer horizon allows brands to make smarter, more confident investment decisions.
2. Invest smarter through measurement: modern methods clarify what truly drives growth
   For us, measurement isn’t a box-ticking exercise – it’s how we make better decisions about where to allocate budget. By modernizing our approach, we gain a clearer view of what’s driving growth and can invest with confidence. This ensures every pound of spend delivers maximum impact.
3. Layer methods for confidence: validating one method against another builds credibility in results
   We validated our Conversion Lift Studies against geo-tests to strengthen the inputs into our Marketing Mix Model. This multi-method approach adds rigor, improves reliability, and helps demonstrate the true value of marketing investments.