How Incrementality Testing Can Rescue Your Marketing Spend 💰

What happens when you flip the switch? A guide to incrementality for the modern marketer.

TL;DR:

• The Illusion: Last-click attribution often mistakes existing brand momentum for new growth.

• The Goal: Use incrementality to separate “organic” customers from those who only bought because of your ads.

• A/B Testing: Simple and direct, but can be susceptible to seasonal noise.

• Geo-lifting: Ideal when user-tracking is limited, though requires deep regional data.

• Budget Holdouts: The ultimate reality check, showing exactly what happens when you turn off the taps.

• Causal Inference: The gold standard for accuracy, provided you have the data science muscle to back it up.

• The Takeaway: Instead of leaning on fragile user-level data, we can now use aggregated insights to get a high-fidelity view of what’s actually working and, more importantly, what isn't.

---

Last-click attribution has always been a bit of a comfortable delusion. We tracked the final touchpoint and called it a day, but as privacy walls go up, we’ve had to confront a stubborn truth: knowing a user clicked an ad doesn’t mean the ad was the reason they bought.

Incrementality testing is the “what if” of marketing. If we turned off this channel in Nairobi for a month, what would actually happen to the bottom line? It’s about separating the organic customers from those who only arrived because of your intervention.

Depending on your appetite for risk and the state of your tech stack, there are four main ways to get to the truth.

1. A/B Testing

The classic approach: split your users into a control group (no ads) and a treatment group (ads), then compare.

• The upside: It’s simple, intuitive, and does a great job of isolating a campaign’s direct impact.

• The catch: It can struggle with external noise like seasonality. If you’re a smaller app, getting a statistically significant division of users can be a tall order.

2. Geo-lifting

Here, you’re dividing by map rather than by user, turning the lights on in Nairobi while keeping them off in Accra.

• The upside: It’s a lifesaver when user-level tracking isn’t an option. It’s also relatively quick to roll out for large-scale campaigns.

• The catch: It’s a heavy lift. You need deep data on regional demographics and market size before you start. There’s also the “spillover” risk, marketing in one region often bleeds into the next, muddling your results.

3. Budget Holdouts

Essentially, you keep a slice of your budget in your pocket for specific regions or groups to see how they perform without the extra push.

• The upside: It gives you a very clear picture of what happens when you turn the taps on or off. It’s the ultimate reality check for your spend.

• The catch: It requires significant resources to manage properly, and if you have five other campaigns running simultaneously, the results can get messy quickly.

4. Causal Inference

This is the high-brow version. You use statistical models and “synthetic control groups” to map out what would have happened in a parallel universe where you didn’t run the ads.

• The upside: It’s incredibly reliable and accounts for external factors.

• The catch: It’s complex. Unless you have an advanced data science team on speed dial, it’s a difficult and often costly mountain to climb.

Picking your path

Success here is less about the “best” method and more about the best fit for your current reality.

If you’re running complex cross-channel campaigns and need granular accuracy, the investment in causal inference is usually worth the wait. For those needing a faster, broader read on large-scale regional spend, geo-lifting offers speed, though you’ll sacrifice some precision.

If budget is the primary constraint, holdouts are a grounded, affordable starting point, while A/B testing remains the go-to for simple, single-channel DIY experiments, provided you can live with a bit of statistical noise. It’s about balancing the precision you need against the resources you actually have.

Our Takeaway

Whether you’re fine-tuning your user acquisition strategy or maximising your marketing budget, the ability to accurately measure the incremental impact of your efforts is crucial.

Incremental analysis doesn’t just improve short-term performance; it also provides long-term benefits. Marketers can use it to avoid cannibalising organic traffic and ensure that their paid efforts are generating truly beneficial results.

---

If the universe is this loud about it, you might as well sign up for our App Growth Summit happening in Nairobi 😁

Join Our Community 👥

Come hang with us. Check out the blog, follow along on socials, and stay in the loop as we share the wins, lessons, and experiments shaping growth marketing today. 🤝🏾
PS: The really good stuff? You’ll find that on our TikTok. 👀😂

Got big goals? Good 🤝🏾

Tell us where you are and where you want to be and we’ll help get you there.