Are you still basing most of your marketing decisions on the ‘Highest paid person’s opinion’? It’s time to hold yourself to higher decision-making standards instead of just winging it. Here’s how.
Adobe Stock: Don't just listen to the Hippos
Every few decades, someone decides the food pyramid is wrong. Fat’s the enemy, or maybe sugar, or maybe bread, or maybe red meat? Or maybe red meat and fat are good now, but sugar is still bad. I don’t know, I can’t keep up.
This year, RFK Jr has taken a swing at it, flipping long-standing nutritional advice on its head with the confidence of a man who believes consensus is a personality flaw.
Whether he’s right or wrong isn’t the interesting bit. It’s how we decide what is right or wrong as we collect more evidence. Nutrition and medicine still rely on a scientific hierarchy of evidence.
This should embarrass marketers because we make most of our decisions based on Hippos (the 'Highest paid person’s opinion), not evidence.
Advertisement
In a past life, before I became professionally irritating to marketers, I trained as a doctor at UCL. One of the first things they teach you is how decisions get made. Not just what to do, but how certain you can be that doing it is the right call. The concept that underpins all of that is the hierarchy of evidence.
At the bottom sits expert opinion. At the top sit systematic reviews of high-quality studies. The higher you climb, the harder the work gets, but the easier it is to sleep at night if something goes wrong.
Marketing decisions rarely carry life-or-death consequences. But I believe our work should be taken seriously, and better things will happen when it is (like getting budget, for example). So we should hold ourselves to high decision-making standards instead of just winging it.
This might sound like the type of blue-sky guff only available to those who can afford the very most expensive McKinsey consultant, but I promise you it’s really not that complicated. And comes with the added benefit of not letting click-through rates and Zuck-bucks (cash from Meta) dictate your life.
I’ve dug out my old textbooks and translated the hierarchy of evidence for marketers in the hope we can finally move past the Hippo.
The outcome we care about in marketing is simple enough: behaviour change and the profit that follows. The confusion stems from how confidently we claim credit for causing either.
At the bottom of the pile sits expert opinion. Frameworks, trend decks, keynotes, confident TikTokers, white papers with suspiciously few footnotes, and loud marketing wankers on LinkedIn [Editor’s note: great to finally see some humility, Andrew]. Marketing is absolutely saturated with this stuff. It’s not totally useless, but it’s dangerously seductive. There’s no method, no data, and no way of knowing whether the person speaking is right or just loud. It’s great for early-stage thinking, killing a Tuesday afternoon or if you run out of toilet paper.
One step closer to better decision-making is the case report. A single campaign, a set of results, and a story about why it worked. This is where most award papers live, along with agency case studies, and where most marketing conference talks reach. These are genuinely valuable for inspiration. They show what might be possible. What they don’t do is prove causality. One example, no matter how glossy, does not a strategy make.
I’m a card-carrying IPA and Effie-lover. The weak case studies in these databanks (when they are presented alone) sit here. The strong ones incorporate some of the evidence higher up the Pyramid of marketing fuckery (Pomf – an alternative naming for Home, you chose which to use, I don’t care).
Advertisement
Things start to get trickier when we move into comparisons. Case-control studies thinking is everywhere in marketing, often without people realising it. You look at buyers versus non-buyers. Exposed versus non-exposed. You compare differences and tell a story about what caused them. It’s cheap, fast, and incredibly tempting. It’s also where marketing logic frequently goes to die. Without proper experimentation (randomization and control), you have no idea whether the difference you’re seeing was caused by your activity or is even related. Rosser Reeves built an empire on this kind of reasoning. It didn’t end well (Rosser Reeves Fallacy). Case-control studies are great for convincing your boss you did a good job if you’re 100% sure they are stupid.
Cohort studies are a step up, and marketing has some genuinely strong examples here. Tracking the same people, households, or brands over time gives you a much better sense of how growth actually happens. Purchase panels, long-term brand tracking, and work like the IPA’s incredible Long and Short of It all live here. My new System1 and Effie book, coming out at the end of January, squarely falls into this category. Heck, even Ehrenberg-Bass regularly uses this level of evidence. It’s incredibly useful for understanding patterns and trajectories. What it still struggles to do is prove that a specific action caused a specific outcome. Yes, it’s interesting that penetration always goes up with brand size, but can you claim that’s why the brand grew? No, you can’t. I will double-lock my door in case Sharpy Boy decides to read this.
If you want to get serious about causality, you need Randomised Controlled Trials (RCTs). Proper ones. Not things pretending to be them, like Google and Meta were recently caught doing. This is where you deliberately hold everything constant except the thing you’re testing. Good A/B tests. Clean geo-experiments. Platform lift studies done properly. These are gold when it comes to justifying tactical decisions and should be far more common, as they do really show causation. Their only real limitation is time. They’re excellent at short-term effects but much weaker at capturing scale and long-term ones. Goboony’s geo-experiment is a good example of this or System1 and TikTok’s The Long and the Short (Form) Of It, which is a collection of decent brand and conversion lift studies.
Also, whilst we are at it, good creative measurement is technically RCTs. This isn’t me exposing my proverbials, rather it’s useful to think like this as it helps marketers choose whether and when to bother with ad testing. System1 always tests creative in the same way (our core method doesn’t change per test) and always with a nationally representative set of people in each market, although we do add on other audiences and questions on top when needed. This means these are actually controlled tests and can be compared. Watch out for measurements like this where the benchmarks are made up of random collections of sample groups, and the survey method never changes, which technically relegates the evidence to a case-control study.
Above all that sits the meta-analysis, which is rare but the cream of practical marketing decision-making. This is where you pool results across many studies and look for consistent effects, while also exploring what changes by category, channel, or context. This is where executional truths start to emerge. It’s the closest marketing gets to saying “on average, this works”. Think work that aggregates econometric evidence or systematically proves the role of creativity in driving profit, like Paul Dyson’s work. Or Essence MediaCom’s Profit Ability 2.0, again combining MMM to share executional facts about media channels. There aren’t many other examples of them, which tells you how hard it is.
Advertisement
At the very top is the systematic review. A pre-specified, transparent trawl of all relevant high-quality research on a question, followed by a no BS honest summary. This is where marketing laws live. They’re often broad, sometimes frustratingly so, but they’re the closest thing we have to solid ground to build strategy from. If you want an example, you have to dip into academia, like this journal article that concludes positive emotional advertising outperforms negative.
None of this means you should stop listening to experience, intuition, or your nearest Hippo. Good marketers absolutely have instincts. But the best marketers are aware that the very flatly termed “insights” actually has a rainbow of quality behind it that should inform these instincts and Hippos.
The food pyramid may get flipped every few years. The hierarchy of evidence doesn’t. And if marketers want to be taken seriously when budgets tighten and scrutiny increases, we could do a lot worse than borrowing the decision-making diets of professions where guessing isn’t an option.
At the very least, it gives you a polite, structured way to tell your Hippo to take a hike.