Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
The Paramark Playbook: Scaling Success with Marketing Science | Pranav Piyush
36 Plays4 months ago
SUBSCRIBE TO LEARN FROM PAID MARKETING EXPERTS 🔔
The Efficient Spend Podcast helps start-ups turn media spend into revenue. Learn how the world's top marketers are managing their media mix to drive growth!
In this episode of the Efficient Spend Podcast, Pranav Piyush, Co-Founder of Paramark, shares his journey from consulting and product management to becoming a founder, with a focus on optimizing media mix and marketing spend. He discusses key lessons from his time at PayPal, Dropbox, and Invoice2go, touching on topics like conversion rate optimization, attribution challenges, and incrementality.
About the Host: Paul is a paid marketing leader with 7+ years of experience optimizing marketing spend at venture-backed startups. He's driven over $100 million in revenue through paid media and is passionate about helping startups deploy marketing dollars to drive growth.
About the Guest: Pranav Piyush is the Co-Founder of Paramark, a leading media mix optimization platform. As an expert in performance marketing and data-driven growth strategies, he has helped companies like Dropbox, PayPal, and Invoice2go scale through innovative experimentation and manage multi-million-dollar marketing budgets.
VISIT OUR WEBSITE: https://www.efficientspend.com/
CONNECT WITH PAUL: https://www.linkedin.com/in/paulkovalski/
CONNECT WITH PRANAV: https://www.linkedin.com/in/pranavp/
EPISODE LINKS:
https://www.groupm.com/about/
https://marketingplatform.google.com/about/search-ads-360/
https://www.google.com/partners/
https://about.rogers.com/?icid=R_COR_JOR_OB4PMA
https://www.marketme.co/
https://themindfulsteward.com/categories/
https://www.healthline.com/nutrition/omega-3-supplement-guide
https://www.healthline.com/nutrition/magnesium-supplements
Recommended
Transcript
Aggressive Budgeting for Marketing Success
00:00:00
Speaker
The best CMOs and marketing teams are not just trying to hit their goals, they're trying to beat them. So that's the first thing. It's like get into a more aggressive stance with your budget of how do you beat your forecast every month and every quarter. That's when CEOs and CFOs are really going to start to unlock some additional purses.
00:00:27
Speaker
For now, welcome to the show. Thanks for having me. Super excited to be here and yeah, looking forward
From Consulting to Marketing Entrepreneur
00:00:32
Speaker
to this. I'm very excited to chat all things media mix optimization with you and you have a really interesting career background and trajectory kind of starting in consulting and product management and then moving to marketing and then, you know, moving to to being a ah founder.
00:00:49
Speaker
um Could you just contextualize for for folks your experience more from the perspective of optimizing a marketing mix
Conversion Optimization Experiences at PayPal and Dropbox
00:00:57
Speaker
and marketing spend? Yeah, totally. So it was a winding path into marketing, right? So CS undergrad.
00:01:05
Speaker
loved databases, loved SQL. I was probably the only technical person in my consulting firm that I was at. And then, you know, I was surprisingly, I was a Salesforce admin for my consulting company. Like the only person who knew who was, you know, that's how it is.
00:01:22
Speaker
But fast forward to, you know I spent a bunch of time at PayPal doing more sort of new business incubation. Dropbox was very similar. I was part of the initial growth team for Dropbox business. So that meant we were trying to prove out, is there a business there? Because 2013, Dropbox is very sort of consumer, prosumer, you installed on your mobile phone, you installed on your personal device.
00:01:45
Speaker
And so I worked really closely with the initial advertising buyers at Dropbox. They were sitting there doing all this experimentation on Google.
00:01:56
Speaker
And on you know probably a bunch of other channels, LinkedIn wasn't really a big thing back then in terms of ad spend. you know Google was like the big first step, brand, non-brand, all of that stuff, competitors. And I was sitting right next to that team, literally right next to that individual, optimizing the landing pages and the checkout flows.
00:02:17
Speaker
and running a whole bunch of A-B tests and then looking at our you know main Dropbox business homepage and figuring out what should we be saying there? How do we get more people to convert? How do we increase the trial start rate? How do we increase the trial conversion rate?
00:02:31
Speaker
So my journey with marketing actually probably started there and understanding the conversion rate optimization sort of, you know, game and working very hand in hand with the media team. So every month we were sort of, you know, being asked to present to the CEO, the chief business officer, how are things going? Can we become more efficient? Is there, there, there, right? And in terms of scaling that business, in addition to all the good stuff that was happening organically and product led and all that fun stuff.
Performance Marketing at invoice2go
00:03:02
Speaker
Long story short after that, spent some time at Dropbox doing other things as well, other than sort of growing the Dropbox business team. But it piqued my interest in terms of, OK, what is that person who's sitting right next to me doing? And shout out to will Will Wong. who went on to do amazing things and is now I think an operating partner at Andreessen helping other businesses kind of grow. Took on a GM role after Dropbox and that meant something very different. That was sort of bringing a completely new product to life at Magento and involved sales, involved marketing, involved product, involved you know post sales, customer success, support. It's very broad.
00:03:46
Speaker
And I wasn't directly sort of working on optimizing media mixes there, but I was working with every single team to figure out how do we build this business. And that obviously involved figuring out marketing strategy, but also marketing execution. um Fast forward. I got a call from one of my former bosses at PayPal who was like, Hey, I know you always wanted to kind of go into marketing. So I told him one day, like, Hey, one day I would love to run run marketing. And he basically offered me that job at this company called invoice to go. And they got bought by bill dot.com.
00:04:16
Speaker
in 2021. And so that was sort of the first time when I was looking at 10 million plus in just performance marketing spend on mobile across Apple, Google, you know meta, Twitter, affiliates, everything that you could possibly try.
00:04:35
Speaker
And you know I was sort of overseeing a really smart performance marketer, and a growth team, and a brand team, and a lifecycle team. And so my job there was like essentially figuring out, OK, where do we place our bets in terms of how we're going to grow? And this was 2020, 2021, so right about the same time when all the stuff on mobile was kind of going haywire.
00:04:57
Speaker
And that kind of got me into, you know, thinking about the problem much more deeply and we can talk more about it, but that's the context, right? So early 2010s, much more just conversion rate optimization type of person, and then closer to 2020.
00:05:16
Speaker
We're getting more involved with all the complications of attribution and measurement. I remember at Dropbox, we were doing a bunch of both first touch and last touch. We had built our own internal DIY sort of tagging on the website based on UTM codes and what have you. And so, you know, I was the one who was writing all the SQL queries to do all that work because, you know, we were a scrappy young startup back then and you don't, you don't have a whole bunch of analytical support.
00:05:44
Speaker
So very familiar with the old way of doing things, the new way of doing things, and then eventually you started a business around it. The new way of of doing things is still one that I think folks are slow to adopt understandably. And there's a lot to to say around how are you optimizing a marketing mix to incremental conversions? You mentioned a couple of your experiences there. Was there any kind of aha moment you had throughout that that you thought, wow, this, this system is inherently flawed or broken?
00:06:21
Speaker
You know, I was very naive early in my career. And so I just kind of went with the flow of first touch, last touch type of methodology, mostly because I was so sort of bottom of the funnel focused. I was focused on optimizing the website and the checkout flow and the onboarding flow. So I wasn't really thinking about how media works, how marketing works. That's your world. Your world is very focused on let's move this metric. Let's move conversion rate up or whatever it is.
00:06:47
Speaker
but One thing that was really interesting to me was we ran an experiment on one of those, I think it was the homepage of you know the the product that I was working on back then, this is Dropbox. And I remember changing the call to action from something like you know start free trial to try it free. Super simple. 17% increase in trial starts after that. And I was like, whoa, what is going on there?
00:07:12
Speaker
And, you know, everyone was like super excited. I was super excited. I was like, yay. I'm like, you know, this is amazing. But then you realize two months pass and you look at the trial conversion lift. Did you see a lift there?
00:07:28
Speaker
And the left was much, much, much lower compared to the 17% that we were all celebrating. So that was kind of an interesting aha moment for me is like, hey, are we looking at the right metric when we're running these experiments? And are we giving it enough time for the cohorts to mature to see if there's true incremental gains for the business and not just the immediate metric that is very easy to track and call a win on?
Post-iOS Tracking Challenges in Marketing
00:07:56
Speaker
That was one aha for me personally. The second was yeah you're working in in a growth team, and this team's or this experiment's got a 5% lift. And that team and that you know experiment has a 7% lift. And that team has like a 12% lift. You get all of these. like And then the finance team's like, ah you guys, the business isn't growing 5% plus 10% plus 7%. So what you guys are all reporting, guys and gals, um doesn't add up.
00:08:27
Speaker
So that was another aha moment for me of, are we seeing false positives? And, you know, I didn't know the statistical terms back then. I understood statistics, but we weren't sort of going super deep. So that was another aha moment. The last one was, you know, at invoice to go when we were doing all this work on mobile.
00:08:47
Speaker
And the iOS stuff was starting to happen. Like, okay, how are we going to measure anything anymore? um And that was a very clear aha moment for me. It was like, okay, we need to find a different way of measuring the impact of all of these dollars. So it's just like staring me in the face, given that literally one month we stop all the tracking.
00:09:06
Speaker
And we have to figure out you know pre and post, how are we going to run the business? And you know that opened my eyes in a big way to like, okay, what is going on and what do we have to do? So being in the hot seat, being in the practitioner seat, right that was obviously a very big aha. And that kind of sent me down the route of incrementality and MMM and all that fun stuff. you know Frankly, when when you have a little bit of a GM type of a background where you actually run businesses, you get tuned a certain way, which is much more, I would say at the high level, right?
Pitfalls of Last Touch Attribution
00:09:38
Speaker
If you don't see a 5%, 10% lift in your new sales, what what are we talking about here? Um, that makes a lot of sense. And yeah, from, from my perspective, it, it really hit me with, with iOS 14.5, where a majority of our mix was deployed to iOS based install campaigns that were working really well. And then that metric changed over overnight.
00:10:03
Speaker
I ah also remember we were testing a app network and it was one of those, you know, dollar 50 CPMs, really, really high impression volume. And we were concerned about the quality of the impressions. We turned them off and nothing changed in our core data. And then we were like, Oh, okay, it makes sense now.
00:10:24
Speaker
What I want to ask from from your experience you know managing media mixes at scale, let's kind of extrapolate out what are the implications of over-relying on Last Touch and some of this, I like that you call it tracking and not and at attribution.
00:10:43
Speaker
But in terms of where dollars get deployed across a media mix, because what I see is that what ends up happening is that certain long-term brand or long-term impact bets might get ignored.
00:10:59
Speaker
things of that nature. ah But from your perspective, what do you see as the as the challenges there in terms of um where media mixes can tend to skew and what the risk in that is? So for for most marketers, when they're just starting out and they're early in their career, this is their first marketing job where they're dealing with a serious amount of money and budget, right? There's a little bit of imposter syndrome, like, oh shit, I've never done this before.
00:11:26
Speaker
There's a little bit of like, I need to be able to prove the impact. I need to have a spreadsheet that I can show to my boss or to the finance team about exactly what happened. It's natural. The first time any sales or marketing or any sort of person working with a big budget is going to have that sort of concern. So the first thing that happens is you do things that are easy to measure. You do things regardless of whether they are actually going to move the the business in a positive way, you do the things that will look good in a spreadsheet. And that tends to be the things that are bottom of the funnel, direct response, click-inducing things. That's it. So the whole world just indexes around that. Now, the fun part is everybody's doing the same thing. So everybody's going after bottom of funnel, direct response, click-inducing things. So now the costs start going up.
00:12:16
Speaker
because everything is an auction. And so now, whether it's meta, whether it's LinkedIn, whether it's you know Google, your costs are constantly going up. And you're constantly trying to squeeze out more juice from the same bottom of the funnel direct response, click-inducing thing for your spreadsheet.
00:12:31
Speaker
It's a road to nowhere. So that's the first thing that happens. The you know the second thing that i'm I get annoyed at, and marketers do this to ourselves, we will say things like, there are certain things that have long-term impact. There are things that have short-term impact. This perspective comes from a personalized point of view, not an aggregate point of view. So what do I mean by this?
00:12:55
Speaker
We think that oh when we see an ad, we don't go out and buy something right off the bat. and so We may buy something six months from now. right so From our perspective, that looks like a long-term thing. The reality is when you take a probabilistic view of the world and you say that there are thousands of people out there, there are thousands of pranav out there,
00:13:15
Speaker
and all of them are exposed to that same ad, some of them are going to get influenced by right away because they are in market or because they have seen something else six months ago. So they are in market right now. And so this idea that certain things work in the long term is actually flawed in a big way.
00:13:34
Speaker
Everything works in the short term. If it doesn't work in the short term, it will never work in the long term. To maybe ah counter or maybe even compliment that that point, I do strongly believe in the importance of a diversified media mix. right There are advantages and disadvantages to to different channels, but holistically, I think as a brand scales, they should be adding several channels that have you know complement each other in interesting ways. One of the outcomes of me focusing more on MMM and incrementality has been that I've started to think about that relationship a little bit more deeply.
00:14:16
Speaker
How do we think about TV budget in conjunction with non-brand or brand page search, right? Things like that. What is your perspective? And perhaps it changes as a brand scales in different stages, but what is your perspective on what the optimal components of a high functioning media mix could look like?
High-Functioning Media Mix Components
00:14:38
Speaker
Oh, I mean, I hate benchmarks. the The reason I say that, I don't think it's it's okay to have a cookie cutter answer, so right? Stage of business, volume of spend, type of business, type of vertical would all change the answer to that question. So I don't think that there is one answer. If I'm starting off, you know i can I can say with confidence, there are brands that have started from zero to one who have relied and done everything just on Google search and done it really well.
00:15:08
Speaker
and there are brands who have gone zero to one right and i literally mean zero dollars in revenue to like a million dollars in revenue i'm just giving you like two hypothetical examples who have done everything just off of linkedin organic social and you can look at those and go like wait what like which one's better like there's no answer to that there's no such thing as the optimal thing now Long-term, let me sort of maybe add something to that. What is the most important thing is where does your audience hang out?
00:15:37
Speaker
and What is the message and the creative that is going to resonate with them? And if you can figure that out, then I think that's like 90% of the battle. Then there's a question of like, you know, are you going for people who have well-established patterns on how they do research, right? That's the search sort of methodology. Or are you you know going after people from a different lens, which is Hey, I think a lot of people hang out on LinkedIn and even though they're not intent, they're not searching, they're actually right there and very open to consume content. So, you know, that's like getting into more of like human psychology and both will work for you. Now, when you're at scale, you've got 50 million in revenue. You're spending 10 million in marketing. I would imagine.
00:16:30
Speaker
that the ultimate goal for a marketing leader and a finance leader is predictable growth. I want to be able to forecast how I'm going to go from 50 million in sales to 60 million in sales or 70 million in sales. And then it's a question of you need to be able to hit goals in the short term, right? I need to hit my in-month, in-quarter goal. And I also can't take my eye off of what happens in the quarter are after that.
00:16:53
Speaker
and the quarter after that. So the way to think about then optimizing your mix is some amount of budget that's dedicated to people who are bottom of funnel, intent-based. I'm actively searching for a thing. If your market is big enough, if you're actually sort of business strategy is good, there should be many of those people. If nobody's searching for what you're looking for or what you're selling, you have a more fundamental sort of business problem.
00:17:18
Speaker
And the second is then how do you go after people who are not searching for anything right now, but you know that they're probably going to search for something in six months. And if you can be the top of mind for those people when they do the search, then your CPMs and your CPCs are just all going to go down. So what is the ideal mix? I actually think it's 50-50 at scale, where you take 50% of your budget and you're focused on the yeah I hate these terms, but I will use them because there's no better term. ah Brand marketing, top of funnel, awareness, whatever you want to call it. and Then 50 percent, that's very bottom of the funnel that's focused on hitting and capturing folks who are in market right now for and they're searching for something and that is not your brand. I think that's that's a ah fair answer. There could be um a lot of folks listening to this that are
00:18:14
Speaker
managing a multi-channel media mix, much to your point, are spending several million dollars, are not currently using MMM or incrementality, or maybe they're just starting.
The Value of Marketing Experimentation
00:18:27
Speaker
And I would, I would hypothesize that more likely than not, a majority of those folks are not 50-50, but more like 80-20 towards performance. And please, you you know, keep me honest here with the the CMOs and other marketers that that you talk to, would love your feedback on, is that accurate? And if so, for marketers that are in that situation where you know they've gotten the
00:18:54
Speaker
They're hitting their performance goals. Maybe they're hitting their their you know revenue goals, their their CAC targets, what have you, but they have this thought that maybe I should reallocate this 20% away from Google paid search into a massive brand partnership or or something along those lines. What would you say to folks in that situation?
00:19:15
Speaker
I mean, listen, I think the the best CMOs and marketing teams are not just trying to hit their goals. They're trying to beat them. So that's the first thing. It's like, get into a more aggressive stance with your budget of how do you beat your forecast every month and every quarter? That's when CEOs and CFOs are really going to start to unlock some additional purses. So that's one. Don't be in a defensive mindset. Be in an offensive mindset. The second thing is,
00:19:44
Speaker
Um, I come back to experimentation. You have to set aside 10 to 20% of your budget on experimentation to answer exactly that question. Should I take maybe not a hundred percent of my brand paid search spend, and maybe I should just take 10% of that and shut it off and allocate to something else. And all the statisticians and data scientists listening in, if they are, are going to go like.
00:20:09
Speaker
hey, you're not going to get enough signal, blah, blah, blah, and they're going to get on me from a statistical lens. I appreciate that. I understand that. And you're right. But at the same time, you have to take a pragmatic approach, the one that actually drives action and result in the business. And if you set aside 10% or 20% of your total budget for experimentation, and you can, even if you're doing quasi experiments, and they're not perfectly clean, but if you can run in a quasi experiment three times in a row and replicate it,
00:20:38
Speaker
It's probably worth something. And so that's what I would say. Take that, reduce the spend, put it on a brand sponsorship, maybe sponsor a podcast, whatever, right? And see the effect on the metric that you care about. Learn enough statistics where you can start to do that. You don't have to be a data scientist to do this. That's a misconception.
00:20:58
Speaker
Yeah. Can you think of any kind of noteworthy experiments that that you've ran in the past that were kind of similar in that, that had that type of positive outcome? I mean, all the time we've been doing a lot of experiments with our customers. And, you know, I can't name names here obviously, but the big one, the big sort of, aha, we were literally talking to that customer today is they had this hypothesis.
00:21:19
Speaker
about a certain type of go-to-market motion, this is in international markets, that would have a very positive effect on their net new sales and net new customers. And the interesting part is like a very complicated go-to-market motion with not just marketing but also sales and very sort of you know locally sort of optimized in certain regions.
00:21:41
Speaker
And the control was none of that. The control was that broad-based marketing efforts and broad-based sales efforts in other parts of the world. And that was a massive you know risk for the business to take and say that, hey, we're going to put all our eggs in this like one city and see what that does to the business. But that was the only thing that they were doing. right They did have always-on media running for the rest of the region. and So that was a big sort of step where the top-down leadership was like, this is a strategy we must test. This is going to work. And it wasn't a massive bet in the sense of like the total budget, but it was a risk. That came from top-down leadership saying, we're going to treat everything as an experiment.
00:22:22
Speaker
So that's where the culture comes in, right? Where you can't do this just bottoms up. You're a CMO, you're a VP of marketing, your finance team has to be aligned that you have to set aside a certain amount of budget for this type of testing. If you expect the same return from every single dollar, you're not going to find the big winners. So that's one example. Another example, I mean, Google paid search is such a good one, right? Like it's such a good, for for every measurement company, that's like the first experiment that you should be pitching to your customers.
00:22:51
Speaker
And it works every time. Hey, reduce your spend and see if it does anything. Nine out of 10 times, you can see that you're overspending. That's a no risk test at this point. I think that these that these problems become you know accelerated as a brand builds more of ah a foundation. i'll give ah I'll give a counter though, Paul. So there was a you know there is a business. They're very risk averse on taking that spend away because they're worried about the competition.
00:23:21
Speaker
And they're worried that you know they'd other people bid on their branded keywords. And from their perspective, it's a reputational thing of if a C-suite prospect is searching for their brand and a competitor is showing up in the first link, even just the look of that is not something that from a brand safety perspective, it's like ah it's like a brand reputation perspective type of decision.
00:23:45
Speaker
rather than an incrementality-based decision. So I can appreciate that. There's a nuance there, but you have to have the conversation that, yes, even though it's not incremental, we're doing it for this reason.
00:23:58
Speaker
the brand differentiation as it relates to you know building more of these partnerships, um no better word for it, brand marketing, that can be pretty pronounced too, right? If you take two different brands and one of them has a partnership with a well-known respected influencer, celebrity, et cetera, there's probably a ah impact on on on that. 100%. Long-term. I mean, look at what what's his name, Ryan Reynolds, right? Has achieved. It's the opposite for him. He's like going out there and taking his platform to all of these businesses and having a massive impact.
00:24:38
Speaker
The struggle that I find with some of those larger influencer, you know, promotions is, is it an authentic relationship or is it a, you know, it's the same as like spending money on meta with like no targeting and bad creative. How do you think about that too? Actually like, you know, this is something I've been thinking about a little bit more trading off paid dollars for organic social dollars. You had a post the other day about organic social. You know, if you are a CMO and you're like, do I spend the next hundred K on Facebook or do I hire a badass content marketer? How do you think about something like that? Yeah, it's a good question, isn't it? I was thinking about it even after I wrote that post. Listen, I would definitely invest in organic social regardless of the size of the brand. There are some people who say like, why do it? You get more bang for your buck on paid, you get more reach.
00:25:27
Speaker
That's probably true for 99% of brands. the It comes back to experimentation. I think organic social is a great testing ground for whatever you're going to do unpaid. So for me, I would secure the organic social sort of foundation first.
00:25:42
Speaker
and use the lessons from there to then amplify on paid. Yeah, that's my take. I will also take the counter of that, which is if you have a great social person and you put them on paid social, they will probably crush it. The separation of like organic and paid social and there's like a separate team that's going to manage paid social, to me, it's like a dumb idea. ah You should have your organic social superstars doing your paid social content.
00:26:07
Speaker
they know how to create engagement. I will say as somebody who considers themselves to be really good at paid social, organic is hard. Organic is not easy. So I want to i want to kind of transition here to talk a little bit about Paramount. To ask you a very pointed question, the MMM space has gotten really competitive.
Marketing Mix Modeling Market Dynamics
00:26:24
Speaker
I just went through an evaluation process where I looked at eight different brands um and we ended up selecting one, right? We're going through onboarding right now. And even throughout that process, there were you know more that came into the woodwork that I was like, should we take a call with these guys? right It's a competitive space. So why play in this space? Why why start this this company? you know It's a great question. I come back to a very simple answer. And I ask every customer this, how confident are you about hitting your goals next quarter?
00:26:56
Speaker
Right. The answer is very revealing. I asked them, where are you going to place your next bet? The answer is very revealing. So why should more companies exist in this category? Because nobody has yet produced a solution that gives you confident answers without drowning you in technical jaron jargon.
00:27:15
Speaker
or charging you a million dollars, or you having to spin up your own data science team or marketing analytics team to reconcile everything that they're saying so that you can actually trust it. I'm not going to put you in the spot and say, who did you pick? I tell this to every CMO is like when you are making that evaluation, whether or not you are talking to Powermark, ask very simple questions of after you have onboarded somebody.
00:27:38
Speaker
Has that increased your confidence level to a point where when you show up as a CMO at the QBR, at the annual business review, at the board, that you no longer have anxiety the day before about, oh my God, do I believe my own numbers? One of one of the learnings that I've kind of had throughout this process is what the use cases for MMM, a lot of vendors will overpromise in certain circumstances. For example, I think if when operating at scale, granularity becomes a conversation. We have thousands of campaigns. Can you show me the incrementality for this one thing? It becomes really tricky. As I understand the the differentiators from some of the the larger models, it really comes into kind of the the data that that is input into the into the system, whether that be spend, spend an impression, some combination of that. and also how they're calibrating and how they're they're running experiments. What's your kind of perspective on you know the the things to look for in in MMM, maybe like the must haves, nice to haves, et cetera?
00:28:43
Speaker
I think the first thing that you will see is out of the eight that you evaluated, I'd be surprised if any of them are left in the market in the next three years if they have not added incrementality testing as a way to calibrate your MMM. So that's the first thing that I would actually look for is MMMs are great. They're correlational.
00:29:04
Speaker
There's a lot of people who tell me otherwise. Lots of really smart data scientists and professors will come at me and go like, no, no, no, they can be causal. And let's just say I disagree respectfully. And there might be estimates of causality, but they're not causal.
00:29:20
Speaker
Those are two very different things. To play this out, no, no, no. To say this very specifically, right? It could be spending at scale in our medium X. We increase spend in Facebook and then the MMM picks up some sort of correlation and then attributes an increase in conversions to that Facebook spend, ignoring maybe the foundation of spend that we've already done historically,
00:29:43
Speaker
or a lot of other factors that could lead that cohort to converting. And almost I was thinking about today, it's almost like last touch attribution in a way, if the spend is increased and the conversion happens at that time, it doesn't mean that spend that happened six months, 12 months ago did not impact that at all.
00:30:03
Speaker
I mean, that's one way of thinking about it, right? I come back to like just definitions, right? Because we can all play the semantic games to convince ourselves that no, no, no, it's causal, right? It's like, we can play that game.
00:30:18
Speaker
The reality is, if you think about what does causality mean, like let's let's ask ourselves this question. Everything that we talk about comes from science. It doesn't come from marketing. It comes from science. It comes from biology and physics and chemistry, and some of it comes from economics. How do scientists think about causality?
00:30:36
Speaker
You run experiments. You run randomized controlled trials. And everything else in the scientific community is a way to think about hypotheses, not about proving or disproving hypotheses. So causality, just the word, the the idea of causation, you can't remove that from the concept of experimentation. And therefore, I say that incrementality testing is your best bet. It's the gold standard. Now, the question is, if you don't have that, what do you do?
00:31:06
Speaker
if you don't have that then you have you know estimates of causality and correlation and it's all sort of trend analysis right this is the trend and therefore this is the trend and like yeah those two trends match up but that is not causal if you want to be true to the word and That's our sort of purest perspective now, you know, one thing that I will say is so so that's the first aspect of what you should look for when you're looking at a vendor, right? Are they doing incrementality testing and and mmms together? And I think most people will start doing that if they want to have a long sustainable business because when you don't have that you have doubt and People poking holes at that correlation as well. Did you consider this?
00:31:50
Speaker
And well, we also did that. And it's like you're opening yourself up to a surface of attack that is not needed if you had incrementality testing. It ends all debates. So that's one. The second thing that I would say is how can you show up in front of a CMO and a CFO?
00:32:11
Speaker
So what do I mean by that? Can you make all the stuff actionable that helps you actually beat your forecast for the next quarter? So tying all of the insights to actual action and recommendations is something that you're going to see platforms evolve to do. And if they don't do that, I think that's going to be a missed opportunity in the market. I'm kind of laying out our our playbook here.
00:32:37
Speaker
um The third, which I think is really important, is the level of customer service and the level of customer success. There is, ah you know, some people have the opinion you're going to automate all of this. And I think that's a dangerous assumption.
00:32:53
Speaker
the human in the loop is absolutely going to be needed. Even the gen AI companies have come to the same conclusion. You need a human in the loop. And the reason is that all of these tools are only as good as the data that you feed in.
00:33:07
Speaker
I'll give you a very simple example, right? You have a 10x spike on this day for this metric. Had nothing to do with anything. It's just a bug. No amount of machine learning and AI is going to do, you're going to lead to false correlations if you don't have that human in the loop that's looking at that stuff. So those are three things that come to mind. There's probably a longer thread there, but yeah.
00:33:31
Speaker
you know It's funny, so sometimes we see these these spikes. And I'm sure you know as marketers, we go into our weekly ah performance meeting, and we we look at changes week over week. And we when we do see the conversion spikes or or changes, we immediately try to attribute it to something that we did. There's this like thought of, oh, yeah, well, it's obviously something that happened in paid search. And it's the correlation doesn't equal equal causation argument.
00:33:59
Speaker
um And what you said regarding incrementality testing totally totally agree with that. there's And there's a lot of ah assumptions that that we make as well. um And the automation, i'm I'm fearful of the marketing industry over relying on on some of this stuff because we lose out on a lot of the special sauce that leads to kind of outsized returns. And what I mean by that Is take a channel like facebook for example they have these automated advantage shopping campaigns it's automated targeting automated and optimization there's no thought of incrementality there. There's certain ones where you can even exclude audiences and so they become retargeting campaigns.
00:34:49
Speaker
The CPA will look amazing, but is it incremental? So that's that's obviously a challenge. Yeah, I mean, you you're absolutely right. For any walled garden that doesn't actually have all your first party data, that's a big issue.
00:35:05
Speaker
And even if you did have your first party data, the you know the matching of that data to their own data is often not that great. So, ah you know, Meta just released something yesterday or today about sort of integrations with third party folks and everyone's like riled up. This is going to be awesome. I'm like, no, it's not. Like it's, say you know, let's be honest. Like it's a little bit of.
00:35:26
Speaker
um Yeah, we should talk about that at someday. So, you know, I think the other thing that I was going to say is the week to week stuff. It's also just noise. And we forget that. When you have a week over a week, like 2% change, and you're like, oh, that was strange. No, it wasn't. Go look at all the week over week changes for the last 300 weeks. And you'll see that 1% to 2% changes are very common. it's it's It's pretty interesting when you start to like think broadly about all
Balancing Creativity and Measurement in Marketing
00:35:53
Speaker
of this. The one last thing I would say is,
00:35:55
Speaker
No amount of measurement is still going to solve your growth problems if you don't have great creative. right So at the end of the day, what are we doing here? Measurement is like Robin. Creative is Batman. You want to minimize your effort and energy on measurement while maximizing your effort on on creative. that's That's what I think.
00:36:14
Speaker
Okay. um I guess the the last question um is, yeah, what would you what would you want marketers to to take away from this conversation? Like like one thing that you know CMOs folks should should really start thinking about.
00:36:27
Speaker
If I were in a CMO's role, I would send everybody to Statistics 101 for a week, a year, and ask them to just learn probabilities, understand statistics, and I think that that's a very actionable thing that you can do. This used to be the case in the 80s and the 90s. There were a lot of marketers who understood math, and I think more folks need to understand that. I know, it's kind of weird, but that's what I would say.
00:36:53
Speaker
Just go study it a little bit. Awesome. Thank you so much, Pranav. Thanks for having me, Paul. This was fun.