Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Google Product Manager turned Incrementality Founder | Zachary Epstein
31 Plays6 months ago
SUBSCRIBE TO LEARN FROM PAID MARKETING EXPERTS 🔔
The Efficient Spend Podcast helps start-ups turn media spend into revenue. Learn how the world's top marketers are managing their media mix to drive growth!
In this episode of the Efficient Spend Podcast, Zachary Epstein, founder of Haus and a former Google Product Manager, discusses the alignment of incentives between ad networks and advertisers, the complexities of performance measurement, and the importance of quickly testing and learning from marketing experiments. He shares his insights on the evolution of ad platforms and democratizing access to sophisticated marketing tools.
About the Host: Paul is a paid marketing leader with 7+ years of experience optimizing marketing spend at venture-backed startups. He's driven over $100 million in revenue through paid media and is passionate about helping startups deploy marketing dollars to drive growth.
About the Guest: Zachary Epstein is a marketing analytics expert with over a decade of experience, former product manager at Google and founder of Haus, a company specializing in causal inference analytics and data infrastructure, helping companies optimize their marketing spend through sophisticated experimentation and measurement tools.
VISIT OUR WEBSITE: https://www.efficientspend.com/
CONNECT WITH PAUL: https://www.linkedin.com/in/paulkovalski/
CONNECT WITH ZACHARY: https://www.linkedin.com/in/zach-epstein-b7b10525/
EPISODE LINKS:
https://www.haus.io/about
https://ads.google.com/intl/en_PH/home/
https://www.facebook.com/business/ads
https://www.marketingevolution.com/knowledge-center/incrementality-in-marketing
https://bookdown.org/mike/data_analysis/causal-inference.html
https://support.google.com/google-ads/answer/
https://www.youtube.com/ads/
Recommended
Transcript
Advertiser and Publisher Dynamics
00:00:00
Speaker
Philosophically, things are generally aligned. I mean, advertisers want performance for for their media. Publishers want to give enough performance for their media for you to continue investing with them. Like marketers have choice. You can spend with Google, you can spend with Meta, you can spend with TikTok, you can spend with Programmatic.
00:00:16
Speaker
There's a million places that you can go and try to get some some outcome of growth. But the the challenge is ad platforms have a lot of other incentive. If you're a publisher, you are trying to get your advertisers results. You're trying not to upset your users too much by putting advertising in front of them constantly. Like if you've ever, hey, I've gotten the same ad seven or eight times and it's like, well, they probably have like a tight cap of when they're trying to spend the money and we need to be serving the impressions. and Generally, compounding frequency is more likely to lead to a conversion outcome. But actually, where where things probably deviate the most is that publishers need to monetize the corpus of their inventory.
Ad Network Incentives: Aligned or Not?
00:01:00
Speaker
Zach, welcome to the show. Thank you. Great to be here. I'm really excited to chat with you today. I think this is going to be an important conversation for folks looking to optimize their media mix and also think through incrementality. We'll talk about your experience founding house as well as your experience at Google. I did want to start with kind of a ah bold question, though, because you've spent so much time at at Google. When you think about the relationship with incentives for ad networks and advertisers,
00:01:31
Speaker
Do you think that incentives are aligned there? And if not, where could there be a kind of misalignment? Yeah, it it's a great question because philosophically, things are generally aligned. I mean, advertisers want performance for for their media. Publishers want to give enough performance for their media for you to continue investing with them. Like marketers have choice. You can spend with Google, you can spend with Meta, you can spend with TikTok, you can spend with Programmatic. There's a million places that you can go and try to get some some outcome of growth.
00:02:00
Speaker
But the the challenge is ad platforms have a lot of other incentives. Whereas like you know if you're if you're a business, it's generally like, how can I get as much as possible for the least amount of money as possible? like There's probably some differences in that that might not always be the case, but I'd say overwhelmingly that's true.
00:02:17
Speaker
um If you're a publisher, you are trying to get your advertisers results. You're trying not to upset your users too much by putting advertising in front of them constantly. Like if you've ever, hey, I've gotten the same ad seven or eight times and it's like, well, they probably have like a tight cap of when they're trying to spend the money and we need to be serving the impressions and generally compounding frequency is more likely to lead to a conversion
Channel Focus for Brands
00:02:40
Speaker
outcome.
00:02:40
Speaker
um But actually, where where things probably deviate the most is that publishers need to monetize the corpus of their inventory. like You can't just serve you know like Google head programs like YouTube Select or YouTube Premium or like, hey, these are the very top advertisers. And it's like, yeah, everybody would love to be at a Marques Brownlee YouTube video. That's great. He's a famous tech reviewer. like That would be awesome. But what about you know the much smaller creator with a very small audience that's very niche?
00:03:11
Speaker
Maybe this is where you want to spend your money. Maybe maybe it's not. Maybe it's correlated with performance. Maybe it's not. um But the challenge is, if you're if you're YouTube or you're meta, you need to <unk> advertise you need to have enough AdSense revenue across that ecosystem so that people continue uploading to your platform and you know adding to to the the content landscape. So um the publishers had this like multidimensional problem of,
00:03:36
Speaker
How do I get advertisers enough performance such that they come back? How do I get creators enough revenue such that they continue to view us as like a large player in the system? And how do we make our users happy where they're not getting these completely out of context or getting hammered with ads constantly where, you know,
00:03:54
Speaker
like This is something that's being tested at these platforms all the time. It's what should be the really relationship between organic content and and ads. And what are the things that you know if you've ever seen like in YouTube, if you try to skip around, it shows you ads right again. And a lot of those are based on experiments. And it's trying to figure out what is you know where can you push push the consumers a little bit more versus where do you need to say, hey, like we've actually gone over
Advertising Metrics and Efficiency
00:04:17
Speaker
the line. And how do you continue to test that?
00:04:19
Speaker
From the perspective of a marketer who is in charge of making decisions on where to spend their money, how do these incentives impact that decision decisioning framework? Meaning, is it that we have to look at channels that maybe or have more premium placements. Are there concerns over testing on channels that are showing too many ads, specifically from a performance marketing perspective? How should marketers think about that?
00:04:50
Speaker
Yeah, I mean, I think the the main thing that I would be, and I say this as like the person who who works for an experiments company, but but I truly believe is that one general hypothesis that I have in the world of advertising and marketing is that it's deliberately complicated. These these platforms give you so much data. They give you so many metrics. Like, oh, I'm interested in views. Like, how about engaged views? How about the other types of, and you get, you you always get the data to tell the story that you want to tell. like Are you doing well? How do you communicate this to your internal teams? How do you get leadership or finance ah to buy in on the investments that you're making? One of the the places where measurement is really important is, are you actually driving the outcome that matters to your business?
00:05:31
Speaker
i don't think most companies care too much about how they're able to do that. If they're able to do it, whether you know it's great metrics or it's like other ad networks that you don't feel as great about, if you're able to drive that material performance and you're driving customers who have the right long-term value or the right AOVs, it's not necessarily as concerning. so I don't know that I would look at it of like, oh, I need to advertise with a platform. Whether the frequency is 15 or the frequency is 3, if it's driving the incremental outcomes, I think most folks trying to be really efficient with their marketing spend wouldn't care too, too much. The place where it's really tricky is that it's really hard to get to that ground truth of, did I actually drive an incremental sale?
00:06:13
Speaker
like That's the piece that is really tricky because you're going to be able to say it drove conversions. You're going to be able to say it drove views or impressions or time spent or or any you know bevy of other metrics or brandless studies that all of these folks will will provide for free.
00:06:28
Speaker
um But ultimately, it's, did it drive that incremental sale or not? And um so I think from that standpoint, ah the network doesn't quite matter so much from that performance lens. The other piece that I think is really important is just this may, for for a very pretty technical person, I think ah a very non-technical answer to your question here is like, where are your customers? like that That'd probably be the the number one thing I think about is like, where are your customers?
00:06:53
Speaker
ah really large, sophisticated, one of the top advertisers in the world that I worked with for my time at Google. They had the perspective that 90 plus percent of their advertisers, of their potential customers are going to be with Google or Meta, just globally. like It's going to be in one of these two places.
00:07:09
Speaker
Yes, we could advertise in all of these other places, but like let's just get those two really, really good. Let's know exactly what they're driving. Let's understand exactly how to move money across them. And maybe we're giving up on 8% or 10% of the addressable population. But if you think about it from an opportunity cost standpoint, getting these those two places right is more important than
Experimentation in New Channels
00:07:30
Speaker
anything else. not Not everybody feels that way, but that was something that I've seen work incredibly successfully.
00:07:35
Speaker
Yeah, I agree with that as well. And I think you see a kind of recurring story of scaling where folks will start with one or two channels and really get that right and then continue to add budget there until they're really hitting their diminishing returns at which point you expand to different channels. The calculation that maybe folks don't think about at that point is that the marginal effort in launching a new channel might be not ah ROI positive if you could just continue to scale in a few channels. And i'm a I'm a big proponent of diversification, but I'm always thinking about that as well. Is this going to take a lot of time? Yeah, I mean, from my perspective, I think about diversification as sort of a risk preference. like it's it's great you know It's almost like thinking about your own money. like You could just be putting it all in Google and Metastock, but
00:08:29
Speaker
There is value in some diversification, especially with highly correlated markets. And Google and Meta philosophically would be would be pretty correlated. I mean, the the brand that I was mentioning before was a brand that was spending over a billion dollars a year in advertising. So it was somebody who was spending an absolute fortune in those two channels. And and truly the the way that they thought about it was your Your point was the exact right one and sort of where, you know, sometimes I think about marketing measurement in the world of like the matrix of like you take the red pill and you see how far the story goes. They would have argued that most companies do not understand how to actually even understand or interpret diminishing marginal returns.
00:09:09
Speaker
Like, how specific are you on it? How, like, how well do you know of what another dollar will do to your business to grow in certain channels to take away from others? And I think their hypothesis was that it's really, really hard to have that answer across 10 channels, but it's still hard to have it even in two. But I think the idea of like that is where, you know, dollars to donuts, that's the place where you're going to go. And within a Google or meta, there's also many, many different ways to buy your media. And like they they have,
00:09:35
Speaker
Many channels or or tactics within those networks, but um I don't disagree. And I say this from this is a pretty extreme point of view. Not everybody believes in that. But I think if it if it were my marketing budget, I would certainly have diversified more than than they did. um But I think.
00:09:54
Speaker
You're right it's like it's getting it set up it's getting it to scale it's understanding it and i also wouldn't go into a new channel miss i was committed to that channel. like ah The reason i'm going here is i'm gonna figure it out i'm gonna spend the amount of time i'm to spend the amount of resources i'm gonna figure out how to measure it and figure out how to build creative for it.
00:10:12
Speaker
Because it it sounds great to, oh, let's just throw stuff against the wall. The the challenge there is often I think companies will over diversify, not give those channels an opportunity to even be successful, and then determine that they didn't work. And the reality was it could have worked, and just if you had gone about it in a different way. Let me ah go a little bit
Marketing Changes for Small Brands
00:10:31
Speaker
deeper on that point. And I want to get into your time spent at at Google and and House, of course.
00:10:36
Speaker
When it comes to new channel testing, often there is an evaluation process. We look at a couple of different channels, what makes the most sense for our our brand, and then we'll give a ah budget. We'll give some sort of expectation of what our KPIs are, and then we'll we run the test.
00:10:52
Speaker
What often happens in those circumstances is that you launch a new channel, you're figuring it out, you're figuring out what creative works, the bidding, everything. And the ones that I've scaled and seen success with, it often happens that performance is a little bit like crazy out of the gates. And then you're seeing your CAC, your CPA's kind of come down and then they level out. And then you're like, okay, this is what my my normal is.
00:11:15
Speaker
What I also see in those circumstances is that you'll try to run an incrementality test and you measure the beginning stages of that spend before you've actually figured it out and made sure it's optimized. Is that a situation that you see a lot with advertisers that you work with? Yeah, absolutely. And and again, it's not even necessarily a new network. It could also just be a new ad format or a new bidding type or like hey, you know often with conversion optimized systems, that conversion optimization takes a little time to to get going. You have to spend a certain amount in order to get those thresholds to to go to a more efficient standpoint. I think this is where context really matters, is um are you evaluating the product in the right way? Are you giving it enough time? Are you giving it enough budget? These are some of the places where I think we we actually really shine at houses. So often, companies will come to us and they'll say, we ran this test and something didn't work.
00:12:10
Speaker
um Or, we hey, we stopped spending money on this channel this week and we didn't see an impact or we saw some huge impact. Not realizing that often the reason they weren't spending this week was some non-random force in their business that made it an actually a pretty poor comparable to to other times in their business.
00:12:31
Speaker
so I think getting things set up correctly is is really important because ultimately when you're when you're thinking about setting up like a test budget for for a new channel, you actually know more about this than I would, but a budget could be 20 grand, it could be 30 grand. And I think often it's a little bit of a finger in the wind of like, oh, what is what do we have as a testing budget? Whereas I think if if you had really prescriptive statistics saying, hey, we are significantly more certain of an outcome and being able to measure it over a period of time,
00:13:00
Speaker
If you spent 40 instead of 20, I think most companies would spend 40 and it also just gives them the opportunity to be more prescriptive of what does it actually take to get a real significant read on a new channel. I think so many experiments are structured to fail initially where, Hey, I turned something off for two days, but it turns out, you know, the product is a seven day consideration cycle. And
Incrementality Testing
00:13:23
Speaker
like that, that happens all the time.
00:13:25
Speaker
for for brands that are you know maybe smaller startups and so ah just scaling their their media mix. They're spending on a ah few different channels. they're They're seeing some performance and you know they're testing creative. They're testing different ah campaigns. They haven't really tapped into incrementality. They haven't really systemized their ah experiment design or you know created a roadmap for it. What are some tips or or things to look out for that you would say to to those advertisers?
00:13:56
Speaker
yeah and and then this is where also As the person running an experiments company, I like to try to have a very pragmatic point of view. The the nice thing about being a smaller company is that your business feels changes in your budget. You will notice, I mean, you will notice, hey, we saw a lot more sales coming in.
00:14:14
Speaker
we We cut budget, we see things going off. where It gets really tricky is when you start with dealing with these companies with tens of millions, hundreds of millions, billions of dollars in budget, where things are not so clear cut. It it can take a long time to to see those changes. You need very, very precise estimates, whereas smaller businesses can have very significant lift.
00:14:33
Speaker
um Often what we see with incrementality tests, like not always, but generally, Lyft is somewhere between 0 and 20%. I think that is a fairly common outcome and to to be in that range. But if you're a very small brand, you can absolutely see a 50% lift. You can see an 80% lift. Those are not terribly surprising because the marketing is really kind of driving the underlying business.
00:14:56
Speaker
So I would take it a little bit from an anecdotal lens of, you know, you can run, quote unquote, less precise or less tested experiments of like, spend more and see what happens, spend less and see what happens. I think this is where I think taking advantage of the entrepreneurial mindset, like something I like to say a lot is using it as a pioneer tax.
00:15:15
Speaker
where you are trying to discover something new and that might be expensive. That might mean you're leaving sales on the table. It might mean you're spending some money inefficiently. But unless you do those efforts or you take those steps, you're never really going to know. So I think starting with that more heuristic layer of we're going to make these changes and see what happens to hey, we actually have a budget that's large enough to like, we should invest in this correctly. We need to get it right. Like the difference between 5% and 7% left is a really big difference for us. The difference between $100 CPIA and $130 CPIA. It's really, um that is where businesses will get to from a ah state of maturity. But I think in the earlier innings,
00:15:54
Speaker
um um like I'm curious for for your perspective too, like how much you diversify it early on. Early on, I probably wouldn't diversify too much because I would want to really get an understanding of the things that are I'd be in meta, I'd be in Google, maybe I'd be in TikTok. Perhaps for certain businesses, other channels would make a lot more sense. but i' you know I'd start with the main ones, and then I'd also ask my question of where are my users and and and work on those things. the The only other thing that I would strongly recommend folks who are getting started here is only change one thing at a time. Don't go up in one channel and down in another channel at the same time, because you're just it's going to be really hard to to observe those differences and ah feel confident in sort of the outcomes. By changing one thing at a time, you are more prescriptively looking at at what the levers and in the business might be.
00:16:44
Speaker
It becomes so challenging too when you're operating at scale and you're trying to isolate
Insights into Google Advertising
00:16:49
Speaker
experiments. Even from my perspective, you know I can control changes in our media mix and be very intentional about the budget that we're changing. But if homepage is owned by product or the onboarding funnel,
00:17:06
Speaker
changes, you know, and if there's not a clear roadmap or like idea of when those things are happening, that can kind of mess up experiments as well. Absolutely. And and that's that is where more sophisticated experimentation really helps. is like The way you're able to normalize across treatment and control groups, having placebo tests to like balance out error really neutralizes a lot of those exogenous variables that could potentially impact the experiment. Again, like that is where it could go. I think starting in the earlier in the earlier places of
00:17:37
Speaker
even directionally having an understanding of the cause and effect of of ads and and sales in the business. but But you're exactly right. This is one of the places where it's so tricky is ad platforms are constantly changing and businesses are constantly changing. And you're trying to experiment within that landscape. So it's sort of like you you're building this amazing sand castle, but you're building it on quicksand. So that's one of the challenges.
00:17:57
Speaker
I definitely want to talk about more sophisticated experimentation and you know what you're doing at house. First to to start though, you you spent you know over six years at at Google working very closely on on their ad products and more specifically on YouTube. Could you give a high level overview of kind of what you were working on while at Google and also kind of the spend and scale that you were looking at with within that Sure. So when I when i started at Google, I was a data scientist. I ended up leading ah a data science team there. We generally worked with a lot of the very sophisticated customers who were focused on incrementality. ah netflix is i ah like Netflix is a big example. Hulu is another example. So I was deeply embedded with the conversion lift team for a long time, working with people like you all of the different product managers and and engineers and
00:18:49
Speaker
How do you scale up experimentation infrastructure? How do you build incrementality testing? How do you have, you know, we were using GhostBids architecture, and but then also around geo experiments and ads data hub, like you name a measurement product from Google. I touched it and in ah in a variety of ways over the years. Those types of businesses, we're talking about billions of dollars in ad spend. So very, very large scale. The last two years I was at Google, I was a product manager in privacy safe ad targeting. So I was focused on third party cookie deprecation, IDFA deprecation, whether it's, you know, more advanced keyword targeting, advanced contextual targeting, what are the different ways you're able to target users without knowing exactly who they are.
00:19:25
Speaker
So we have this pretty broad spectrum of of the the different types of things that that customers were using and got a really good sense of what do large companies care about, what do smaller companies care about, and how do you make advertising work with within these ecosystems, at those
Strategic Channel Selection for Large Brands
00:19:41
Speaker
types of budgets. Because as you say, like the there's a really significant difference in spending a multi-billion dollar budget from a five million dollar budget. the The targets you go after the amount you're trying to try new things, the you know the ah pioneer tax you're willing to pay can can vary quite a bit. Sure. If you kind of had to break that down into different stages of spend and some of the you know trends and but within each stage, what what would that look like?
00:20:09
Speaker
Generally speaking, smaller companies use more automation than larger companies. And so that is like a pretty fair characterization. So systems like Performance Max, ASE, often smaller advertisers ah run run more of that media. Obviously larger companies are running it as well, but very commonly large enterprises are focused on where their media is going. What impressions are they serving on? Which creators are they supporting? Because, you know, if you you even think about YouTube is an example. When an advertiser pays for for an ad placement, they're de facto funding that creator's business.
00:20:48
Speaker
And so there's like broader questions around who do you want to be in business with? And do they have the values that you support? Do they have the ideals that you support? So larger companies often are a little bit more more careful around that. But the beautiful thing about these marketing ecosystems is they fundamentally scale. So there are companies who spend billions of dollars a year in search. There are companies that spend $5 a year in search. There's companies that spend hundreds of millions of dollars in YouTube. There's companies that spend almost nothing. So so you really do have that that broad spectrum. The more you put into these conversion optimized signals, you'll have different ways of reaching diminishing marginal return curves. Often, I think for that reason, larger advertisers tend to focus more on reach and frequency and maybe brandless surveys or or some of these other metrics. The most sophisticated performance marketers I've worked with tend not to care about those metrics as much.
00:21:38
Speaker
Those are narratives that come out of the ad platforms of, wow, this is your reach. This is your frequency. That sounds great. But what is it doing for me? Well, I like that. That's where it gets a little bit fuzzier and you know smaller and medium sized businesses tend to care about that a little bit less. But on the yeah, certainly on the large end, like they're often buying multiple products.
00:21:59
Speaker
Whereas I could see some advertisers, you know, smaller, medium sized businesses who are, I'm buying VAC on YouTube or I'm buying a reach campaign, but maybe they're not buying everything. Those larger companies are often buying versions of everything. And the question is just how much and what is the right scale and support for them?
00:22:18
Speaker
Sure. I think that's generally something that I've kind of heard as well and and seen that the more that you're operating at scale, the more resources you have to dedicate to the thing, the more that you can start to customize a lot more and look more granularly on these, you know, almost esoteric or like very specific areas that maybe, you know, start up with a two person marketing team just like doesn't have the time to dig into.
00:22:44
Speaker
Yeah, I mean, it's ah it's a great point. Like something that I've worked with a lot of advertisers on was always the question of how well would this channel scale? Because let's just say you have a billion dollar marketing budget and the most you could ever spend on this channel is 10 million dollars. Even say it is an incredibly efficient 10 million dollars. It's very promising. All your customers are there. Is it worth dedicating the time and resources and you know recurring infrastructure to manage that channel when you know that it will not be able to scale for your business versus investing in something you're not as sure about, but if you really get it right, you can spend $200 million on it. It actually often will make more sense more more sense from a time perspective to spend your time there just because you know that it
00:23:26
Speaker
will continue to be a large place that you that you invest, whereas some of these other networks, it can just be challenging because it's always an opportunity cost trade-off of people's time and and the resources that
Balancing Performance and Brand Strategy
00:23:39
Speaker
they have. You can't do everything. How do you mirror what drives the most impact to your business to the resources that you have?
00:23:47
Speaker
Let me ah let me kind of give you a ah case study in that actually or or or maybe an example. so and Over the past, I would say two to three years, the kind of marketing and and startup community has transitioned from talking about how much they've raised to profitability.
00:24:04
Speaker
and Budgets have become constrained. I mean, in some industries, maybe not, but I would say on on the whole, folks are trying to be more efficient with ah their spend. And so what that means is that from a performance marketing perspective, they're running more things that are optimized towards performance. They're really being mindful of their CAC. They're moving away from more like harder to measure areas. right What I think i think that that can be challenging though in in some ways and what I've seen sometimes is that when you're operating a media mix at a certain level of scale, let's call it a you know seven figure media mix, that actually you are doing yourself a disservice by testing more and more performance channels
00:24:56
Speaker
where you might be ignoring the effect that like brand marketing upper funnel reach-based media has on your overall media mix. What would you say to marketers that are in that situation where you know the CFO is very concerned about revenue, CAC, the the business fundamentals,
00:25:17
Speaker
And you might have this thought that we're actually spending a little bit too much on on Facebook performance and we need to start doing some things like TV and other channels. I think this is a great question. And one area that I would like dig into the underlying premise a little bit is that finance departments, like these companies, every trend you said is absolutely correct. Companies are trying to move towards, Hey, I need to understand what I'm getting for my dollars. But one of the challenges is when every ad network has a conversion API and you're being hit by ads on multiple platforms, you have a huge attribution problem where you as the marketer, the the marketing team will be able to say, these are the conversions that we drove.
00:25:55
Speaker
But there's going to be a crying wolf challenge with those other other teams because the number of conversions that are being reported by ad platforms will exceed the number of sales that a business has. like And we're we're moving closer and closer to that world is as kind of those those types of APIs come into effect. so Even the places where, hey, we're driving performance, it's getting harder and harder to know if you're actually driving performance. Because when you are spending money, the the signals that one ad platform will have are similar to the signals that another ad platform will have. They're trying to target the same types of users. They're both optimizing to a conversion event. One person signs up and three people take credit for it, and you're not able to deduplicate it. and
00:26:35
Speaker
So I think the the underlying problem of, I think, how conversion APIs are going to impact advertising is just going to get larger and larger. But your your broad point, I think, is absolutely a ah correct one. is you know Folks are going to overly focus on those signals because they get a feedback that they feel like they can explain to folks. When and but in martin like brand marketing, whether television or reach campaigns or prospecting, you might not see it. And I don't think that that's right because ultimately those users can drive value to your business, especially if you're targeting and you're like, I know these are the types of people who are our customers. I know these are the places that they're spending. Often when you're buying these reach campaigns, you can actually buy them at a much less expensive CPN than some of these like conversion optimized signals because your
00:27:22
Speaker
you know, operating on pretty efficient CAC targets or pretty cut their CAC targets. um So there's there's definitely this argue for diversification. the The challenge that I would add or like the the place that I where I kind of come from on this is people have always said that brand marketing wasn't very measurable and in a world where you're moving towards efficiency, it makes sense to go towards the conversions. I actually think we're also entering a world where there's more fragmentation of ad platforms. There's more conversion APIs. And even if you think you're getting something for it, you don't know who you are. You don't know which impressions were actually driving
The Importance of Retargeting
00:27:56
Speaker
conversions. You don't have any semblance of incrementality.
00:27:59
Speaker
And with that in mind, actually brand is a little bit easier because you're not holding yourself to that same standard. So I absolutely think there's a role for for brand marketing. You know, one of the beauty of experimentation platforms like ours is that you can use them with brand campaigns. You can use them with conversion optimized campaigns. You can use different combinations of brand and conversion to see if your upper funnel is driving lower funnel performance, which has been a ah long time narrative from ad platforms. I remember, you know,
00:28:27
Speaker
In YouTube, we were trying to sell this narrative in 2016, 2017. We'd say upper funnel drives lower funnel performance. We had no idea. We just wanted to sell upper funnel and people wanted to buy lower funnel. But now we're getting to a place where with sophistication and marketing measurement where you can actually test and and run those hypotheses and I think can can help quantify the impact of upper funnel in a way that that might not have been shareable before.
00:28:52
Speaker
I think it from a marketer's perspective, it also helps to have a lens on consumer psychology and the psychological process that ah a person, a human being goes through to to making a decision about taking an action um to make a purchase and buy something. and it's it's It's not always a ah linear path. and um you know There is an emotional component to that that a lot of times brand advertising can afford to switch where direct response advertising doesn't have that same kind of capability.
00:29:22
Speaker
Yep. And I think another one where, you know, a lot of folks in the world of incrementality will push pretty hard against retargeting. You're sort of like, oh, if you're remarketing, like they've already seen your ads, the next ad is not nearly as likely to be incremental. Often in the the world of advertising, I would fall on that side of the spectrum. But when you know about your buyers and you know about their consideration cycles, so often you just need to be top of mind.
00:29:45
Speaker
because they're still thinking about it. If I'm thinking about like I just bought this shirt at Uniqlo over the weekend and I had been thinking about it for a while and you know what like at some point they just need to get you over that line. You're kind of like deciding what exactly you to do and when exactly to do it.
00:29:59
Speaker
There's always an opportunity for you to be there. Again, at Google, we used to call it the zero moment of truth. But the idea of like when is the person, like they're actually going to make this decision. They're actually going to change their their behavior. And ideally, it's that your ad is present that at that point versus someone else's to eventually steal the conversion rate in the cycle.
00:30:17
Speaker
I started my career at a retargeting platform, and and I think it's even more important for brands that have a launch longer consideration cycle to just get really meta on you. I'm in the process of evaluating media mix modeling vendors and actually looking at house for some other things and you know um looking at eight to 10 different vendors right now. um I don't think I've gotten any retargeting ads yet. but you know, this is going to be a process that takes three months for us to conduct, to actually select a vendor. And um I visited everyone's website, like people should be showing me ads, people should be showing me ads, driving me to content, driving me to webinars, driving me to whatever. um And you could say, yes, it's it's not as incremental, but because they already know about you. But we also know that there's a certain um amount of impressions that a customer needs just to even remember you, let alone make a ah consumer action. Yeah, I mean, I'll say as an aside, like that, I think that's one of the the
00:31:10
Speaker
you know, in crowded marketplaces where there are so many MMM vendors. Yeah. It's like, how do you, how do you stand out and how do you make sure that you're the, the one that kind of comes top of mind or is in first choice? Uh, absolutely. I think there, B2B is a whole different side of the world where there's always like a relationship between like how creepy do you want to be? Because like that, that's where I could go down a very deep privacy, privacy landscape. I worked on topics API and flock for a little bit at Google and.
00:31:35
Speaker
a lot of background and like the ways in which user data can be used negatively, which is is not perfect and is something I don't appreciate about the advertising ecosystem. It's good that we're trying to to move away from that. But exactly as you said, like for for those types of of cycles, how do you make sure that you're present? How do you make sure that you're available?
Founding of House: Democratizing Ad Tools
00:31:53
Speaker
Hey, these are the, like something, you know, you could use like like video ad sequencing or like different sequences of creative to say like,
00:32:01
Speaker
This is the upper funnel. This is the and what other platforms are using and how do you continue to to drive activity amongst customers? Like, absolutely, it makes sense. Sure. um Let's talk a little bit about your evolution at House. So I think a first question, we talked about it when we first connected, um your motivations for for starting House. What's specifically about your time spent in Google working on the basically most sophisticated ad platform in the world? What about that experience led you to being like, you know what, I need i need to start something i new. You know, I have to get more involved in this incrementality space. so
00:32:39
Speaker
Yeah, I mean, for me, so much of it was about democratization of technology. i I worked with companies like Netflix, like the most sophisticated companies in the world who had teams of causal economists, engineers, product managers, all trying to understand if their ads were a little bit better or worse.
00:32:55
Speaker
And then you think about what the next ad, you know, the next competitor of Netflix, not only do they have less money, not only do do they have fewer resources, but they're also at this huge technological disadvantage where they're not going to be able. They they will not have the advanced capabilities that a Netflix would have or an Amazon would have or a booking dot.com or or any of these really savvy, large publishers. And I just thought, like, gosh, I.
00:33:21
Speaker
this isn't fair in the world. like I want everybody to have access to these tools. And then as well on the targeting side, you just see how many dollars are being wasted. I think everybody, there's the old quote of 50% of my ads are wasted, I don't know which half. I think ultimately, just to my core, I've always believed in people should get value for the dollars that they're spending. And there are cases where, whether it's like Performance Max or these other systems, where not only do they not tell you about, hey, there's these model, the views of the world of what's actually happening with your advertising, we're not even gonna tell you where your ads are at. And it's like, that's not a good ecosystem.
00:33:59
Speaker
And I just thought I know there's these other methods. I know there's these other tools. I feel really passionately that everybody should have access to world class scientific resources. But it is not feasible for ninety nine point nine percent of businesses, whether an expertise or an infrastructure or the cost of it. Where even if you're like, hey, I do want to invest in this.
00:34:19
Speaker
spinning up a 15 or 20 person team versus using a system like house like i I really wanted to democratize a lot of what I had had in my head and the amazing kind of like advancements in causal inference and econometrics that have been making their way into businesses, but how do you do that in a way where um We work with small, you know, just a company like house. We work with some of the largest advertisers in the world, companies like Fandules as a good example. We also work with really small companies. We only spend a couple million dollars in marketing and are trying to figure it out. And I i love that that whole spectrum can can find benefit in these tools and methods. And um often incrementality is one of those
00:35:00
Speaker
areas where once you get a taste for it and you realize, oh my gosh, this is what my media was actually driving for me, it gets hard to stop because you can't unsee the things that you see. And I think that's true for me. And it's true for people from ad platforms. It's true for people who have run really large marketing budgets where we'll have customers who are spending $250 million dollars on one tactic, run a sequence of experiments, and then turn it off. And that is a $250 million dollars reallocation to their businesses. We've had companies who have told us, we can IPO faster because of you. We've had customers tell us, like this has been the most dramatic thing we've ever seen. We've had customers who are like, we've paid off our contract for decades. You get into these really large spectrums of outcomes. And the the funny thing is, if you were to ask me, is it surprising? I'd say no.
00:35:48
Speaker
And it's not surprising because I've just seen firsthand from the platform side, all of the ways in which you're you're spending your money. and And often I like an ad platform still like casinos in Las Vegas, Google's beautiful offices and their free lunches. It's, you know, ideally it's built on winners, but ah but ultimately they have to monetize all of their inventory. And I love the idea of helping align businesses and, you know, their dollars are going to things that directly impact them.
House's Approach to Experimentation
00:36:14
Speaker
I love to talk about some of the specific experiments and some of the the learnings just to kind of contextualize it for the audience though. ah What has your product evolution looked at at house since you started ah I think 2021 and what are the reasons that it kind of has evolved as it has over the years?
00:36:35
Speaker
Yeah, were we are very focused on high velocity causal inference and experimentation. there's you know we've We've never been on more on the MTA side of the world. like I think I have some pretty strong opinions there, but I think MTA is infrastructurally going to be going through some changes. It's it's definitely a hard space to be in. And I think MMM is something that is obviously becoming a lot more common. the The question that I always ask myself is, do I have anything novel to contribute here? Is it unique? Is it differentiated?
00:37:03
Speaker
we We work with a lot of, you know I'm sure all the MMM companies you're you're thinking about partnering with, we are partnering with a lot of them. We're sending our data to calibrate ah their models. But ultimately, I think you one model versus another model, I think there's a lot of folks who are who are working on that. Unless we have something really meaningful to to contribute there that'd be different.
00:37:23
Speaker
Experimentation is really critical in incrementality testing because it's ultimately the only thing that is not a modeled exercise of, hey, here's what we think is going to happen. This is what actually happened to your business. These are actual results. It's based on your data warehouse. It's off of your internal infrastructure.
00:37:40
Speaker
and When I was leaving Google, I knew that I wanted to democratize access to world-class experimentation and incrementality testing. And that is still true today. the The questions that we're moving into a little bit more is, how do we make it more useful? How do we make it more automated? How do we make it easier to be part of the the workflow? So often we'll talk to companies who are like, oh, we need to figure out seven things from this experiment because we can't do them very often. And we just can say to them, what if you were running a you know and it' a new experiment every week or every other week, or you're running a lot of these things that because we've reduced all of the friction and all of the engineering left where
00:38:17
Speaker
it de-risks any individual experiment. That's where folks are able to to do a lot more, drive better results. And ultimately, I think the more you can be grounded in that that truth of this is what's actually happening to your business, here's what would happen as you observe and and see these changes. I think the more impactful of a partner we can be and and ultimately ah understanding we're just one part of the the the ecosystem. But I think the more we can get our customers to ask the right questions and be more curious and run better experiments. If people run great experiments, they have really good outcomes. And I think that is what I'm personally really passionate about. And ah I would love to move to a world where companies aren't just like, oh, we, we have like a test and learn agenda. We're trying something here. We're trying something there to a world where everybody is running their experimentation program the way a Netflix would, I think would be a really awesome.
00:39:06
Speaker
A lot of the ad platforms or have a little bit of a black box in terms of the abilities for incrementality and lift testing, and some more older platforms don't even have that capability, right? Where does house kind of sit in that framework? Everything we do, our customers can validate. We'll supply model weights, you can back out the data in your own data warehouse. It's all a reflection of the data warehouse anyways, but we give the treatment markets, we give control markets, we share exactly what we're doing and how we're doing it. There's obviously some special sauce. like We have ah teams of economists and professors. like We have like truly a world-class experimentation team and and ah who've built really novel models and are able to get more precise outcomes than things that would be out of the box. like More precision, tighter confidence intervals. There's a lot of science innovation there.
00:39:59
Speaker
But ultimately, we will tell you our customers, this is how it works. This is what we're doing. If you want to see it in your own data warehouse, here's how you would do something like that. Often, for for the companies you do ask, they'll audit it once, and they'll be like, oh, this is what we thought. And then they move on. um So I think my my thought was that people don't want to trade one black box for another black box. And and I think, like again, like there are some amazing um MMM companies, but I think that's a little bit of a risk in that world, because you moved into this modeled world, and it's sort of like Like who do you believe in the most? I think there's something really powerful and like, this is a discreet experiment. We ran it in this period of time. This is what the outcome, what you lose in is a little bit of the extensibility and there's some cool stuff that we're working on there. But the the value is that it is the sort of like the least modeled version of of anything in in the marketing measurement space.
00:40:47
Speaker
I know we're wrapping up one one last question for you because I know you mentioned it before. I would love to hear your your strong opinions on the infrastructure changes to multi-touch attribution. Of what's going to happen with MTA? Correct.
00:41:03
Speaker
Ooh, that's a really good one. Multi-touch attribution is fundamentally challenging that you're trying to deduplicate off of a conversion where, you know, most of the world is moving away from UTM parameters and you're moving to like view-based conversion events. There's going to be companies who are hey, we're building this partnership with platform X, Y, or Z. But the the way to build a deduplicated view of a user across networks is only going to get more challenging. like I don't think there's an argument to say that that's going to get easier and more straightforward. You're still going to need a resolved user ID across networks, which isn't really possible. So you move into like more people who call synthetic events, or they'll they'll they'll model it in different ways.
00:41:44
Speaker
um the The challenge is that the the need is real. like what were my incre like What was my deduplicated value to my business that operated yesterday? What are what are the ah what is it kind of doing for us? um Something to to plug, and maybe we we should do another podcast in a month, but we're we're actually launching something where we are not tracking users in any individual way, but we're going to be able to help companies with the more day-to-day optimization using causal data that that we sit on top of. and So, you know, whether it's companies like House or other companies innovating, I think there's going to be a lot of innovation here. But if I were a marketer, I would probably use Google Analytics and that would be sufficient for now. And, you know, you can say, oh, well, there's the bias of GA, but like, you know, you could use last click, you
00:42:30
Speaker
UTM parameters are also in a little bit of, we'll see how how long UTM parameters last. I think that's potentially on the the privacy chopping block as well. But um I would use those types of systems and the the the key piece is like, so often when you talk to companies, they're like, we use last click, but we don't like it. And it's like, what you can do is you could look at last click, but not use it for that optimization. And I think that might be a space where where MTA is ah is more common. But but even if you i mean if you look at it broadly, like companies like Rockerbox or Northbeam, they're building and MMMs. I think that is kind of the the direction that these companies are are generally trying to go.
00:43:05
Speaker
Thank you so much for for the conversation today, and where can folks find you? You can find you can find me on LinkedIn, ah Zach Epstein. We're at house.io, H-A-U-S.io, and our amazing head of strategy and ops, Olivia, is all over Twitter. So folks might might see her there as well. I would love to get her on the podcast too. She'd be better than me. Awesome. Thank you so much, Zach. Awesome. Thanks, Paul.