Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
036 - From Meta to Statsig: Driving Business Impact Through Experimentation
126 Plays16 days ago
This week, I’m joined by Timothy Chan, Head of Data at Statsig.
Tim has a fascinating background — he started his career as a scientist developing life-saving drugs before pivoting into data. He went on to become a Staff Data Scientist at Meta and now leads the data function at Statsig, one of the world’s leading experimentation platforms, recently acquired by OpenAI.
In this episode, we dive into the power of experimentation and how Meta embedded it into every aspect of product development.
We unpack:
⚙️ What great experimentation really looks like
🏗️ How to build a world-class experimentation function
💬 How data teams can use experimentation to influence business decisions
⚖️ The balance between speed, rigour, and impact
👥 Why stakeholder collaboration is the true differentiator of high-performing data teams
Tim shares brilliant insights from his time at both Meta and Statsig — including how to think about experimentation as a cultural capability, not just a technical one.
If you care about driving real business impact with data, this is a must-listen.
Recommended
Transcript
Introduction to Stacked Podcast
00:00:02
Speaker
Hello and welcome to the Stacked Podcast, brought to you by Cognify, the recruitment partner for modern data teams. Hosted by me, Harry Gollup.
00:00:13
Speaker
Stacked with incredible content from the most influential and successful data teams, interviewing industry experts who share their invaluable journeys, groundbreaking projects, and most importantly, their key learnings.
00:00:25
Speaker
So get ready to join us as we uncover the dynamic world of modern data.
00:00:34
Speaker
Hello everyone,
Building a World-Class Experimentation Culture
00:00:35
Speaker
welcome back to another episode and of the Stacks Data Podcast, the show where we dive deep into the strategies, stories, and skills that power today's most impactful data teams.
00:00:48
Speaker
Today, I'll be talking about one of the most powerful levers a data team can pull, experimentation. Not just about running A-B testing, but building world-class experimentation cultures that drive real business value.
00:01:05
Speaker
Joining me today
Timothy Chan's Transition to Data Science
00:01:06
Speaker
is Timothy Chan, the head of data from Statsig and a former staff data scientist at Meta, where he helped scale experimentation to levels most companies can only dream of.
00:01:18
Speaker
Kim's career has taken him across many different industries and now to a high growth startup called Statsig. Really excited to have Tim on the the show today with us. son Yeah. Welcome to the show, Tim. How are you doing?
00:01:33
Speaker
Thanks, Harry. Really excited to be here. i Love the topic too. So. Excellent. Yeah, we all, we're about to obviously died dive into it, um Tim. It's it's really, I suppose, your specialty in your career. and But I suppose for the audience, it'd be great to get um i just a bit of ah an overview of your your career journey so far. um And what's led you to your role at Statsig?
00:01:57
Speaker
Yeah, when I was going through college, I had very much in my mind that I wanted to do science. Science was where it's at. I wanted to become ah get a PhD. I ended getting a PhD in chemistry. I worked in the pharmaceutical industry and I will consider myself a card-carrying scientist.
00:02:12
Speaker
That's how I like to think. And that's what training has led me to. what my training has led me to During my career though, however, I started getting more and more interested in the world of business.
00:02:23
Speaker
And I kept trying to think about how can I combine science with the world of business. And I was stumped for a while until I stumbled upon a career called data scientist. And I was like, I had an epiphany where this is absolutely what I wanted to do. It was applying scientific rigor to making and informing business decisions.
00:02:44
Speaker
Like when that light bulb went off, I was like, okay, okay, what do I need to get into this field? and So I went and started of learned learning all the technical skills in terms of Python and SQL, but I already had the scientific thinking and training to be able to jump it effortlessly into this field.
00:02:59
Speaker
Excellent. I mean, that's a massive change from you know designing designing drugs, essentially, to data science and and commercial businesses. How did you deal with that deal with that change? and how but I think it'd be really interesting to maybe touch upon what yeah what did you do on to make that change as well and to get into that but that sector?
00:03:20
Speaker
Yeah. So like my, when I worked in the pharma industry, I was designing drugs, trying to discover the next cure for Alzheimer's disease. um That's literally was a project I had. And these projects are obviously very complicated, very challenging. There's so little of the science and biology that we truly understand.
00:03:37
Speaker
So, so these projects are kind of exciting because they're like game changing to humanity, but they're also really high risk, high reward. But my day-to-day was looking at trying to look at the data, trying to figure out you know how can we possibly tweak the molecule we're working on to get us all the goodness, but also remove all the badness, such as toxicities and things like that.
00:03:56
Speaker
So that was like just sort of building a really good intuition for data and also really examining when you're looking at a data point, what is this actually telling you? And what is it more importantly, what is it not telling you?
00:04:08
Speaker
And that kind of thought process translated very easily over into the world of tech. I feel like in the tech industry, folks are very hungry for these data insights. And if you're in the right environment, and my first job was at Meta, or Facebook at the time it was called, in that environment, folks were hungry for like information, for insight, for data.
00:04:28
Speaker
and And that was a really fun environment to work in. There. so So as I moved to tech, I had to i had sort of the what I like to think the thinking skills as a scientist, but I was just missing a technical component. So I took a boot camp.
00:04:41
Speaker
So 13 weeks boot camp. It was like full time. So you had to like quit your job, jump in for 13 weeks. You had to pay. ah forget how much the boot camp cost, but um they just drilled you every single day. You 40 hours a week. You were working in Python.
00:04:55
Speaker
You were working on rebuilding machine learning. algorithms by hand. And so that was very intensive and really got my technical skills up to the bar that was necessary for me to start a career.
00:05:07
Speaker
Excellent. It's great to hear um everyone's stories into into data. People come from love such diverse backgrounds, but clear that that sort of scientific mindset was clearly being been crucial. all and would be great to, yeah, understand bit more about your your time at Meta, and I suppose what did that teach you about experimentation and suppose particularly scale?
Experimentation at Meta
00:05:36
Speaker
When we talk about scale, i like two things come to mind. One is like the number of users you need for experimentation. And ah what was it really fascinating to me is I worked on a team where we were doing experiments on the Facebook Blue app.
00:05:50
Speaker
So like the one that's got that's that's famously got like two or three billion ah users on it. Obviously, like when you're working on that scale, it's not hard to get sample sizes, but that's also a well-optimized product.
00:06:01
Speaker
So like the effect sizes you're trying to look for are on the small side, probably like 1% or 1%. But you had enough users to do that. ah Then I also worked on teams that were like, hey, we're working on a brand new product.
00:06:13
Speaker
ah There are currently no users today. And so you have to learn to how to see the process on how to get those projects up and running to the point where you can get experimentation going. It's not hard for a very speculative Facebook um new product to get thousands of users, but that's obviously thousands are not interesting at Facebook scale. You're going to need to get to tens of millions, hundreds and millions eventually. So they're looking for like big bets here.
00:06:39
Speaker
But they still use experimentation though at that scale of just like 10,000, 50,000 users and and use that to really understand what your users are doing. Sort of look at how they're using their your product, sort of testing new features and seeing how those are doing on it. Obviously the effect sizes you're looking for, if you want to grow from a 10,000 user product to a 10 million, um which was obviously ah consistent in Facebook's ambitions,
00:07:04
Speaker
you had to look for bigger bets. You couldn't be looking for 1% wins. You had to be looking for 10, 20% wins, which I think if you're not doing that at that scale, you're not working on the right things.
00:07:15
Speaker
But what was really cool is it was the exact same tool that we used on tens of thousands of users as we did for hundreds of millions of users and exactly the same process. And so that was really interesting for me in terms of something I learned from Meta.
00:07:28
Speaker
The other part when we talk about scale is how much experiments are are people running. and And Meta, experimentation was was a standard playbook. Every single product team used experiments.
00:07:42
Speaker
It was a It was a very data-driven culture. Folks are ah hungry for data, as I mentioned before. Experiments, to me, are like the best form of data you can generate.
00:07:53
Speaker
You have control for all variables in a randomized controlled trial. You've only changed one thing. You don't have this, when you see an effect, You don't have this debate over correlation versus causation.
00:08:05
Speaker
It's literally causation. This is the scientific gold standard for causality. And I think because it's so simple and easy to understand, you don't have to have like complex regression models and where you're trying to ah build counterfactuals and you can debate whether you've done that correctly or not, no.
00:08:21
Speaker
yeah It's actually like really intuitive that, no, this data is pretty solid for the takeaway, where the what we're inferring from it. And so you so folks who are not trained in any sort of like advanced statistical methods can understand how this can apply to making good decisions.
00:08:38
Speaker
So I got to see that happen at Meta throughout the company, throughout the teams. Different teams employed it differently. But I i think the unit what was universal was that Meta was solidly a data-driven culture.
00:08:51
Speaker
Data trumped opinions. um It was okay to throw an opinion out there and be wrong as long as you had the data to show that. Excellent. Well, look, that was obviously your time at Metsa and Julie, some great
Statsig's Experimentation Tools
00:09:04
Speaker
guests there. And that obviously led you to Statsig. So, yeah, I suppose it'd be great to to give the audience a bit of a better understanding of who Statsig are and I suppose your what what your role is as head of data.
00:09:15
Speaker
Yeah, so Statsig is is building the data platform for product builders. So anyone interested in building a digital product, there are a standard set of tools, we believe, those that are not available in the market that would be very useful. And so we want to bundle all those together and put them into a single...
00:09:35
Speaker
a package that works efficiently. Feature flagging, experimentation, have something called dynamic configs, multi-armed bandits, all of these things, having a metrics catalog, having a data layer, all of these are sort of like an all-in-one platform that we are currently building towards.
00:09:50
Speaker
We had seen that this sort of platform works really, really well at big tech companies. They've invested in order to in order to provide this and they use it heavily to their advantage. But if you're a small, medium-sized company or if you're high-growth startup,
00:10:04
Speaker
you don't have the time and resources to build this yourself. And so we are hoping to make it easier for you to have access to that just by buying Statsig. Statsig is a first and foremost, a data company.
00:10:17
Speaker
And so head of data is a pretty pretty critical role here. I like to say that The data team touches every part of the company. We do everything from evangelism and marketing and thought leadership with writing blogs.
00:10:31
Speaker
We work with customers that are pre-sales that are trying out Statsig for the first time and walking them through how to be successful with it. We help existing customers with their problems, also gathering feedback on what features we can build.
00:10:43
Speaker
The data team also builds a lot of the computations and statistical methodologies and a lot of the big data backend for us and also for our customers. And then we also optimize the ah the query engine as well.
00:10:57
Speaker
So statistic is, suppose, almost like an out of the box solutions that yeah you've developed, I suppose, the knowledge from how big tech tackles experimentation and and clearly all of the other key areas that well that you want to share.
00:11:13
Speaker
It's the product development cycle. um When you have a idea that folks are thinking of building on their product, then you can go through the engineering cycle of building it.
00:11:23
Speaker
And then you want to be able to slowly, once you have the new code or new feature you've built, you want to you don't want to just ship it. That was the old way of building software. The new way of building software is to release it slowly or to do it in a carefully controlled manner and measure the metrics on how well it's doing.
00:11:41
Speaker
That lets you verify ah the metric you've that the feature you've built is working for your users, that it doesn't have any unexpected bugs, it doesn't have any ah unexpected consequences, such as ah maybe harming your top-line revenues.
00:11:53
Speaker
And then you can once you see that that goodness is there, you can now roll it out fully and then try to look at the data and try to figure out what's the next idea you want to build. We want to be the platform
The Importance of Experimentation in Product Development
00:12:03
Speaker
that supports all of that.
00:12:06
Speaker
Brilliant. I suppose that's going to tie in so well to obviously what we're going to talk about today, which is I suppose developing a well-class experimentation culture. And that means you know far more than just yeah running great A-B tests, which we what we'll dive into. But as always always on the pod, let's let's start with the with the basics, Tim.
00:12:28
Speaker
what What is experimentation? why what Why should people care about it? As a data analyst or data scientist, you have many, many tools in your toolbox, if I would to metaphorically say, trying to and get gather insights about your users and your product and how well it's doing.
00:12:43
Speaker
A lot of them get into this like observational approach where you're just looking at your data, you're looking for interesting differences, you're finding out, hey, for some reason iOS users don't convert at the same rate that Android users are, then you can begin to start hunting for bugs, things like that.
00:13:00
Speaker
Experimentation though is very unique as a tool in the toolbox. It is, as I said before, it is like the A-B b testing and randomized controlled trials is the scientific gold standard for causality.
00:13:12
Speaker
It is very easy to just, ah the way I usually describe experimentation to folks is that it's really simple. You just have to be able to randomize your users. into into two buckets, so test and control.
00:13:24
Speaker
And then you have to apply 100 plus year old statistics to figure out is there a difference in how that my two groups are performing. you can Because it's so simple, you can apply this to so many different situations.
00:13:36
Speaker
And so experimentation has evolved to become a standard playbook in product development and testing and how tech companies are doing it today. There's not only are you you can understand whether like the feature you're building is being built correctly and whether there's any like unexpected bugs or regressions, you can also use it for like, hey, there's two directions we can take our product, but we don't really know what our users are going to do better with.
00:14:02
Speaker
Well, you can test it. Anytime you have a business question where you're like, hey, this is really critical. We get this right. Very often you can frame this as an A-B test and gather some data in order to make a data informed decision.
00:14:15
Speaker
And then i think what I've noticed comes from the ah starting to do experimentation is this experimentation is also the most powerful lever for build starting to build a data-driven culture.
00:14:25
Speaker
Data-driven culture is where, you know, when you're making big decisions, are you using opinions or are you using data? Experimentation is a great way to bring data to the forefront in a way that's ah very trustworthy and robust.
00:14:37
Speaker
Excellent. I suppose that and the data culture as well, it yeah and it takes time to to build data culture, which I know we're going to to touch on. One of the things I think is an interesting area to maybe speak about is some of the misconceptions that but you you might seen around experimentation.
00:14:57
Speaker
What are they to you, Tim? One of the ones that I often see is that experimentation is something that should be brought in when a company is a little bit more more mature. Yeah, i but you know what I, there's a few misconceptions. And i think the one that you brought up is is sort of my biggest pet peeve.
00:15:14
Speaker
If you were to, when people think about A-B b testing, they think about these experiments, such as like the famous, think Google had a 46 shades of blue experiment where they tested the, ah what color, what shade of blue the links on Google search should be.
00:15:28
Speaker
And they found out that this one particular shade lifted clicks and all therefore revenue by something like 1%. You know, when I think about what i What I don't like is that the poster child for A-B testing is all these big tech companies doing these experiments where they're micro-tuning little things and finding like 0.1% wins that are ending up worth like tens of millions of dollars.
00:15:52
Speaker
ah That doesn't do A-B testing any justice because it makes it inaccessible to the rest of us. It's like, okay, that's great for Google. That's great for Microsoft. That's great for Meta. ah But I'm just say normal startup or I'm just a tech company with just 10 million users.
00:16:08
Speaker
This looks really hard and impossible for me to do. So I hate those stories. So I think what I would love it is is if people start, if actually people realize that experimentation actually works even better at smaller scales.
00:16:21
Speaker
you don't need hundreds of millions or billions of users. You can get away with as little as like 10k. I've seen even like 2k, 3k user experiments do just fine.
00:16:32
Speaker
It all boils down to the effect size you're looking for. If you're looking for the 0.1% wins of like micro tuning your shade of blue, that only works if you're a Google. But if you're a smaller company, you obviously, I think 0.1% wins are not what you're looking for.
00:16:45
Speaker
you're looking for 5%, 10% wins. At that scale, you actually need very, you only need like 10, 100K users to be able to detect those. So that to me is like the biggest misconception is how many users you need and who should be doing experiments.
00:16:59
Speaker
Once you realize that that sample size is not a blocker, that actually means Experimentation should just be a standard like way of rolling out almost every single thing every single thing you build.
00:17:10
Speaker
Because you want to be able to... It's just measuring what you're building in ah in a statistically rigorous manner. And who wouldn't want to do that? ah You can confirm the wins that you think you're you're getting.
00:17:21
Speaker
You can look for unexpected things. The great part of experimentation and what makes it really, really powerful... is the unexpected surprises. If all A-B testing ever did was sort of confirm to you that your ideas were right, I think that's not useful.
00:17:37
Speaker
I mean, maybe maybe it's the ego boost you would get from it. But where where the real power in experimentation is, is being surprised. And you would be, most people, if you're unfamiliar and you're just going to experimentation for the first time, you start to realize a lot of your ideas are not, never as good as you thought they were.
00:17:55
Speaker
And so it's very humbling, but also gives you a proper tuning. You can start understand like, okay, okay, moving user behavior is actually really, really hard. But then there's also things that are unexpected to happen. And those are where where you start to realize like, hey, I can actually rethink how users, my users are using the product.
00:18:12
Speaker
I can also find other bugs and ways to turn a, perhaps you have a feature that lifts some very key metric by 4%, but there's also a bug in there. You can now turn that 4% into a 5%. by fixing that bug.
00:18:24
Speaker
Experimentation is really good at elucidating some of the finer details so over what's going on. Well, you said one of the things that, you know, is like about a val but data culture and I suppose an experimentation culture, Tim, what what are the foundations for yourself to to to build a great data and an experimentation culture? And you're going to get, I suppose, that foundational points, but also about how you actually go about implementing that because, you know obviously someone like yourself, you know, you're all you You're highly analytical. You're cutting from that that scientific background. But there are equally many people in the business that that maybe haven't come from the same background and don't understand the value of it. So,
Foundational Elements of a Data Culture
00:19:04
Speaker
yeah, it'd be great to hear about what are the foundations of a great experimentation culture and how do you actually go about implementing them?
00:19:10
Speaker
the The really good foundations of a data culture is starting, it starts off with just the data. your Your company needs to be collecting data and it needs to be collecting trustworthy data.
00:19:22
Speaker
If every time there's a data point, ah people are going to raise all these issues over like, hey, is this a logging bug? is it is this some Is this metric the same as that metric? That that makes it really hard to trust.
00:19:33
Speaker
the numbers you're seeing. So if you have a, if you started getting really high quality data into your system, um that's a prerequisite absolutely for a data-driven culture because data itself already has enough quirks that you don't need to be questioning ah the numbers as well.
00:19:49
Speaker
The second thing that you need for a data culture, i think is the, when you're starting to work in areas that folks have less conviction in and more uncertainty.
00:20:00
Speaker
so there's So what this can look like internally is you have ah half the room on one side of a debate over whether some we should take some action and the other half try and try to advocate for the other. And and then that meeting, it's just opinions.
00:20:14
Speaker
And so this is where like data, when it's brought forward, actually ah can be really, really important and shine. So that's sort of a prerequisite as well. When you start to work on areas where there is some amount of debate and uncertainty over what the right thing is.
00:20:28
Speaker
And the second thing is sort of an inquisitiveness, or sorry, third thing is sort of an inquisitiveness where folks just start asking really, really solid data questions or questions that can be answered with data.
00:20:39
Speaker
So that curiosity comes in, so folks are inviting data's perspective yeah into the discussion. That to me is sort of like the foundations for what it needs culturally as a people org for a data-driven culture to really start to establish.
00:20:53
Speaker
When you are talking about how do you go about building this, I think a lot of it should be intuitive. I think you you do need to invest some amount of like high quality work into high quality data.
00:21:05
Speaker
But I think naturally as a company evolves beyond the, hey, ah we've built our MVP, the one that we had really, really strong conviction behind. But now we want to start figuring out how we can.
00:21:16
Speaker
we've and we've And we've got some amount of users on board. it 1,000, 10,000 something like that? you can actually start interrogating that data and bringing the perspective of your users into using data ah into these discussions.
00:21:30
Speaker
How do you go about that that piece of the stakeholders? You said, obviously, when you start getting them, inviting them to to be part of everything that's like, so that analytics by design, having data and experimentation from day one, but how do you get to that point? How would you personally deal with maybe a difficult stakeholder who maybe didn't understand the value of of experimentation? because i think that's something that both data leaders, but also yes so seniors and practitioners can sometimes struggle with is that stakeholder doesn't understand the value that they can bring.
00:22:01
Speaker
Yeah, it may be great to hear some examples of how you might tackle a situation like that.
Engaging Stakeholders in Experimentation
00:22:06
Speaker
I see a couple of strategies that work really well ah for me and from other folks as well. The first is if you can find a situation where half the room, where where you are you have a split room, like, hey, we have a very critical strategic decision.
00:22:20
Speaker
Do we want to do A or do we want to do B? And half the room says we should do A. The other half advocate very strongly for doing B. And you have this sort of like a deadlocked room.
00:22:31
Speaker
And what usually happens in that case is that somebody pulls rank and says, like, we're just going to do this. That's a perfect situation to raise the to raise the hand and say, hey, maybe we can answer this with an experiment.
00:22:45
Speaker
And if you happen to think of a really good experiment to design that can inform this decision, that is sort of a perfect scenario for making experimentation look good. Because no matter what, the experiment is going to confirm half the room is right and is going to show to the other half that they were wrong.
00:23:01
Speaker
This is where you start to realize how important you ah how data can trump opinions and ego. Everyone in the room suddenly has seen the value of experiments in coming to the correct answer for this.
00:23:14
Speaker
So that's usually a really good situation if you happen to have that. It's like find the most controversial idea and just design an experiment to test it. The second one is you kind of need a unique opportunity to be able to do that strategy.
00:23:26
Speaker
um The other thing I also also tell folks is find a feature that or or something that somebody is working on that there's high conviction behind and there's no controversy and everyone thinks it's going to be like really, really great.
00:23:40
Speaker
Try asking people how good is it going to be? What is what metric is it going to move? That is actually just a very basic question most data people should be asking is, hey, we're working on this.
00:23:51
Speaker
what does this what What does good look like in terms of metric movements? so What are we trying to do? Are we trying and to improve signups? Are we trying to improve revenue? Are we trying to improve retention? There should be a metric that's, it's it's a good exercise ask your org what metrics is this feature going to be moving.
00:24:08
Speaker
ask them to fight for getting an A-B b test set up and then use that situation of like of an uncontroversial feature to suddenly ah be able to look at the data and be able to point out one or two unexpected things.
00:24:22
Speaker
And that's also like ah another hidden power of experimentation is it can point out things that you were not expecting to see. So perhaps the metric lift wasn't as big as folks expected,
00:24:33
Speaker
Or perhaps that when you carved up of for this new feature that we're building, um which users are benefiting and which ones are being harmed? um That's another great way to surface some data that you wouldn't normally ah be able to collect.
00:24:46
Speaker
And then you could also then ask yourself, why is this the case? Why is it users with slow internet connection are really hating this new feature ah versus folks a good internet? And is there anything we can do to follow up and make this better for everybody?
00:24:59
Speaker
So that's the other ah way is that use use experimentation as a way to take an uncontroversial feature and generate some insights that people were not expecting ah to be able to see.
00:25:11
Speaker
I love that one. suppose particularly the dividing of the room and that also sets the tone for for the future, right? Is that actually sometimes you're going to be wrong, but ultimately I think it's it's helping the stakeholder understand that This A-B test, this is going to show us what our customers actually want.
00:25:31
Speaker
And so that's ultimately would be well what what most companies are there to deliver. They want their customers to act in a certain way or they want to them to have a certain experience, which is, as you said, going so it's new to move a metric. And as much as people like to believe they've got the right answers, the data is who will be the only thing that can really show you what that customer is actually doing at scale. Yeah.
00:25:53
Speaker
Do you know the fact I like to to bring out is that to um making products, making successful products in this world is actually really, really hard. And I think like we don't sit back and like actually reflect on that.
00:26:06
Speaker
If it was really easy to make successful products, everything everything we built and every company that gets started would be successful. But it's actually really hard. And what that means is that as human beings trying to like steer a company or steer a product,
00:26:19
Speaker
We are often, we are wrong a lot of times and we are right some of the times. And experimentation is a great way to be able to help you identify which situations you're right in and which ones you're wrong in.
00:26:29
Speaker
And also is a great way took to calibrate and rethink your mental model for how your product works and how your users are engaging ah with your product. Diving a bit deeper into actual experimentation, Tim, how do you ensure, suppose, first off, that an experiment's designed well and and is executing you what you want to execute, but then also is interpreted correctly and actually acted on as well? Because I think that's the you know one of the the key parts. You can
Running and Scaling Experiments
00:26:59
Speaker
run ah you know and a well-class experimentation, but then how do you ensure it's actually driven and acted upon?
00:27:07
Speaker
i you know There's plenty of folks out there who will talk about all the prerequisites and all, and what does a running a good experiment look like. i often tell people when they're starting out, yeah, there is like a totally a best practices and a playbook you should be following to set yourself up for success and to run a really good experiment.
00:27:28
Speaker
But what I don't want that to be is a way for people to say, wow, this is really hard for my first time. And then they end up talking themselves out of running an experiment. And so what I usually tell folks is it's probably better to run an imperfect experiment than to not run an experiment at all.
00:27:44
Speaker
And so with that said, there's like sort of a bare minimum you need. For me, the first one is you should have some idea of what you expect your test to show. And what that means is like there's it actually means you need a hypothesis, but I don't want to use that fancy jargon.
00:28:01
Speaker
You should just Listen, you changed something. What did you expect your users to do with that change? And how do you expect that to be measured? Because you will need to measure it for an A-B test to be successful.
00:28:13
Speaker
You will need a metric and you will need some sort of expectation of how it's going to move. Then I would also ask the question of like, what else do you expect to not change? And you can put things like, hey, like just some very basic guardrail metrics are really strong. Like, hey, crash maybe we should check crash rates.
00:28:30
Speaker
um We don't expect crash rates to regress. um Let's put that in. Put some metrics in that it would be nice to look at. And then just run the test. ah Run the test. um Take a look. ah The first few times you run it, you will find out you may have done it incorrectly.
00:28:45
Speaker
And that's fine. It's a learning process on how do you iron out all these little kinks. Hey, like we didn't get this metric plumbed in correctly. Or boy, it would have been nice if we had this other metric here to answer this question.
00:28:58
Speaker
These are all things that like as part of the experience of like sort of like riding a bike. You just have to, you get better and better as you go at it and you won't get it right the first time. And that's fine. It's a bit of a journey to do that.
00:29:09
Speaker
I suppose, and then how's it acting for? And so that I think that's great advice to, you know, you're better off running the experiment and it being slightly wrong than than not running it. And then you've done your experiment. I think this is something that I see, I suppose, as just data professionals struggling with in general, that you've done your data work, but now how you actually help and ensure the business are acting on it in the right way?
00:29:32
Speaker
I think ah every experiment needs ah an experiment owner. So somebody that's sponsoring the experiment. That person should be in charge of also doing the readout and making sure that folks that that information is shared widely and is done to to a reasonable standard.
00:29:50
Speaker
I think by just shedding light on data, and putting it out in front of and publicizing it, it's really hard. If you have data that that's pretty strong, it's really hard to ignore.
00:30:01
Speaker
That's the nice thing is that data often sells itself as long as you put the light of day on it. And then if somebody is actually making a decision that's counter to what the data is showing, I would call that out and force a conversation because I find that the nice thing about A-B b testing data is it's really hard to hand wave and discount it.
00:30:22
Speaker
So that is sort of my advice for folks starting out if you're at an org that's not traditionally data-driven over like just put it in front of people and ask them, why are we making this decision if this data says this?
00:30:34
Speaker
I think standing there for us to there is, is, is really important. Making yourself heard and making sure yeah, that you're, you're pointing people in the, the, the direction of, of the data.
00:30:46
Speaker
In your experience team, you've obviously, you know, I had stats that you, it sounds like you work with many, many different teams with the tool and and how they do experimentation. What have been some of the, the things that you've seen that say, I suppose most teams overlook when running experiments?
00:31:02
Speaker
What separates good from great experimentation is really about scale. Folks usually on their experimentation journey will run one experiment and then go great.
00:31:14
Speaker
And then folks, if done correctly, folks will will say, okay, let's get another experiment going. And then they'll another experiment going. It's very much a handcrafted process where somebody, some individual data analyst or data scientist is crunching the numbers, but that's limited by the by the number of data scientists you have.
00:31:34
Speaker
So at some point in time, that person will want to figure out how do you scale this so we can actually run maybe two experiments at once or three experiments at once. That's when you start coming up with things like, hey, maybe we should template this into a notebook, into like ah into a Python notebook, and be able to run it.
00:31:49
Speaker
Folks will template it as much as possible, but there's still often a lot of tweaking that's needed for each individual thing. But that's the point where you can start running sort of like a handful of experiments at once.
00:32:00
Speaker
To go past that though, it's it's really hard ah to get into the dozens of experiments at once. You will probably need a some sort of formal tooling, whether that's in something your company has invested in and built that could be that might require like two engineers and a data person um to build build such a system.
00:32:19
Speaker
Or today, obviously, as somebody from Statsig, I would advocate buying a tool today. You're going to find it's far cheaper. You're going to find that it's going to have a lot more bells and whistles than what you can build internally.
00:32:30
Speaker
There's a lot of really good and experimentation platforms out there. So not not even just Statsig, but there's some other good ones as well. So once you get that, I think now the bottleneck is going to be how much ideas you have? And this is when going from good to great really is.
00:32:46
Speaker
What great looks like is we are testing everything. We are testing the big strategic experiments that have a really critical hypothesis. We are testing rollouts of new features, but we're even testing bug fixes.
00:32:59
Speaker
And this this was like a big highlight for me when I was at Meta. was I would see somebody would have like a small bug fix and it would be really obvious like, hey, this like there was a typo in the code, we fixed the typo.
00:33:12
Speaker
Of course this is gonna be good, right? And they would ask me, we run in do do I have to run this behind an A-B test? And I remember thinking, yeah I told the person how hard is it to set up a test and he said, ah well, it's it's probably like a couple clicks in the UI.
00:33:28
Speaker
And I thought it like, well, why not? Let's do that. And so the the individual set up this bug fix behind an A-B b test. He ran the experiment for only two days of data and he brought it back to me and said, well, well what do you think?
00:33:40
Speaker
And then we walked through and we looked and it absolutely fixed the bug. We could definitely see that. but What we also saw was another metric go negative. And I remember thinking, hey, why is this happening here?
00:33:52
Speaker
And we looked closer and we actually found out that there was another bug that this one had actually introduced. It's one of these times like you fix a bug upstream, you funnel users down lower stream and then hit they hit another bug.
00:34:03
Speaker
And I remember looking at him going, wow, we almost didn't run this A-B test. This would have taken us probably six to nine months to have found this other bug if we hadn't run this test. And that's really highlighted to me the power of like everything should just be tested.
00:34:17
Speaker
You're really bottlenecked by your tools though. And if as long as your tooling makes it really, really simple to run an experiment, almost trivial, that's where you want to be. That's what great experimentation looks like.
00:34:28
Speaker
That makes so much sense. There's such an and an interesting example as well, because obviously, you know, there's plenty of organizations and teams out there that would never even dream of doing an experiment on something as simple as that. But it it makes sense when the trade-off is is so simple, right? What's a few clicks going to take out my day and out of what I need to to invest in? And and obviously, it's showing that you know, things can have on unseen consequences. And and obviously, as you said earlier, you know, it's finding the unknown and the the surprising things, which is the exciting get about experimentation.
00:35:03
Speaker
How do you get there is say its is, I think, is that where a tool like Statsik can really come in and and how many experiments could Statsik help a company companyfy run? You know, could you go from 20 to 100 to experiments? so Yeah, we, are largest customers are running hundreds of experiments simultaneously.
00:35:25
Speaker
And so over the course of a year, they could have 800 to a thousand experiments down, maybe even more ah than that. So there are experiments, there are companies that are in the, when we talk good to great, that are in the great category, ah in my opinion, that are on Statsig. We also have many folks who are just starting out um their experimentation journey.
00:35:44
Speaker
I think the key value that we're delivering for these folks is that we will put you on this journey. And we will do it in a very cost-effective manner and make the experimentation as accessible as possible.
00:35:56
Speaker
The bottleneck shouldn't be your tooling. The bottleneck should be how many ideas you have and how much engineering bandwidth do you have to be able to run these tests. But you should never have problems doing things like randomizing your users, collecting data, or crunching the results.
00:36:11
Speaker
Those are the things that we make as accessible and and easy as possible. When it comes to, obviously, well as you start to scale, Tim, you come to, and we sort of have covered this up, I suppose, but the last you scale, how do you decide and how you prioritize which experiments to run when maybe resources are are limited and you aren't a meta who can just do a few clicks and and it and it works? what What's your approach to to companies that may be looking at ah focusing in on a more limited amount? How would you go about prioritizing it?
00:36:44
Speaker
That's a great question. the For prioritizing experiments, you if if you are limited on your sample size and number of users or even engineering bandwidth, like these are sort of the normal constraints that you have, you usually would prioritize based on what is the what is the impact you expect for this experiment or how controversial is this idea? There are obviously some features that you're working on, such as bug fixes, where without data, you probably would still go forward with it.
00:37:14
Speaker
But then there's going to be some key things where like, hey, this is actually pretty controversial. We need data for this. So those ones you definitely want to prioritize. The other category I would go for are things that are big changes. so These are the things where it is actually really important to know what the unexpected sort of like, hey, is this change as big as we thought it was?
00:37:33
Speaker
Is it working? Are users behaving as we expect? For example, if we've increased the number of signups, what happens to those users after they sign up? that would be good to get in terms of ah and an A-B test. Are these high quality users that are like engaging, paying, you know, whatever it is your objective, your app or product is trying to drive?
00:37:52
Speaker
um or or are they low quality? These are things that an A-B b testing ah can help you pick up. So I would go by impact both from a strategic perspective, which ones have the ability to change the direction your company is going to be building in, or impact in terms of how big of a change are you putting in front of the user?
00:38:10
Speaker
Clear, simple advice there, Tim. I think one of the things I know you're also passionate about is obviously around this building of of trust and collaboration um with the commercial and um products teams.
Collaboration Between Data and Product Teams
00:38:26
Speaker
How do you go about building that and how do you go and make sure that you are having that conversation from day one with with the product teams and how they seek this by design um part of that look at the conversation?
00:38:40
Speaker
Yeah, this this strikes to the heart of an issue that bothers me greatly in the world of data. um data A lot of data teams at some companies are sort of very much siloed from the rest of the organization.
00:38:55
Speaker
Metaphorically, but maybe physically true, is there may be actually like a floor that says like, this is the data floor, this is the analytics team. And they're completely separated and isolated from the rest of the company. What ends up happening is that they come up with all these great work and insights and then they have nobody to deliver this to.
00:39:12
Speaker
Either no one's listening or these things are just the insights are sort of not running lockstep with the rest of the company. Like you're not generating insights that are useful in a timely fashion.
00:39:23
Speaker
You may be working on problems that folks were the we're thinking about six months ago, but not today. for example. So I think it's really important to get close to the heartbeat of the company.
00:39:34
Speaker
Find a product team stakeholder, usually a PM or engine manager, and understand from their perspective what are they trying to do? What are their goals ah for their product?
00:39:44
Speaker
And then what are their, how are they going to get there? And what are the biggest questions in the back of their mind? That's my favorite question to ask like a PM is sort of, what what are the biggest questions you have right now?
00:39:56
Speaker
And don't don't ask them like, hey, what's the biggest data question you have? I like to leave that, ah how do you answer that using data to up to the to a data expert, but just go with what are the biggest questions you have?
00:40:07
Speaker
um What are the things that if you if you knew the answer to would be the most helpful? You can take that information and convert that to data questions. And that's how you can sort of stay lockstep with like, what are the decisions being made today? um What are the decisions that we would like to make if we had data there?
00:40:25
Speaker
And that's how you make sure that the data team is doing things that are useful. When it comes to A-B b testing, you need a little bit more lag time as well, because it does require some amount of time to collect it, to run the test, to collect the data.
00:40:38
Speaker
and get the results. This is where it's really important to be um doing something that's timely with and and in the same timeframe that a critical decision is going to be made.
00:40:49
Speaker
And so in order to do that, you have to be talking to the, ah who are your, basically the consumers of your data. You need to be making sure that like, hey, this is actually a question they want answered. I love that, Tim. b Asking the the right questions is so, so important. um and Even that small detail that you picked out about how you framed that question, and I think is really important because the questions are what uncover the the why behind what then that they may be coming and asking you in the first place. And and I also love how you said, be proactive and go find these people and
00:41:24
Speaker
and just unleash that curiosity on them. But without making it too technical, we without making it about data, just keeping very much in the in in the context of what are you trying to solve?
00:41:36
Speaker
Yeah, a really good data person, and like in my opinion, is the one that can translate a business question into a data question. Don't put that burden on your stakeholders. You should be the expert on how to convert a normal business question into a data one.
00:41:49
Speaker
And think about what what the right business question is. i how how'd you How do you get the stakeholder to tell you the right business question? I think there is another one which, you know, don't just jump in and and think about how you're going to pull all of that, that and the rights information out of the stakeholder. think it's really important as well.
00:42:08
Speaker
Yeah, I agree with that. When it comes to this, we're almost um at at time now. So I suppose final things before we go into your final thoughts would be to how do you keep make sure that the team is aligned on that business context? And what advice do you give to your team to ensure that they are asking these its right questions and they are being a true business partner?
00:42:34
Speaker
Yeah, I think at the end of the day, it is about the decisions you're making. And it's ah what i I've noticed is that some data teams can get in this habit of like, hey, keep drilling more rigorously and only generate like really high quality insights.
00:42:49
Speaker
I think you you have to figure out what is the business problem that folks are trying to solve and how can I help unblock them now? as fast as possible. Now that means generating data that is ah that you're 80% confident in, it's way better to do that in a timely fashion than to generate data that is that you are like say 100% confident in, but deliver it six months late.
00:43:13
Speaker
Like that would be bad. So I think you need to be making sure you're lockstep. You need to be making sure that the what the insights you're generating are useful ah by the time you are able to produce them.
00:43:27
Speaker
I think that's just, that's an org issue that's communicating with folks. you know Meta solved this by making sure that data people actually didn't, we we sat with our ah teams. So I usually sat next to engineers and PMs.
00:43:42
Speaker
I didn't sit sit next to data people. the The data team was really dispersed and they sat with their individual teams. It was not possible for me not to know what folks were working on. ah just because of the physical setup.
00:43:55
Speaker
But also of your day-to-day is meeting with folks that are in your media team. That's how you stay tuned to what the business context is. But you also have to be aiming upwards, like where are the decisions being made and making sure you are aligned with those folks. so Excellent. I feel there's great lessons there.
00:44:12
Speaker
If you're a practitioner, make sure that your world is focused in on what your PMs, what your wider product team is focused in on. And if you're a leader, maybe thinking about how you can structure your teams and what the ways of work can ask to really encourage and foster that type of behavior and that close partnership as well, because that can massively set up your team to success as well.
00:44:40
Speaker
Well, Tim,
Conclusion and Final Advice
00:44:41
Speaker
that's stuff story's wrapping up there. But before you go, what's your final piece of ah advice, i suppose, to to building a great experimentation and and data culture? If you want to re recap some of your key key points.
00:44:56
Speaker
Yeah, you really should just try. it is way better to run a flawed experiment than to run a perfect experiment. Sorry, it's better to run a flawed experiment than to have not run an experiment all.
00:45:10
Speaker
Don't try to strive for perfection. This is a little bit like riding a bicycle. You need to be practicing it as you go. I think I also encourage folks, for like you especially like in the good to great conversation, like Reconceive your notion of what an experiment is.
00:45:24
Speaker
It doesn't have to be the greatest A-B test, doesn't have to be the most strategic one. Sometimes testing bug fixes is okay. Sometimes just testing big major changes that that people have high conviction behind is also useful. So I think my just my general theme is just experiment.
00:45:41
Speaker
ah You'll be surprised. In fact, that's the goal of experimentation. You'll be surprised at what you might find. Excellent, Tim. It's been a pleasure to have you on the show. Thank you for for joining me.
00:45:51
Speaker
Harry, thank you so much. This was a pleasure too. Excellent. Well, that's it for this week, folks. We hope you enjoyed the episode. um Feel free to um leave any comments. And i'm I'm sure Tim will always welcome a a connection or a follow with you. You're keen to stay up to date with ah the stuff that StackSegger building and what Tim's sharing with his expertise in in the space. So we'll see you all soon.
00:46:15
Speaker
Thank you very much.
00:46:18
Speaker
Well, that's it for this week. Thank you so, so much for tuning in. I really hope you've learned something. ah know I have. The Stack Podcast aims to share real journeys and lessons that empower you and the entire community. Together, we aim to unlock new perspectives and overcome challenges in the ever-evolving landscape of modern data.
00:46:40
Speaker
Today's episode was brought to you by Cognify, the recruitment partner for modern data teams. If you've enjoyed today's episode, hit that follow button to stay updated with our latest releases.
00:46:50
Speaker
More importantly, if you believe this episode could benefit someone you know, please share it with them. We're always on the lookout for new guests who have inspiring stories and valuable lessons to share with our community.
00:47:02
Speaker
If you or someone you know fits that bill, please don't hesitate to reach out. I've been Harry Gollop from Cognify, your host and guide on this data-driven journey. Until next time, over and out.