Transcript

Speaker: Hello and welcome to the Stacked Podcast, brought to you by Cognify, the recruitment partner for modern data teams. Hosted by me, Harry Gollup.

Speaker: Stacked with incredible content from the most influential and successful data teams, interviewing industry experts who share their invaluable journeys, groundbreaking projects, and most importantly, their key learnings.

Speaker: So get ready to join us as we uncover the dynamic world of modern data.

Speaker: Hello everyone, welcome back to another episode of the Stacked Data podcast. Today, we're diving deep into ABE experimentation at scale, how companies can leverage rigorous testing to drive product decision, improve customer experience, and create real business impact.

Speaker: Joining me today is Dimitri, VP of analytics at Flow, one of the world's leading health and wellness apps. Flow's built a data-driven culture that embraces continuous learning and innovation, running over 1,500 experiments to refine and enhance their product.

Speaker: In this conversation, we're going to explore the fundamentals of experimentation, the journey of scaling from early stage testing cycle to full-fledged experimentation frameworks and the challenges that comes with running hundreds of tests at once.

Speaker: We'll also discuss how Flow's balanced speed and and quality in its experimentation approach. And and here's some real world examples of how data-driven insights have really shaped their unique products.

Speaker: So if you're thinking about building or or scaling experimentation in your own company, this is definitely an episode not to miss. Excited to get into it. And yeah, welcome to the show, Dimitris. A pleasure to have you on How are you doing?

Speaker: Thank you, Harry. Very good. All good. Thanks for inviting me. Happy to be here. Actually, I think this is the first podcast for me. So yeah, excited. Brilliant. Well, ah yeah, honored that you're this is the first one and excited to to get into it. I've loved learning about how you guys have scaled experimentation to a level I haven't really seen ah at any other organization, maybe other than your metas of the world.

Speaker: But yeah, I suppose it would be great for the audience to to give a bit of context to yourself and your career and your current role at Flow. Yeah, sure. So let me start from myself.

Speaker: I'm Dimitri. I'm originally from Belarus and recently moved to London. I've worked at Flow for... more than six years now and started as, let's say, founding analyst there.

Speaker: And now I'm VP of for analytics with a 30 person team of brilliant people responsible for product marketing and finance analysis. And if we're talking about Floor, perhaps not everyone knows about Floor.

Speaker: So basically it's number but one women's health app. Initially started as period tracker, but now it's more like a platform that combines period tracking, like other modes, trying to conceive pregnancy modes with personalized health insights.

Speaker: And yeah, having, you know, around 75 million users globally, it's our monthly active audience. So we are pretty huge. Amazing. Yeah. I mean, the scale, I think is what's going to be most impressive about this episode, Dmitry. So let's jump into it and let's start with the basics. yeah can Can you explain AV experimentation is in its simplest terms and why it's so valuable for data-driven sort of product companies?

Speaker: Yeah, sure. So A-B testing or experimentation is basically a way of testing product or marketing changes when we don't roll out a specific change for everyone once it's ready, let's say, but we preserve a control group, usually one control group that doesn't receive this change, this feature for some time.

Speaker: And then you measure the difference in target metrics between these groups, basically usually test group and control group. In a nutshell, it's, it sounds very simple.

Speaker: I can explain why it's valuable actually. Two main things I can mention here. I suppose there are, there are many more that can be mentioned, but I would focus on only two.

Speaker: First is velocity because experimentation basically helps you to parallelize product changes, to chose only those changes that really impact user behavior and other business goals positively.

Speaker: And basically running experiments gives you ability to scale your growth tenfold. Recently there was a post by our CEO Dmitry Gursky and we are running around 200 experiments simultaneously during a day.

Speaker: So it speed ups us a lot. And actually I would focus here that we embrace parallel testing a lot because without parallel testing, it doesn't really help to speed up things.

Speaker: So I know there are some people that are against parallel testing, but we embrace the pros of it. And the second valuable thing in experimentation is accuracy, accuracy decision-making that you observe and gives you the most precise accuracy in estimating the impact of, you know, some specific features on user behavior.

Speaker: So, yep, shortly, this is it. We can go deeper into each. So I suppose it's in a nutshell, it's the ability to compare two different features to see which one's going to have more impact on business, on the product, on the customer.

Speaker: It enables you to make data informed decisions on on what product direction, what feature change. How did it all start for Flow? You've obviously scaled to to running 200 tests simultaneously, but you mentioned you were one of the founding analysts.

Speaker: How did it start and where were the key motivations for having experimentation embedded within ah the product's lifecycle? Initially, we started in 2019, I think, and we already had massive audience, maybe around 20 million active users.

Speaker: And we started with some small experiments with content, with insights that we serve for our users. As far as I recall, it was some experiment experiments with our health assistant feature.

Speaker: It's basically was, it was some dialogues with users about health conditions or some, you know, feeling some insights. And we had some tests there, first tests.

Speaker: And experimentation actually helped us again to run things in parallel, ah basically testing more than one hypothesis at the same time.

Speaker: And we try to increase the number of people that finish a specific dialogue, virtual dialogue. And I think general experimentation was the the only, maybe not the only, but the best way here, ah because the overall impact on very high level metrics was rather low.

Speaker: Because not many users at that point were like reacting on these dialogues. But also like maybe even more importantly that we tried again a few different variations for specific messages and call to actions.

Speaker: choosing kind of the best that fits our goals and user goals. And basically the idea was to maximize the number of people that finishes dialogues and get real recommendation of how to, how to move forward with the issue.

Speaker: Then I think we also started to experiment ah with some small experiment with payment screens as well. The idea was to ensure that everyone understands what they're paying for and the information about subscription is clear.

Speaker: For example, sometimes users understand how the trial period works, but sometimes not, and we don't want to have many refunds after payments and that's why we decided to make it more clear and optimize the number of refunds.

Speaker: And yeah, it was pretty good in terms of overall reaction from the team. I think the team was impressed how quickly and clearly we can provide specific recommendation, what to roll out and how to choose a winner.

Speaker: So I think that was kind of the starting point of the future scale. Brilliant. so I suppose it sounds like you had some success and some wins off of some clearly monetizing areas of the product through experimentation, which I suppose has led to this increased sort of scaling heading journey that youve you've been on within the experimentation space.

Speaker: Now, I think the technical aspects of experimentation, which we will come to a bit later, but I think first and foremost is the the cultural ah aspects and ah getting that transformation in and and that ah experimentation culture, not just within your data team, but within your your stakeholders and having that from the center of what they're doing. So how did you approach sort of I suppose, getting buy in from different teams and stakeholders and building and enabling an experimentation culture at Flow. What are your strategies in that space?

Speaker: Yeah, very good question. First, look, setting and like setting experimentation culture is continuous process. You cannot set something once and use it forever.

Speaker: You need to improve it. And we always see some points how we can make it better. And it's a journey. Everyone needs to be prepared for that because otherwise, you know, it's a question of expectations.

Speaker: So i think culture is a good question because i think we started from education. were continuously investing and are investing significant amount of time explaining key concept of experimentation.

Speaker: Currently it's more focused on new team members, right? When someone joins and maybe comes from another type of company. And right now we are focusing more on team by team introductions and education when we can not just, you know, run through some general things, but also to provide some personalized examples, you know, that are more relevant for a specific team.

Speaker: If we're talking about content teams, it's how to measure content. If we're talking about growth team, it's how to measure user and revenue growth. So education is very important generally.

Speaker: And it's hard to to say when it became like that, but at some point of time, ah critical cultural evolution actually came from our leadership.

Speaker: when they started to replace more like opinion based discussions with statements like let's test it. And you know, that helped a lot because when you start to doing it top down, of course it helps.

Speaker: And one more, I think, additional factor here was that we had clear goals for teams. We use objectives and K results framework to set goals.

Speaker: And we also try to decompose these goals on analytics side to provide more clear vectors to the teams, how they can achieve their goals. Right.

Speaker: And once the team know what they need to move, experimentation becomes more like ah valuable tool instead of fancy statistical feature.

Speaker: So it's combination of our efforts and buying from our leadership and changing it top down. Brilliant.

Speaker: I think that you need to sort of a dual-prong attack, work with senior leadership and help them see the the benefits. And I suppose when they have a tangible example have seen real impact in front of them, then become yeah i think that's a great way to to get that buy-in. You can get that aha moment, um which I think is what those teams are ah striving for. And then it sounds like there was a lot of waving the flag around the rest of the business and explaining the benefits and and helping them see the value as well to get all the approach it from both bottom up and and top down to to get full penetration.

Speaker: And as you said, sounds like it's ah it's a constant and and evolving journey as well. Damesha, when it comes to, I suppose, best practices for experimentation? ah How do you ensure that product and analytics are both on the ah both on the same same sheet and on the same page when it comes to, I suppose, aligning what what the best practices are and how they should approach experimentation? Because think that's also a cultural challenge in itself.

Speaker: Well, yeah, definitely. I think when we started, just started, it was more like some evolutionary way of doing things, but the team grows and basically it means that, you know, different teams can run but little bit different things differently from process side.

Speaker: So I think we have a few... a few tools that help us to run it more in unified way. So for example, we use Jira for tasks tracking and in Jira, we have a special ticket type, which is called experiment. And basically it has ah specific process behind this ticket type.

Speaker: And all teams try to follow it. This helps a lot. And this was done with with the help of our product ops team that help to more or less standardize the process of running.

Speaker: Then we have template for experiment template documents with all important things to fill in. We can say that it's kind of checklist and so all the teams try to follow it as well.

Speaker: And I think third part, which is really important here, we have our in-house experimentation platform. that is basically used to run and analyze experiments.

Speaker: Again, we can discuss it in more details or you can find some some details about it on our medium because we have, i think, around two two articles about it.

Speaker: We can put some links in the show notes to the documents and the medium articles. Great, yeah. And then you'll be able to check how we work there and how we use experimentation platform.

Speaker: So yeah, basically to wrap up process is more or less standardized using Jira. Some important technical, analytical release specific things are standardized using ah Confluence experiment template doc and experimentation platform is, you know, unified way of running things and analyzing them.

Speaker: Excellent. And Dmitry, you mentioned there, obviously, about the ah platform. I suppose that's the the next area I'm keen to understand. just I suppose more about the technical aspects and the infrastructure.

Speaker: what What infrastructure and tools do Flow ah rely on and and how do develop them to to support scaling 1500 experiments?

Speaker: Yeah, so in 2019, we actually started from very simple tools and technologies, rather simple, I'd say. So by that time, we already had an internal user profile service.

Speaker: And this service helps to store all important user attributes for each user. And then then these attributes can be used to target features and experiments. And, you know, it helped us a lot because this experiment extension of this user profile didn't take a lot of time and it was more or less almost by design initially, and but a user profile is more like backend thing. And when we're talking about experimentation platform, it's not just backend, it's also front-end.

Speaker: So we have a team that, um, tries to basically the goal is to improve this experimentation platform.

Speaker: We started building it somewhere between 2019 and 2020.

Speaker: I think that in 2019, it was more or less early stage for the markets. So we decided to build its our to build our own platform rather than buying something.

Speaker: think right now you have more options to to choose from. But definitely this combination of two things helped us a lot. Two We, of course, have a lot of not a lot of, but more like specific dashboards dedicated for deeper experiment analysis.

Speaker: So yeah, there's boarding, user profile as targeting and experimentation platform for backend and frontend. These are three main things that we use every day.

Speaker: Brilliant. And I suppose if you were in the position now, it sounds like you've built a lot of your the technology to manage this internally. You alluded to it now, but if you were starting this journey now, would you would you still build this entirely or would you look to to buy? or Yeah, good question. So i'd say that I'd recommend any way to run some evaluation of build versus buy.

Speaker: We did it back in time, but again, it was more less early stage. And I think... Today, it's higher chance to buy something that fits your needs.

Speaker: Yeah. So depends on the context, depends on the company, but I think right now build versus buy becomes a little bit more complicated question because you have more options on the market, right?

Speaker: So I'd say this is inevitable to to run this type of analysis. Brilliant. I suppose yeah you you wouldn't know unless you're deep in the weeds and you don't need to to to deal with that right now.

Speaker: So what are watch some of the biggest challenges you faced when scaling to the the level that that you did when it comes to experimentation and and how did you overcome some of the challenges?

Speaker: Basically, I think I always can can mention a few. Let's try to focus on the most important ones. I think there are a lot of articles talking about balancing quality and velocity.

Speaker: And of course, this is ah common issue for everyone. Common question, open question, how to be So ideally we want to be as quick as possible and have the best quality of course, but that's not always possible.

Speaker: So it's always a trade off around it. Quality can be kind of a separate issue, not just in balancing, but more like, you know, to get trusted results.

Speaker: it's It's always can be a separate one. ah think, um, From not just experimentation as a tool, i can mention more like a thing that I think everyone is tries to tackle is overall strength of hypothesis that we are testing and of course, reporting around it.

Speaker: And maybe more related to our company, again, maybe to do little bit bigger companies is about unified approach for um ah running things to analyze them.

Speaker: It's always a challenge because teams are decentralized and they of course dependent one. from another, but it can be still an issue because they're a little bit isolated as well.

Speaker: So yeah, many of them we can dig deeper into specific life if you want. Yeah, I think the the balancing of quality versus velocity would be so i'd love to expand a bit more on how how do you make that trade off? you have a ah process internally that where you'd look to, I suppose, decide on what sort of trade off you're going to make? I think that that'd be really beneficial to understand. So what you look at to to make that decision.

Speaker: Yeah, so I think if we're talking about this specific trade-off, velocity and quality, generally we need to, we cannot we cannot slow down.

Speaker: so we optimize both things at the same time. How we ensure the quality, I think we do it um in different ways. so generally it's about unified methodology first. We need to get results that we can trust instead of, you know, getting some random methodology and saying that this is the correct thing.

Speaker: What else? If we're talking balance again, i think We run a lot of different small initiatives, for example, to tackle quality.

Speaker: Around two years ago, we started analyzing experimentation process as we analyze everything else. For example, where we have more failed or we call invalid experiments.

Speaker: Are there any specific teams that struggle with this issue more? Can we somehow group them into, you know, underlying reasons and then tackle reason by reason?

Speaker: Yeah, something like this. For Velocity, we're doing a little bit different thing. So, for example, last year we launched a little bit different process for evaluation of experiments.

Speaker: We call it... for some reason, experiment segmentation, but it means that delegate decision-making for simple experiments completely to stakeholders.

Speaker: And we provide tools for them to be sure that the results you get, you can trust. And these actually helps to speed up things.

Speaker: So yeah, doing a lot of things to balance this. Generally, we are paying a lot of attention to velocity and quality very important as well.

Speaker: So yeah, different directions, different initiatives there. Hope it was clear because yeah, I mean, like but there is no one specific answer to that, I'd say.

Speaker: sounds like um what with what you mentioned about sort of enabling self-serve users with tooling, it sounds like some the more smaller product changes and some of the, I suppose, that the smaller picture changes, which are going to have maybe a lesser impact. you You empower your stakeholders to to be able to run them at a high velocity.

Speaker: And then maybe some of these more bigger sort of transformational changes is where you really sort of double down on um whilst velocity is also important, but quality then becomes that that key key factor because it's going to have such a substantial impact. or is Is that the case?

Speaker: Yeah, exactly, exactly. And I can mention a few things, for example, how we ensure specific things, how we ensure that we are balancing them well. So, for example, sometimes when we need to roll out something quicker, we can use... um Swap testing, as we call it.

Speaker: ah Basically, it means that we launch something with a smaller test group. We rely on proxy metrics only. It means that we have some short metrics that possibly predict long-term outcome.

Speaker: Right? And when we see positive signals, we can swap the groups and basically have smaller control control group and bigger test groups. So test group with change.

Speaker: and leave it for some time to ensure that after long-term metrics are calculated, the results are the same. So I think this is more specific example, how you can ensure that you're testing something something quick, you're allowed something quick, but at the same time, you'll be able to recheck yourself, to recheck the results of a specific change and roll back if something doesn't go well.

Speaker: Amazing. Has there been an instance where you've slowed down and experimentation specifically to ensure quality? yeah Has there been, as say, these bigger sort transformational changes, or or do you tend to make lots more of smaller iterations to to the product? What's, I suppose, that that that decision-making process on how to assess whether it's making big changes, or do you just focus on making more smaller granular changes?

Speaker: So I'd say that we cannot slow down usually and we pushing hard to achieve results. What we actually do is, as I said, we started to analyze, for example, invalid experiments where we had some issues with setup or something like this.

Speaker: So right now we track this metric and we try to keep it a reasonable level, the reasonably small level. And once we see that there's something going on and the metric increases, we deep analyze it and try to fix it.

Speaker: ah Usually it's more like team or specific domain issue. And usually we can tackle it separately. So yeah, we we don't slow down usually. Yeah. But again, we're we're trying to at the same time monitor quality and run some actions to bring the quality back on the level that is acceptable for us.

Speaker: that's That's great. And it sounds like it's become core to to the so the culture and it's all systems go when it comes to to to experimentation. How, Dimitri, have been able to, i suppose, structure your teams in a way that is going to...

Speaker: effectively enable this quick delivery of of experimentation. So I think that's something which i know a lot data teams and leaders I think will be would be interested in in understanding.

Speaker: ah Well, I think this is not just about data or analytics team. I think first it's more about engineering or marketing or something else that helps you to launch things quicker.

Speaker: So I didn't mention it, but I think we've done a lot of actions to help you to actually run things much quicker than before.

Speaker: We have, I think, very high percentage of no code experiments, which are set up through some configs, through some specific things that don't require coding.

Speaker: And it means that, um you know, you can test things without involving engineers for full time. This is, I think the first important things that everyone can follow to ensure that you are quick enough.

Speaker: Then I think we try to decentralize teams and decentralized teams. We're talking about product marketing, basically all the teams.

Speaker: They have their separate goals, their separate objectives and key results, which of course contributes to the company objectives and key results. And if we're talking about analytics team, we use embedded approach.

Speaker: So team members are actually embedded into specific domains and they work there for 80-90% of their time.

Speaker: They help with goal setting and they actually share these goals with the team, which helps to be more focused on you know achieving these goals.

Speaker: And this decentralized embedded structure also helps to be in the full context of what team is focused on how you can help.

Speaker: So yeah, it brings another level of cooperation and collaboration between team members. Yeah, basically, I think these are the main main points I can mention right now.

Speaker: But route there are some smaller things, of course, but I would focus on these ones. Excellent. And you mentioned yeah that you're running 200 experiments simultaneously.

Speaker: How do you coordinate that? How do you avoid conflicts, maintain clarity? And I suppose to compound the question, Dimitri, how do you ensure you're actually getting valuable learnings and effective measurements from these experiments all at once?

Speaker: Yeah. So first, um, parallel testing is not bad. Everyone, again, uh, some people think that it's, um, risky.

Speaker: It might be in some way that's true, but parallel testing from my perspective has much more pros than cons. But of course it requires some additional actions and things are to be sure that, you know, you're not affecting other teams, for example.

Speaker: So I think our teams are focused, they work on some specific feature usually. or a set of features, and that's why they control what they launch in this feature, right?

Speaker: So we usually try to avoid conflicts within a specific feature. So some parallel experiments can be run if they only intersect you know evenly and do not affect one another.

Speaker: But it means that, yeah, sometimes we need to test something more like we are testing one and then we are testing another separately just because otherwise it may ruin user experience.

Speaker: We also try to, again, it's more like a lesson learned. We try to align between teams. And if someone knows that something may affect other teams, they usually align with these teams that, i don't know, there might be some effect of cannibalization or there might be some, you know, redistribution of user attention.

Speaker: so again, it's it's more about... how people start understanding what they can change and how they align with other teams on thiss on this.

Speaker: But yeah, sometimes we havet we have some issues with that as well, but this more like rare case, I'd say. Brilliant. and Well, it's been great to hear about how you you've tackled the mammoth task and clearly it's had huge amounts of of impact for you guys at at Flow. I suppose before I let you go, Dmitri, it'd be great to understand what's a piece of advice you would give yourself if you were just starting this journey again. Yeah, so i think the first is...

Speaker: I'd say that it's a long process, but don't worry, you'll come over it. But it's a process. It's not just one time thing that you made and everything becomes very good.

Speaker: But generally, if being serious, I would say that I may have two pieces of advice for everyone that thinks about experimentation and for me as well from, i don't know, 2018.

Speaker: So use parallel testing to speed up your velocity and it actually will help you to speed up everything tenfold if done in right way.

Speaker: And i think second, it's not directly about experimentation as the tool, but more like general advice is constantly push for strong hypothesis.

Speaker: Because again, experimentation it's is just a tool to choose the best from all. But if you can ensure that overall level is higher, your gains will be better as well. So yeah, push for BetSix supporters as well.

Speaker: Sure, but I hope that's helpful. For me, definitely. No, it sounds it. It sounds it from the and the journey that you've been on. there It seems that that's been key to your your success. So finally, Dimitri, what's next for Flow? Looking ahead, what's um what's what's on the roadmap for for your data journey?

Speaker: A lot of things. Um, well, how we can not mention AI, for example. So yeah, I think we are thinking of adopting AI for decision-making automation.

Speaker: As I mentioned before, some experiments currently are completely managed by stakeholders, but why not to delegate it to AI? Let's check.

Speaker: Maybe that's possible and this will actually speed up everything a lot. What else? I think for quality, we are adopting maybe not new approach for some companies, but definitely new for us. It's meta analysis of experiments.

Speaker: It's when you take corpus of experiments that were, for example, run for six months or something like this, and you try to get insights from it.

Speaker: i don't know, which domains are the best, which hypothesis types are the best, what we should focus on and what we should avoid from doing. So, yeah, this is one of our big bets, definitely. and We are also trying to improve quality through improvements in statistical methodology.

Speaker: So it might be actually a good part to improve both velocity as we will be able to stop experiments earlier, but at the same time, it will give us better quality as well. Alignment metrics, it's always and ongoing issue.

Speaker: And generally we are also switching to experiment as a code approach, which should ensure and improve our quality as well.

Speaker: So yeah, a lot of things. Basically it's, let's say for a year, around a year for us. AI thing may be for a longer time, let's see, but everything changes and um ongoing yeah ongoing, maybe in a year I will give you a completely different answer to that question.

Speaker: Oh, well, brilliant. um I think AI is on everyone's roadmap and finding the right use cases for how they can leverage it in the in the right way, I think is going to be key.

Speaker: So yeah, sounds like exciting times ahead. And and thank you again for for joining us on the pod. Hopefully you've had a a good first first experience. And yeah, thanks for sharing your insights as to how you guys have scaled experimentation at Flow.

Speaker: Thank you, Fary. Thanks for inviting me. Yeah, I think, I hope it will, it is going to be helpful for some teams that are just starting it. And yeah, for someone who just listens to that, don't hesitate. You can ask me something directly.

Speaker: I can help as well. Brilliant. Well, look, we'll put link to Dimitri's LinkedIn. I'm sure if you're an experimentation with as well, there's a lot of challenges that Flow are facing the that I'm sure you could get involved in.

Speaker: Dimitri would be happy to to answer any questions on experimentation or life on AppFlow, I'm sure. But thanks, everyone, for joining us. We'll see you again in a couple of weeks.

Speaker: Bye-bye. Bye-bye.

Speaker: Well, that's it for this week. Thank you so, so much for tuning in. I really hope you've learned something. ah know I have. The Stacked podcast aims to share real journeys and lessons that empower you and the entire community. Together, we aim to unlock new perspectives and overcome challenges in the ever-evolving landscape of modern data.

Speaker: Today's episode was brought to you by Cognify, the recruitment partner for modern data teams. If you've enjoyed today's episode, hit that follow button to stay updated with our latest releases.

Speaker: More importantly, if you believe this episode could benefit someone you know, please share it with them. We're always on the lookout for new guests who have inspiring stories and valuable lessons to share with our community.

Speaker: If you or someone you know fits that bill, please don't hesitate to reach out. I've been Harry Gollop from Cognify, your host and guide on this data-driven journey. Until next time, over and out.