Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Building fast beats building perfect: How AI accelerates experimentation velocity | Akash Doshi
6 Plays2 days ago
Most enterprise leaders are telling teams to just use AI. Akash Doshi thinks that mandate is missing the point entirely.
Akash and Katie name the real bottleneck in experimentation: an administrative tax that buries good hypotheses in the backlog. They show how AI shrinks a two to three month test cycle to one to two weeks, so backlogged hypotheses actually ship.
Akash spent three and a half years leading digital strategy at Delta Vacations, Delta Airlines' leisure arm. He is now the founder of Sequence, AI middleware built to turn scattered AI experiments into repeatable workflows.
Key takeaways
- Identify the customer problem first, then work backward to find the metrics that show whether your solution actually works.
- Hand developers a working mockup, a PRD, or a V1 build before looping them in, so they can focus on harder problems.
- Ship a fast, imperfect version of your test now, since real customer feedback corrects your hypothesis faster than another sprint of planning.
Recommended
Transcript
AI's Role in Market Entry and Decision-Making
00:00:00
Speaker
think now AI experimentation like de-risks you a little bit, right? You're able to build out a test plan, get it into production. If it wasn't right, you've now narrowed down what your North Star should be. And I think the same, so like the different side of that same coin is that the velocity increases, right?
00:00:17
Speaker
You're able to get to market quicker. And that's like probably the thing that I talk about too much, which is that I think in 98% of scenarios, building fast is better than building perfect. The reason that's true is that when you're able to get to market quicker, those feedback loops activate, you're able to hear from the customer. And that really allows you to validate your guesswork. Teams build faster, and they're asking better questions as a result of building faster because they know what their customers are telling them.
Introduction to 'Unite Voices' and Guest Akash
00:00:47
Speaker
Welcome to Unite Voices, hosted by Katie Green. Real stories from the people behind today's most innovative experimentation programs. No fluff, just wins, failures, and the lessons in between.
00:01:01
Speaker
Well, Akash, welcome to Unite Voices. We're so happy to have you. For everybody who doesn't know me, I'm Katie Green, Principal Advocate at Chameleon. I've been in experimentation for about 10 years, and my favorite one-liner is that I sell experimentation. So we're here to talk about the things that make experimenters excited, and one of those things is, Akash, your experience. So please introduce yourself to the group. I know you were formerly at Delta, welcome.
00:01:27
Speaker
Let us know what you're up to Tell everybody why they should listen to you, huh?
Data Fatigue and Metric Relevance
00:01:32
Speaker
Yeah. Appreciate the intro. So Akashidoshi, have spent the past 10 years working across ah commerce, retail media,
00:01:42
Speaker
And product management, ah as Katie mentioned, spent the past three and a half years ah leading digital strategy, ah ah part of that digital strategy for Delta Vacations, which is really the leisure subsidiary of Delta Airlines, working across AI adoption, product management, building out the consumer experience.
00:02:02
Speaker
And Katie, you and I met back in February ah at the Chameleon conference with with ah Gain, of course. And, you know, you and I just had a great chat. It was a great dialogue. And and we thought, hey, why not kind of share some of those ideas publicly?
00:02:20
Speaker
Yeah, we're a couple of nerds just getting our conversation recorded for the internet to have forever, which is classic. Yeah. No, it was so great to meet you in Vancouver. I know, right? Archive it forever. um I'm sure there's probably clips of us chatting in that room in Vancouver. A good shout out for anybody who wants to come to Unite Summit to DM me on LinkedIn. um No, but I think you have done a lot of these, right? You are expert expert at talking in front of a camera, in front of a mic. And I watched your episode with Quantum Metric.
00:02:53
Speaker
And you know you you were great at that one. You had some really good points. And so I was happy I had a chance to kind of take what you and I talked about as it relates to experimentation, but also as it relates to data. Because something you talk a lot about is data fatigue and how there's a battle between Because people are drowning in numbers. But as experimenters, this is our whole life. So can you tell us a little bit more about the the drowning in data argument you have? I think it's a really important one because how do we balance too much data versus the right data? Would love to hear more just because as experimenters, it's quite literally what we do is <unk> determine decisions based on data.
00:03:34
Speaker
Exactly, yeah. And I think that was the great sort of connection with the growing space, which is AI and experimentation, right? And and democratizing technical fluency. So normal folks can run A-B tests without needing this entire sort of ah technical stack or technical team ah leading that work. And so how that relates to data, I think, is really twofold.
00:03:56
Speaker
I think ultimately folks sitting in the business, right, whether you're working in product or marketing, customer experience, we're really just trying to have better answers to questions, things that we want to solve, or we want the right information to understand what our next
Balancing Data for Effective Experimentation
00:04:12
Speaker
steps are, right? What is our vision, our game plan? How do we go about solving a critical customer problem, building that next enhancement?
00:04:19
Speaker
And the role data, I think, you know, the role that data is playing right now is that we spend so much time on building the infrastructure, right? Think about the applications that you host. That's either for a B2B, you know, customer experience, a B2C customer experience.
00:04:34
Speaker
And we build out these really, really robust data layers. We have all these metrics coming in. And, you know, my personal experience is that it's led to a lot of data fatigue, right? Is that we have so many metrics. We often don't know which ones are the right ones to really do what I was just mentioning earlier, right?
00:04:52
Speaker
Have better answers to questions, make better decisions. And I think when you kind of use that framework, it helps you kind of get rid of some of the noise, focus on the signal, and and all of that comes down to like, what are the right proxies, right?
00:05:03
Speaker
And so I think a lot of times... People are kind of reversing the order of operations. They're like, okay, we have all this data. Let's try to make sense of it. I think maybe the better way to go about that is we have a key customer problem we're trying to solve.
00:05:17
Speaker
We have a key enhancement we we want to build out. Now, what are the right indicators to measure if that's a good idea or not? And then go back to the data and you have a bit more of ah ah a focus on what exactly you're looking for.
00:05:30
Speaker
So I think that's kind of the role data fatigue's played and and how we can maybe drill down our focus on what matters most. I want to zoom in on one piece you just said, which is asking the right question, because that's what experimenters are so focused on it. All alright our Our hypotheses, right? That is a very hard thing to say. Our hypotheses, our questions are what inspire really impactful, scalable programs. So tell us a little bit more about, you know, asking the right question, because i think you in particular have such a grasp of the right answer. But what is it to take a step back and ask the right question? And why is that so important when it comes to experimentation and data in general?
00:06:14
Speaker
Yeah, I think that's a great question because it starts to get us into how does A-B testing solve some of the issues we see that large teams are facing right now or even smaller teams. And when it comes to asking the right questions, I think you know we can just kind of build out a contrived example and you and I can can sort of bounce off that is like, let's think about like,
00:06:33
Speaker
funnel conversion, right? You have a look, like, for example, an e-commerce site might have an intended flow that they want their customers to go through. And so you know a data and analytics team might be looking at that step-by-step conversion, notice that there's a drop off on a particular page or a particular action.
00:06:51
Speaker
And so then they zoom in, right? And they try to understand, OK, what's occurring at this step in the funnel that's leading to customers dropping off, that's preventing conversion. And so that could be a variety of things, right? If it's payments page, maybe it's an API failing, maybe they're having trouble adding something to cart, ah maybe the experience is just confusing or the UI just doesn't make it clear how how how they can proceed.
00:07:13
Speaker
ah It could also just be a ah behavioral problem, right? The customer isn't just ready to convert. Maybe they're checking prices. So there's there's like all these things going on. And I think you know what what you know we spoke about in that quantum metric conversation was When we have voice of the customer, we're able to really zoom in on what the issue is, right? Rather than us doing a lot of guesswork and extrapolating and saying, hey, we think this might be the issue, we're not really sure, ah we get a very, very firm indicator on what exactly is going
Democratization of Technology in Workflows
00:07:43
Speaker
wrong.
00:07:43
Speaker
And then I think the opportunity for AI and experimentation is to really optimize or kind of use that as a launchpad and say, okay, we know now what the core friction is. We've identified the root cause. We're not you know kind of targeting a symptom.
00:07:56
Speaker
Now, how can we A, B test a solution to kind of confirm we're going in the right path, right? So like, I think that whole... ah First half is like voice of the customer identifying the correct problem.
00:08:07
Speaker
Now the second half becomes A-B b testing is the right path to making sure we have that right solution in place and and making sure it's fixing whatever we're trying to solve, right?
00:08:18
Speaker
No, that that's that's great. That's exactly the ah perspective that I think a lot of people want to hear because people are going to be tuning into your episode, this episode in particular, because of how much data representation you bring to the table when it comes to experimentation. I think a lot of us come from different angles. That's something really unique to experimentation is, you know, I come from more of the marketing and growth strategy side. I was agency side for many years, most of my career. And You know, I'm i'm coming from that. Some people are coming from the dev side, data, design, right? There's so many different levels to it. And I know that people are really looking for that that data intersection with you in particular and your experience. But something that we struggle with in the experimentation world, and I know you have seen this in other
00:09:06
Speaker
realms as well, just knowing your talking points and you and I have discussed for, gosh, now we've known each other for six months. um But I know it's wild. um But is that it's democratizing, right? When you have so many different levels of experience with technology, even with data, it's it's really hard to democratize a program and bring people in so that they can meaningfully make impact without running into roadblocks. So I know you have a really good take on democratizing technology, not just the data. I want to make sure that's like a huge focus in this episode because that's what I think what a lot of people come to Chameleon for. That's our market differentiator is, okay, with PBX, you don't have to be a dev to code this. with with dev If you're a dev with PBX, you can start part of this process and reduce a lot of the time you're spending on a certain test build. Tell us how that relates to data and what democratizing technology really means to you.
00:10:04
Speaker
and you're You're totally right. And so i think this kind of dovetails into the broader thread, the trend we've been seeing over the past two years with a lot of agentic and generative tools, which is vibe coding. And I'm going to put my preface out that I don't like the term vibe coding, but there's been some data and I've actually, I you know just recently made a LinkedIn post on this, which is that people are saying, hey, with the influx of vibe coding, right? The ability for a normal person who doesn't have a technical background, they're not a developer, They're not a software engineer. They're able to now build and push applications to production.
00:10:41
Speaker
And a lot of that data is pointing in the direction of like AI slop, right? The idea that we're building a lot more technology, the output has grown by an order of magnitude, but we're not necessarily building good technology. And my view here is a little bit contrarian, which is, I think you articulated it perfectly,
00:10:59
Speaker
It's not the fact that, hey, people sitting in the business can now build a full application, the back end, the front end, and push it to production. It's that they can own a greater share of that workflow.
00:11:10
Speaker
And now developers can retain focus on the things that matter to them. And we can be better stewards to our technical teams by having you know the power to now build a mock-up you know in in hours rather than days, or using a generative or agentic tool to build out the infrastructure, or even just a PRD, right like a product requirements document that gives developers a very ah granular focus on what exactly they're building.
00:11:38
Speaker
In my experience, one of the biggest pain points for engineers ah in working with not just product teams, but with the business generally is that there's not enough definition they need to go out and build the thing that the business wants them to build.
00:11:52
Speaker
And so I think part of AI and experimentation is that It's not necessarily just that like a tool like PBX can help you launch an A-B b test yourself within minutes. It's that if there is a more complicated A-B b test, something that is a little bit more technical, we can at least push out a V1, right?
00:12:10
Speaker
We can push out the source code. We can push out sort of this infrastructure where we can now go to developers and say, hey, this is our idea. This is what we've built so far. ah Can we now see it through to production? Can we build V2 of this?
AI's Impact on Workflow Efficiency
00:12:26
Speaker
And so I think that's become really powerful. So if if anything comes out of kind of that vomit of words I've just spat out, it's not just that, hey, you you build end to end. It's that we own more technical fluency.
00:12:41
Speaker
Now we pass off to developers and their jobs are made a whole lot easier. And I think that's that's generally a good thing. No, there's so many things I want to talk about there. But really quick, my dog is desperate to say hi.
00:12:59
Speaker
so Oh, a cameo. love it. He says, he says, I want to know why you don't like the term vibe coding.
00:13:11
Speaker
That's what, that's what Roscoe and I both want to know. I think it's probably a function of connotation, right? Like, I think vibe coding has ah developed this stigma is probably not the wrong word, probably a little too strong.
00:13:24
Speaker
But it's ah kind of like taken on this life of its own where it's like, oh, I'm just going to go into a tool like Cloud Code ah or Lovable or Base 44, any of the big tools that that exist right now, I'm just going to type in what I want and build out an application.
00:13:41
Speaker
and and I think that's maybe the wrong way to look at it. It's more that people who never you know have written a line of code in their life are now being exposed to a functionality that's net new to them. It's making them a little bit more technical in the process.
00:13:57
Speaker
I think generally it's a good thing. I just think vibe coding has maybe... garnered maybe a bad reputation for for better or for worse. Yeah, I feel like when you have a single term for things. i I do also want to underline something else you said because I think it's really important. There's kind of two pieces to it. The first where at the end you said, okay, well, when people have more access to the workflow, they're able to present things a much more well thought out lower barrier opportunity for engineering, right? When it comes to testing. I love, I love that point because it does directly call back to an episode I did with Marcella. Were you two able to meet at unit Unite Summit, Marcella?
00:14:39
Speaker
From Fossil. Okay. We'll make it happen next time, whether it's at Unite Paris or Unite Vancouver, whatever it is. But she, in her episode explained, she kind of uses it as a tool.
00:14:54
Speaker
it's her Her exact quote is, it's harder to stop a moving train, right? You have that momentum. You say, hey, here's like a first version of that. But what i want to build on that is... that it also, ah it frees up time to focus on what matters. So when you're not worried about coding, you're not worried about clogging up the sprints by coding a button color test, right?
00:15:17
Speaker
You can focus on things like feature flags that are really impactful to your bottom line. So I think what this all comes down to is AI is obviously changing the workflow and changing how we're working together. So I'm i'm curious, and I realize you know you and I haven't ever talked about this before, but I just wanted to get your opinion.
00:15:42
Speaker
Do you feel that ai and experimentation is allowing people to ask better questions, or are they just moving faster? i i think both things can be true at once, which is that I think it is allowing people to ask better questions, right? Which is...
00:15:59
Speaker
you have a guided thesis on the way your business should move forward or the way you want to develop a product. And I think AI experimentation is a great great way of gut checking that a whole lot quicker, right? Instead of...
00:16:10
Speaker
taking up two to three s spints sprints, to your point, ah building out a test, building out a hypothesis, launching that, and then you know then realizing, hey, we actually targeted the wrong part of the funnel. We built the wrong feature.
00:16:24
Speaker
This actually wasn't the right test to validate whatever our hypothesis was. I think now AI experimentation de-risks you a little bit, right? You're able to build out a test plan, get it into production. If it wasn't right, you've now narrowed down what your North Star should be.
00:16:42
Speaker
and and then I think the same so like the different side of that same coin is that the velocity increases, right? You're able to get to market quicker. And that's like probably the thing that I talk about too much, which is that I think in 98% of scenarios, building fast is better than building perfect. ah The reason that's true is that when you're able to get to market quicker, those feedback loops activate, you're able to hear from the customer, and that really allows you to validate your guesswork, right? Instead of saying, hey, we think this is the right path forward. When you get something into market, those customer feedback loops just tell you like, hey, this is what they wanted or this isn't what they wanted. And and so I think both things are perfect.
00:17:24
Speaker
like exist in parallel, right? Teams build faster, and they're asking better questions as a result of building faster because they know what their customers are telling them. All right. Perfect. I mean, like I said, i truly could not have said that better myself. It is two sides of the same coin, the velocity and getting those market indicators that really move the business forward. So this is why, again, I ask you these things. I don't say them myself. So great job. I love to hear it.
00:17:50
Speaker
Something else about it is what we call, you know, you and I have talked about this. the tax, the administrative overhead when it comes to testing and how much time you work. You kind of touched on it when you're talking about de-risking, right? It's kind of like an external de-risk of, okay, yeah, you didn't go with this future, but it's also AI and experimentation is drastically reducing the cost of building a test, launching a test, whatever it is. So can you tell us a little bit more about the administrative overhead and how ai is really playing into the overall experimentation workflow?
00:18:23
Speaker
Yes. and And I think the best way to answer this is by way of example. So if we take a sample digital team today, the way the process might go is something like this. You're in a monthly performance review or a quarterly business review, and you see a particular metric or KPI that looks a little off.
00:18:44
Speaker
The teams dig into it. You start to triage, hey, what's a root cause? What's driving this? Okay, now how can we fix this? At that point, the experimentation team might come in and say, okay, we think this is the problem.
00:18:56
Speaker
Let's build out an A-B test or an A-B-C test or however you want to split out that audience to see how they interact with two different experiences and see what the winning one is.
00:19:07
Speaker
And I think all of those steps take a significant amount of time. ah For a normal team that's balancing multiple work streams, it could be two to three months between identifying that pain point and actually getting something out into production.
00:19:21
Speaker
And I think AI and experimentation helps speed up that pipeline considerably. so a lot of that administrative overhead around identifying the problem figuring out the solution, and then actually developing and launching that solution now gets condensed into this one to two week period where if your team has the autonomy to say, okay, data and analytics have identified a key customer pain point, we think we can test something out.
00:19:47
Speaker
Now, before you even go to product teams, before you go to engineering, before you have them build and code out a solution, you can actually launch something that helps you validate that much quicker. And the ability to get to market to our earlier point speeds up.
00:20:00
Speaker
And I think that just saves people a whole lot of time. I think anyone who leads experimentation or A-B testing within a digital org today knows how long that backlog is. And as a result, when you compound that five or six times, what ends up happening is you have a giant backlog of great hypotheses that you want to test that never make it to market, right? They just kind of sit in purgatory.
00:20:22
Speaker
So I think the ability for AI to now insert itself in experimentation allows you to clear that backlog, validate you know a gut feeling you have about the way you should build a product, the way your customers behave with your product. And and so I think it helps reduce a lot of that administrative overhead, that admin tax.
Integrating AI Meaningfully in Workflows
00:20:41
Speaker
I think that's what I tell a lot of people, right? that that's it's a very It's a different way, but I am going to start to pepper in some of your talking points is that overhead is what slows you down. And then you're missing really critical impact. So 100%. I think we could keep this episode 50 minutes, but to spare everybody, us included.
00:21:04
Speaker
Let's get to my last question. I asked the same question of everybody, just kind of based on what we've talked about during the episode. But it's all about what I call the Monday morning advice. I realize it might not be Monday morning when people are listening to this. It's Monday morning for us right now as we record this. But what I mean is what's the tangible takeaway? And in particular, you do such a good job of operationalizing your AI usage.
00:21:27
Speaker
I think a lot of teams are struggling with you know whoever the higher power may be saying, use AI, use AI. and they're like, okay, how? Or okay, I'm vibe coding these things. They're not actually that powerful. Like, what what do I do here? So like, tell us a little bit about when it comes to AI, how do we meaningfully implement it into our workflows and not just let it be another complicator in a chaos factory.
00:21:55
Speaker
ah You're probably the fourth or fifth person just in the last couple months who I've spoken to who echoes some version of that sentiment, right? Which is that enterprise is telling teams, hey, adopt ai in your workflows, but there isn't a meaningful blueprint on how to do so.
00:22:12
Speaker
And so then getting to the root of your problem is like, let's use Let's have a guided POV on the place where using AI meaningfully helps the business. right So like things that come to the top of my mind are the content supply chain. right Smaller marketing teams have always had an issue with producing creative at scale.
00:22:32
Speaker
I think AI has an obvious opportunity there to create versioning, to create new assets, and help you, again, deliver at a quicker sort of clip ah than what exists traditionally.
00:22:42
Speaker
With product managers and engineers, AI can now review your source code, right? Or if you notice a defect, logging a defect is no longer like, hey, this button's breaking, go fix it.
00:22:53
Speaker
You can actually dig into it yourself. have AI use computer use, go through the dev tools, go through your logs, figure out the issue, we can write out the full defect for you. Now engineers are actually surprised when you hand them the defect written up because it has so much context and granularity, they're just able to action and solution on it so much more quickly. And so I think there's opportunities like this where it helps you narrow your focus. So when digital teams are told, hey, go use AI, like, you know, use it to make yourself better, they now can actually understand, okay, these are like a set of different problems that when I encounter them in my day to day, there's a guided POV on how to use how to use AI to actually help me and not to just like kind of spin my wheels.
00:23:36
Speaker
Yeah, I realized that that wasn't the original question I planned on asking you on this. you know Obviously, to everyone listening, we have a conversation before we record this about what we're going to talk about. But in that last one, you know we're talking so much about AI, and I know you have done an excellent job of leveraging it. It's just like you said, so many people lately are coming to me and they're like, my boss told me to use AI, and I'm like using ChatGPT, I guess, and Gemini and Claude. I'm trying to code variations, but...
00:24:03
Speaker
it's not meaningful. What do I do? And I'm like, oh gosh, like i I really am not the expert in this, but I try to elevate the people who are. So thank you so much for sharing that perspective because I know you have a lot of impact and experience in this field. um With that. Yeah. We could probably have a whole separate episode on like, I think the exact pain point you've mentioned, which is that how do we take work streams that people actually, you know, engage in on their day to day and then like insert AI in a way that's meaningful and not just, this is actually creating more work for me, not less.
Episode Wrap-Up and Future Discussions
00:24:34
Speaker
ah But another topic for another time. I know. Sorry, you'll have to tune into part two that we record in another three months. um Okay, great. ah Thank you so much for being on Unite Voices. I appreciate you. And we'll talk soon.
00:24:47
Speaker
Thanks for having me, Katie.