What do you do when the data says one thing and a stakeholder refuses to believe it? Most experimenters never get taught how to handle that moment.

This episode is for anyone who has watched a good hypothesis get lost in politics, pressure, or too many metrics. Cristina walks through how she raises the bar on rigor without becoming the office math teacher, and how she gets stakeholders to accept results they don't want to hear.

About our guest

Cristina McGuire has spent more than five years building experimentation programs from the ground up, first at Expedia Group, then Chewy, and now on the Windows experimentation team at Microsoft. She came into the field as a data scientist with a statistics background and learned along the way that the math is only half the job.

Key takeaways

 1. Anchor every test to one or two primary metrics before it launches, so you are not tempted to swap in a flattering metric halfway through.
 2. When a stakeholder pushes back on a result, quantify the trade-off in dollar terms instead of arguing the statistics.
 3. Revisit your experimentation strategy roughly once a month rather than per test. A good plan is replicable across many experiments.

Why do people ignore good A/B test results? A statistician's perspective | Cristina McGuire, Experimentation Data Scientist at Microsoft

What if optimizing to optimize is actually holding your testing program back? Craig Kistler thinks most experimentation teams stop one step too early.

Craig joins Katie Green to explain why a UX lens breaks testing programs out of their button color plateau, and how intent, not one size fits all rules, should drive every personalization decision. He also shares how his team scores hundreds of competing ideas and brings AI in as a prototyping teammate before a single line of code gets written.

About our guest 

Craig is Senior Director of user experience, personalization, and experimentation at Signet Jewelers. He has spent nearly 14 years building the retailer's UX and testing practice from the ground up, and he was recently nominated for a Kameleoon Experimentation Thought Leadership Award.

Key takeaways

 1. A UX lens turns experimentation into a flywheel: instead of chasing button color wins, Craig's team hunts for usability problems and tests them against millions of visitors.
 2. Best practices like urgency and social proof only work for some shoppers, so Craig segments experiences by intent instead of applying one rule to everyone.
 3. Craig's team scores every experiment idea, from the CEO's to the intern's, on impact and insight before it earns a spot on the roadmap.

Stop optimizing to optimize: How UX drives experimentation velocity | Craig Kistler, Senior Director XD, Personalization & Experimentation at Signet Jewelers

What if the metric you're tracking is guaranteed to go up, no matter what you build? That's not a win. It's a warning sign that you picked the wrong thing to measure.

This episode breaks down why the metric you choose matters more than the test itself, and how an economist's instinct for trade-offs changes what "success" looks like in an experiment. Zach shares the three pillars he used to scale experimentation at Udemy, and how to keep marketing, engineering, and leadership aligned around the same evidence.

About our guest

Zach Flynn is an economist who has spent his career finding the numbers hiding inside a business, from pricing work at Amazon and Udemy to his current role as a pricing and monetization scientist. He walked into experimentation with zero prior experience and ended up rebuilding an entire program from the ground up.

Key takeaways

 1. Choose metrics you genuinely don't know the outcome of in advance, since a metric that always moves in the expected direction teaches you nothing.
 2. Decide your significance level and your decision criteria before the experiment starts, so no one reads the tea leaves after the fact.
 3. Remove process and guardrails first when a program has stalled, since teaching best practices only works once people are actually running experiments.

Pick metrics that surprise you: the economist's rule for experimentation | Zach Flynn, Pricing and Monetization Scientist, Fivetran

Sometimes the ugliest idea on the roadmap is the one that wins the test.

Summary

Katie and Linda talk through why so many e-commerce sites end up looking the same, and how easily ego can crowd out curiosity once a testing conversation starts. Linda shares how she vets ideas for Ecom Ideas and what building her latest project with AI taught her about design systems.

About our guest

Linda Bustos founded Ecom Ideas, a free library of e-commerce UX patterns she catalogs from sites across the web. Before that, she spent years in digital strategy roles, including at Elastic Path and CorporateGift.com.

Key takeaways

 1. Mobile navigation stays plain because most brands skip visual hierarchy, and a small push toward featured categories or product recommendations opens real testing opportunities.
 2. AI speeds up single-component experiments, but it does not respect a design system on its own, so a person still has to hold that line.
 3. Bad-looking ideas sometimes win, so testers protect the roadmap by running risky ideas on medium-traffic pages or throttling exposure instead of rejecting them outright.

Bad ideas can win: what that means for ego in CRO | Linda Bustos, Founder, Ecom Ideas

Most enterprise leaders are telling teams to just use AI. Akash Doshi thinks that mandate is missing the point entirely.

Akash and Katie name the real bottleneck in experimentation: an administrative tax that buries good hypotheses in the backlog. They show how AI shrinks a two to three month test cycle to one to two weeks, so backlogged hypotheses actually ship.

Akash spent three and a half years leading digital strategy at Delta Vacations, Delta Airlines' leisure arm. He is now the founder of Sequence, AI middleware built to turn scattered AI experiments into repeatable workflows.

Key takeaways

 * Identify the customer problem first, then work backward to find the metrics that show whether your solution actually works.
 * Hand developers a working mockup, a PRD, or a V1 build before looping them in, so they can focus on harder problems.
 * Ship a fast, imperfect version of your test now, since real customer feedback corrects your hypothesis faster than another sprint of planning.

Building fast beats building perfect: How AI accelerates experimentation velocity | Akash Doshi

You can run 100,000 experiments. But are you actually making better decisions? Two seasoned platform builders show you where most experimentation programs go wrong, and how to fix them.

Building an experimentation program is hard enough. Building one across three different platforms, four countries, and teams that all believe their own system works better is nearly impossible. In this episode, they walk you through the real problems they've solved: winning trust without ripping out legacy systems, choosing which methodology to standardize on, deciding what to test when you have a tiny window of traffic, and what AI actually changes about how we experiment.

About our guests

Ilya leads experimentation at DoorDash and has built testing systems at Intuit, Robinhood, Uber, Amazon, and PayPal. Makram heads marketplace at ID.me and previously managed experimentation platforms at Intuit and LinkedIn. They've never worked together directly, but they've spent their entire careers in the same tight community.



Key takeaways

 1. Focus on decision quality, not test volume: a company running fewer experiments that actually drive product changes outperforms one running thousands that no one acts on.
 2. When consolidating experimentation platforms after an acquisition, start with shared methodology and metrics, not by replacing the inherited system. Local teams have earned trust with their own platform, and destroying that erodes adoption.
 3. Democratize experimentation through AI: the old silos where only PMs write PRDs and only engineers ship code are breaking down. Give teams the right guardrails, and everyone can contribute.

Decision quality over quantity: why your experimentation program needs fewer, not more tests | Ilya and Makram

AI is like a teenager: 100% confident and 70% right. So who fills the other 30%?

Katie Kelly and host Katie Green (double Katies) dig into what AI actually changes for CRO teams, and what it can't touch. You walk away knowing where to automate safely, how to protect the creative and research work that still needs a human, and what to do tomorrow to grow your own profile in the field.

Katie Kelly has worked in conversion rate optimization since 2008, across agency marketing, operations, and growth. She also built Kameleoon's Who's Who, having reviewed thousands of experimentation professionals to spotlight voices the industry tends to overlook.

Key takeaways

 1. Simplify a process before you automate it, because automating a messy workflow just scales the mess.
 2. Protect entry-level talent and diverse voices, since fresh perspectives and creativity are exactly what AI cannot replace.
 3. Keep qualitative user research human; a 10-minute conversation with a real customer beats a synthetic audience for original ideas.

AI does the boring stuff: Why creativity still wins in CRO | Katie Kelly, Fractional Business Partner, Subject Consulting

Most teams blame the platform when experimentation stalls. The uncomfortable truth is that it's usually people and process.

Paul Davidson was a Senior Product Manager at Expedia Group and Senior Product Owner at NI (National Instruments). He spent years as a product manager on an internal experimentation platform, working on the roadmap and alongside the analysts and engineers using it every day. He has watched thousands of tests, and he knows the patterns behind the ones that go nowhere.

This conversation breaks down how a program drifts into an anchor, and the specific moves that cut one team's test cycles from six weeks to two. You walk away knowing how to simplify your metrics, validate earlier, and hold the trust of stakeholders when real money is on the line.

 1. Trim your secondary metrics and move your primary metric closer to the change you're testing, so readings come back in days instead of weeks.
 2. Validate early and accept imperfect feature work, because a test that fails fast on instrumentation saves the weeks you would otherwise lose validating at the end.
 3. Protect financial goals with circuit breakers and guardrail metrics, then lean on proven proxy metrics so not every test has to move revenue on its own.

Has experimentation become an anchor? Testing for learning, not validation | Paul Davidson, Product Manager

What if your users already know exactly what they want, but your product is still too hard to use? Behavioral science has a word for that problem, and it's not "friction."

Kristen Berman is a behavioral scientist and CEO of Irrational Labs, the firm she built after running Google's behavioral economics group. She studies how people actually make decisions, and she spends a lot of time showing product teams why their assumptions about users are off.

 In this episode, Kristen walks through the gap between what users tell you in research and what actually drives their behavior. She and host Katie Green dig into how behavioral psychology can sharpen experimentation hypotheses, de-risk big product bets, and help teams build a genuine culture of learning.

Key takeaways

 1. When building an experiment hypothesis, start from a psychological theory about why behavior isn't happening, not just from what users say in qualitative interviews.
 2. Out-of-product A/B testing, such as running four versions of a pricing page with recruited participants before touching the live product, lets teams take bigger swings with far less business risk.
 3. The teams with the strongest experimentation cultures tend to share one trait: humility. They assume the first version of anything is probably not the best version.

Why past behavior is your best A/B test signal: Lessons from behavioral economics | Kristen Berman, Behavioral Scientist, Irrational Labs

Stopping someone who already has a finished experiment is a lot harder than saying no to an idea on a whiteboard. Marcela Gutierrez figured that out, and it changed everything about how Fossil Group runs its experimentation program.

Marcela Gutierrez leads digital analytics, experimentation, and consumer behavior at FOSSIL Group, where she has spent 13 years across the US and Switzerland. She is the kind of analyst who thinks just as hard about how data gets communicated as she does about what the data says.

In this episode, Marcela shares how FOSSIL shifted who builds experiments and why that single change unlocked speed across the entire team. She also breaks down how her analysts think about metrics, storytelling, and why human judgment matters more in experimentation programs as AI takes on more of the build.

Key takeaways

 1. Building a test before asking for approval changes the conversation. Coming to a sprint review with something nearly finished is harder to stop than rejecting a rough idea, and FOSSIL uses this dynamic to run more experiments with less friction.
 2. A primary KPI needs a secondary one. Marcela's team pairs each main metric with what she calls its "BFF" metric, so results tell a coherent story rather than just a flattering one.
 3. Analysts are not data processors. The most valuable work they do is understanding organizational context, cleaning data before it goes in, and translating insights for the right audience at the right level of detail.

Harder to stop a moving train: Inside FOSSIL's high velocity testing program | Marcela Gutierrez, Director of Digital Analytics and Experimentation, FOSSIL

Unite Voices is a show where marketing and product leaders share the moments that changed testing at their company, and how they use AI (or not) in their experimentation work.