Transcript
Speaker: We're definitely using AI, but I don't know that we're using AI the way maybe some companies would want us to use AI. We can automate and do all this stuff and you just push a button and go. Okay, maybe.
Speaker: Then the challenge I have is now you're just optimizing to optimize. Yes, you could be doing something for the experience, but do you truly understand what the problem is? do you truly understand what you're trying to solve?
Speaker: I don't want to poo-poo all over it, but it's not where I need it to be right now. Where I would rather use AI is identifying patterns and problems that may not stand out to your typical eyeball, your typical group of people looking at data. So using it from a standpoint to analyze a whole bunch of data data and crunch a bunch of numbers all at the same time to find patterns that you may or may not have seen before.
Speaker: Welcome to Unite Voices, hosted by Katie Green. Real stories from the people behind today's most innovative experimentation programs. No fluff, just wins, failures, and the lessons in between.
Speaker: Hi, everyone. Welcome to another episode of Unite Voices hosted by me, Katie Green. I'm the principal advocate at Chameleon, and I'm so excited to be joined by one of our experimentation thought leadership awards nominees for the UXpert category, Craig.
Speaker: Craig, don't you introduce yourself? Thanks. Yeah, my name is Craig Kistler. I am vice president of user experience, personalization, and online experimentation at Signet Jewelers.
Speaker: So, Cigna Jewelers is the largest jewelry retailer in the United States, parent company of Kay, Jared, Zales, you know, so those commercials that you see during the holidays. I was going use a word, but I'll be nice. No, I'm like, oh, so it's like, it's like kind of, it's a small company, right? Like, oh my God. No, I had ah one or two little brands. So yeah, so I've been doing that. I've been there, it'll be 14 years this fall. 14 years is so impressive. A long time, which means I'm also old. So there's that.
Speaker: um But yeah, when I got, when I was hired there, I was the first and only UX designer that they had. Now, 14 years ago, it still should have, they should have had that before, but they didn't. So I came in and had a small UX team and then with big businesses or exchange things shift. So I kind of had a team and then i started looking at this AV testing thing and experimentation and they brought that in and I'm like, this is kind of cool.
Speaker: Not sure what would do with it. And then the team that was using it was doing stuff. I'm like, this doesn't make any sense. Like this isn't helping it. Like we're just doing stuff to do stuff. And that's when it clicked for me. I'm like, oh, wow, I can actually take user research, usability testing and run A-B tests against those things and create a better experience.
Speaker: And then it was off to the races. I love that. And I think a lot of people can relate to that experience, myself included. i was I remember my first test, I was at an agency and we, for the first time, were like, oh, why haven't we actually tested that this could work? Like we've always just operated on best practices, which are like, I realized triggering words for crow people.
Speaker: But so the people listening to the show are largely experimentation practitioners. They're people interested in what you do. Obviously you have a ton of UX professionals, experience. That's why you were nominated for ETLA. i think that intersection between experimentation is is really critical. And you and I have talked about, slash I've seen this, you talk about this online, which is partly why you were nominated.
Speaker: ah You are very a very good person to follow. So shout out to anybody who wants to follow you on LinkedIn. I think it's a good place to start here. But i think that testing programs you've mentioned, they can plateau. And when you get stuck into testing tiny little features, you know what what does that look like from a UX perspective and how do you recommend people get unblocked? Because I think a lot of people will be listening and saying, oh yeah, like I need more UX integrated into my roadmap or I'm feeling really stuck with where we are. How do you get over that kind of plateau? So it's ah it's ah it's a great question because I think a lot of times
Speaker: It almost feels like there's there's two schools of thought. There's like the original CRO of like, let's test this thing to improve whatever KPI you're trying to do. You know, the the inus infamous button color tests, headline tests, these things that may or may not move the needle, but they get a lot hype.
Speaker: And I think everybody starts there, which is fine. It's cool. Like, hey, if you're trying to improve, why not start? It's low hanging fruit. Try it. You got to start somewhere. Yeah, totally. You got to start somewhere.
Speaker: And- You may get a win or two, you may not, but the challenge, if that's all you're thinking about to your point, you're gonna get very frustrated because you're like, this isn't doing anything. my my My team doesn't care, my execs don't care. Like, why is this happening?
Speaker: We're not getting momentum. For me, when I started looking at it from a UX angle, right The job of any UXer is how do I make this experience better? How do I make it more relevant? How do I provide clarity?
Speaker: And when you start thinking about those things and running experimentations against those ideas, that kind of became more of a... like a flywheel, because then you can start moving something relatively small. So maybe it's not button color, but maybe it's button placement.
Speaker: Maybe it's how visible is this button? Is it smashed a bunch of bunch ah bunch of different things, a lot of clutter? I don't know, maybe. And you can start seeing these patterns show up, like you're finding usability problems, and then you're testing those with hundreds of thousands, millions of visitors, as opposed to what you get in a usability lab.
Speaker: Still valid, but we've also seen things that test well or not well in a usability lab will have alternate results online because you're in the real world and you've got millions of eyeballs coming at it.
Speaker: But it just becomes a really good ah bed of ideas where you can like identify problems and then you hypothesize around those problems. There becomes your your your roadmap from testing standpoint. And I think let's let's talk about roadmap for a second.
Speaker: Something that I don't fully understand myself as I don't have a UX background, right? I i really do firmly come from the growth marketing side of things. Can you tell me a little bit about how AI has influenced your industry and as it relates to experimentation? I know AI is obviously affecting design in a ton of ways, but... you know, UX research is in and of itself so important. And as people are talking about synthetic user testing more and more, like, I'm just curious how AI has really infiltrated UX and the intersection of UX and experimentation. Yeah, I think it's really new.
Speaker: So I think it's going to be ever-changing for me, for my team, for what we're looking at. We're definitely using AI, but I don't know that we're using AI the way maybe some companies would want us to use AI.
Speaker: You know, you hear a lot of, oh, well, we can automate these tests. We can find the problems. We can just, we can automate and do all this stuff and just push a button and go. Okay, maybe.
Speaker: But then the challenge I have is now you're just optimizing to optimize. Yes, you could be doing something for the experience, but do you truly understand what the problem is? Do you truly understand what you're trying to solve?
Speaker: I don't want to poo-poo all over it, but it it's not where I need it to be right now. Where we're where i would rather use AI is identifying patterns and problems that may not stand out to your typical eyeball, your typical group of people looking at data.
Speaker: So using it from a standpoint to um analyze different user flows, analyze a whole bunch of data data and crunch a bunch of numbers all at the same time to find patterns that you may or may not have seen before. And I was just taking note of what you just said, because ah Shirley, who's editing this in the future, um i was like, this is, ah that's a really good clip. I'm like, that's the title of the episode, right? Is, are you optimizing just to optimize?
Speaker: Or are you being meaningful and understanding your be it your behavior problems with your users, right? and i would And I say behavior problems because it's like the infrastructure we're giving them, you know?
Speaker: Right. and And everything will have a different kind of like end goal, you know? often you mentioned but best practices earlier and i kind of cringed because hate them, um personal opinion.
Speaker: But it's like, well, they're just, it's a broad brushstroke. And if that if you if you're starting somewhere, fine. But you know when you when you when you think about this, like you could be optimizing for different things based off of what that person's trying to do. you know the The PDP is typically, your product detail page is typically an entry point. So now it has to function as a homepage. It has to focus and tell the story of why you should care about this product as well as this brand.
Speaker: But for other visitors, it's not an entry point. They've been on your site for 20 minutes. They've narrowed it down between two or three items. Now it's a comparative tool. you know So this this one page has all these different intents and these different types of visitor motivations to solve.
Speaker: And if you just go with a best practice or one metric, it kind of becomes the average of averages. It it doesn't necessarily mean anything. And nor does it differentiate you in the market.
Speaker: So it's it's it's very, I feel like we're getting to what is your main point that you do such a good job of talking about. So I'm not even gonna attempt to do that myself, but you are a particular powerhouse in UX because of your interest and passion and data. And you and I have talked about how that we have more data than ever to create personalized experiences, like you're saying from these entry points. So can you just share with our audience here in the future ah a little bit more from the UX perspective, from your background, what does it look like to build better experiences having more of that data before you go into an actual test?
Speaker: What does that look like in terms of shaping hypotheses and creating roadmaps that are actually impactful? Yeah, so for us, it it becomes less of one-size-fits-all. It really then tips into experimentation as part of kind of a personalization roadmap.
Speaker: So it's not just a versus b it is understanding your groups of users that are coming to the the site or the page and what their needs are in the moment.
Speaker: So instead of saying, okay, I want to make this change for everybody, which to be clear, we've done that. We'll make a change for everybody. We don't just immediately jump into personalization. But we're like, okay, if this can this work better for a specific group of people?
Speaker: I don't mean to keep beating the the best practice horse, but if we think about social proof, the chatter on social proof is, oh it's gonna increase your conversion. Everybody needs to see it You should put it on here. you know this it's Social proof, it's urgency. It's all those psychological triggers that are supposed to move people forward.
Speaker: True, they do move some people forward, but it's not everyone. So it's looking at your data and trying to understand who's gonna respond best to that experience. Who's gonna respond best to the urgency message of, at Quick is gonna sell out versus somebody who just hit your site.
Speaker: They don't even know if they wanna buy what you're selling. Maybe they're looking for a gift. And they're like, well, it's between, we sell jewelry. It's between jewelry, a nice dinner out, a vacation, whatever that is, like, how do I make that decision? Urgency is not helping with that.
Speaker: So if you can work to identify what your intent, what the intent of your shoppers are, then you can start crafting an experience for that intent, which makes that experience for that group more usable, more clear, and they can move forward faster.
Speaker: On the flip side of that, you also have to think about doing this in a way where people's intent moves. So you don't want to create a disruptive experience where what works for, say, somebody who's just starting out needs to also work for somebody who is ready to purchase. And that experience shift can't be jarring because they could go from one intent to another within a page or two.
Speaker: So it's like... It gives you more power, but it also gives you more responsibility to not start breaking things as somebody's going through their experience. I think intent is so important and often overlooked.
Speaker: As experimenters, obviously, the more mature among this group is going to say, it's so important to understand your user needs, but how often are like some of the biggest programs I've ever seen, you know, how often are they actually going in and checking that intent and being honest with themselves about what they're moving towards? So I love this. I'm like taking notes. This is such a good little clip to include about this piece. So that's, that's perfect. I think I, now, now that we've talked about the theoretical and we really understand what your perspective is in terms of personalization data and you know, all of these things, idea generation.
Speaker: Can we talk a little bit about how you do it day to day? Because I know you and I have talked about, I have to include the buzzword here of vibe coding. Okay. You and I have talked about vibe coding and, you know,
Speaker: how how are you using that internally? Are you creating, are there any systems that help you do this in a more regular basis? Because I can imagine somebody listening is saying, oh God, I really do want to focus intent on intent. I think I am, but i I personally, myself, have dealt with so many people have ideas and they're flooding in and I'm like trying to balance my team's ideas with what the user needs and creating, that that that gets down to like politics and diplomacy right at the end of the day. But what What internal tools are you creating to help you tangibly do the things that we're talking about today?
Speaker: Yeah. Well, there's a lot to unpack there because we go a hundred different directions. i know. Maybe you could pick one, but like, you know, I'm just he' so curious to know. Totally fair. I think we'll be able to touch on on most of them.
Speaker: So i want to start off. I'm very lucky. I have an amazing team. like I'm not a team of one. I've got front-end developers. I've got analysts. I've got product people. I've got a whole team of experienced designers.
Speaker: I'm very fortunate in that regard. My crazy ideas and hypotheses and stuff, I can work with them, flip it over the fence, and in days to weeks, I've got an experiment running. So they're amazing. So I realized that not everybody has the team that I have.
Speaker: So then it comes down to, I've got a whole bunch of ideas. Well, we've got to prioritize these. And we do the same thing. Like you can imagine we've got ideas, inputs coming in from our own team, other teams, the brand teams, the leadership team, like everybody and their brother wants to run an experiment.
Speaker: And you're right, it's it's a game of politics. But if we try to be incredibly fair, no matter if it's the CEO or the intern coming in with an idea, they come in we look at them, we score it.
Speaker: Very, very common. We will look at it, we'll score it. We're looking at the level of impact and the level of insight we're gonna get. Like obviously revenue is gonna bubble to the top from ah from an impact perspective.
Speaker: But we're also going look at will this give us additional insight for future tests. It's not going to trump revenue, but it comes pretty close. Because if we can learn something about a segment or a page or something that feeds into something else, well, like that becomes valuable.
Speaker: Long story short, we score the ideas and then the ideas that are scored highest come to the top. um from a vibe coding standpoint, I'm trying to remember all a little touch points you had.
Speaker: We like, we'll use vibe coding to almost i flesh out an idea. Like, oh, we what about this thing? can How does that work? So we'll sketch up maybe a design, um maybe vibe code it, see if it makes sense.
Speaker: I was playing around with something like with the idea of a natural diamonds versus lab grown diamonds, right? They're they're both diamonds. they're exactly the same. It's just one is, takes thousands of years. One is, doesn't. it But chemically, they're exactly the same.
Speaker: But there's a lot of people who think cubic zirconian, fake diamonds and all this. So how do we explain that in a way that makes sense and doesn't overwhelm somebody while they're trying to find an engagement ring or a necklace or set up earrings?
Speaker: So I vibe coded that just to see if this would work.
Speaker: It did. So then it then it goes into our backlog and and starts developing or working through the the development cycle. I feel like i'm missing something things you brought up.
Speaker: Okay. And I was, no, that's perfect. Cause that's exactly, that was my question is like, So many people are using vibe coding in all kinds of ways. Like even at Chameleon, we're creating these internal tools to try and make our day-to-day jobs easier. And it's so funny.
Speaker: um You know, have you seen the website Death by Claude? Okay. But Death by Claude is um this website where you can put in a product or tool. Like you can put in Google Analytics if you wanted to. And it basically tells you if you could rebuild it in Claude.
Speaker: or not. So it's like, I'll just you totally answered it. That's exactly what I was looking for. Vibe coding has just become such a huge part of day-to-day life. And yeah, Chameleon uses it internally for stuff. I know people are using it for prototyping and that's what PBX does, right? That's what Chameleon's PBX is, is a vibe coding tool for ah variations. But it also, you know, now PBX 2.0 has a lot more than just that. But I know a lot of people, I like that you're prototyping. You're kind of saying, okay, this is a really complex test idea. We're trying to communicate a lot of different things, trying to balance all of the different customer needs. Like, can I create something? Because i have personally found I could live my entire life never making another requirement sheet for ah developer. Yeah.
Speaker: like so difficult, especially when you don't fully know what you want, but you're like, I know I just want something like this and I need somebody to bounce it off of. So I love that you're using it for that. I think people listening are going to probably take that with them and say, oh yeah, maybe I'm going to try and vibe code like a complex test or something to see what I can do. It just helps you unlock those opportunities. So I think that's exactly what we're looking for. Yeah. And I mean, like to that point, so much,
Speaker: around the experience comes down to like the interactions or the transitions from point A to point B. In a design, a wireframe, whatever it is does not explain that well to any developer.
Speaker: So using Vibe Coding to prototype it out so it's like, oh, well, if i click on thing A, it goes to thing B, and this is the transition effect. This is how I want to like walk somebody through that pathway.
Speaker: it's It's just Vibe Coding makes it so much easier because you can visually see it as opposed to trying to talk about it and pull up another site and say, well, sort of like this, but then then mix in with this over here and then they're looking at you like if have three heads. You're like, just dude, just tell me what you need. just just Just tell me what I need to do for you.
Speaker: And you're like, God, I wish I knew. you yeah it's like, I don't know. I want something like this and I want you to mash it up with something like this and then magic will happen. Like, I know it when I see it.
Speaker: And I love that. And, um you know, just a side note for if you see me typing away, because I'm taking notes of your clips. And like, usually I get like two or three comments. I'm up to six on yours. So really good clips here. Just, you know, I'm like typing, furiously typing while you're talking. because I'm like, oh, 19 minutes. Take this check. You know, it's perfect.
Speaker: No, but that I mean, really, it is. Yeah. and 100%. I like the way you put that. you know It's all getting to point A to point B. I think you are very, very good at communicating exactly what you're meaning here. So thank you so much for being on this show, first and foremost. We have time for one more question, and it's the question I kind of try to ask everybody at the end. I call it the Monday morning advice.
Speaker: I know people aren't going to be listening to this on a Friday. Typically, they are because we put out episodes on Thursdays. But um I like to just ask, what is your recommendation for the team lead who is really trying to get out of the one size fits all AB b tests. Can, what is the one thing you think they should do tomorrow? Is it start prototyping in Claude? Is it taking time to figure out what their user intent is using data?
Speaker: What is something that you would recommend as the Monday morning advice that somebody takes away from this episode? So i think it I truly believe that creating personalized experiences is is gonna produce more revenue. it's go Let's say it's gonna have a greater impact than any single A-B test or series of A-B tests.
Speaker: And then the question becomes, Okay, so how do i do that? How do I figure out what is personalization? Because I don't want, I don't consider product recommendations personalization.
Speaker: It's just, that's not it. It's just, those are, those are table stakes. Like you have to recommend products and I would hope you have some sort of algorithm that shows I'm looking at product A, show me products similar to A.
Speaker: Now, if you could layer personalization on top of that and make it more fun, then we're gonna, but that's another thing. But for me, personalization is something bigger than, you know, hi Katie or product recs or something. Personalization really means how do I change this experience in a meaningful way for a group of people that differ from another group of people?
Speaker: So how do you do that? It's like, how do you find these big buckets? For us, it started off when we started personalization, it was very simple. We have people who come to our sites to buy engagement rings.
Speaker: That's a very specific task. Buying an engagement ring is very different than buying any other type of jewelry. So my segments were soon to be engaged and everyone else.
Speaker: That's it. Because everyone else was not shopping the same way as somebody shopping for an engagement ring. This is a month, months, months long process. And there's so much emotion tied to it.
Speaker: So when you think about it from that standpoint, imagine somebody coming to a site, they're shopping for an engagement ring and the homepage is littered with messages about a sale for gold necklaces or watches or something completely irrelevant.
Speaker: flip that on its head, somebody who's been married forever comes to the site and there's a huge engagement sale. How is that helpful? So if you just look at those two segments, that's where we started. And we started getting traction to say, oh, if we have a more relevant message to the right group of people, they're going to move forward. They're going to convert higher.
Speaker: It's not rocket science. roop That comes back to user experience. It's a better experience for somebody shopping engagement rings if they can easily find engagement rings. So that's kind of how it all ties back together.
Speaker: But it's like looking for those big buckets of opportunity to start that personalization. When it came to intent, where do we start? Elementary, one-on-one. It was like, how many page views is a person viewing?
Speaker: And we had certain thresholds for low intent, medium intent, and high intent. You look back at it now and we're like, wow, that was very, that was like common sense. But it was just talking about it. Like, hey, somebody who has low intent, they haven't got a lot of page views. They don't have a lot of momentum. How do we treat them differently than somebody who's been on the site, looked at 15 pages and ready to go?
Speaker: And we started with those. We call them our macro segments. And we started talking in meetings about those segments to everyone. And that's where we started. And we did that for probably 18 months to two years. And that's all we did.
Speaker: Behind the scenes, we were doing other things, smaller things, rolling them out. But that was our talking point. Engagement or everyone else. And what is their intent level? That's kind of how we got that ball rolling. And now you just take that and put it on steroids and go into all this other stuff of how that how intent factors into creating a better experience.
Speaker: And I love that. I was just taking notes again. look you So many good, good one-liners here. But I think that's so many people listening. It's it's a wonderful response because so many people listening, myself included, are like, when you start in personalization, God, there's just so much. So I think it's really reaffirming and validating to hear someone with your expertise say,
Speaker: it's not rocket science, start with your macro segments. Like it sounds so obvious to you, but so many people are just so overwhelmed. they're like, I don't even know where I'd begin. And I'm so glad you're able to share that wisdom with us. So Craig, with that, I think that's the episode. I think we're all good here. Thank you so much for taking the time. Everyone should follow Craig on LinkedIn. We'll have that linked in our about of this episode, but thank you so much for joining us. Yeah. Thanks for having me It's fun.
