Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Michael Toland - How to Make Data Products Work in the Enterprise
37 Plays2 months ago
Michael Toland is a Product Management Consultant and blog contributor with Test Double, residing in Columbus, OH. His experience spans 8 formal years of internal Product Management, with a few additional years of doing Product Management without even knowing what the field really was.
In this episode, Michael shared how a data empowered company the size of Verizon was able to drastically reduce time-to-market metrics, experiment, and run data product MVPs in production. The reference data became a cornerstone of Verizon's go-to-market strategy and a glue for different teams and departments.
One of the key takeaways is that to deliver value with data products and architect them effectively, one does not need to be a data wizard but rather have a passion for solving problems.
Michael is also the author of an infrequently updated product satire site, Dignified Product.
Recommended
Transcript
Introduction to Hosts and Podcast
00:00:00
Speaker
Hi, I'm Hewlett Kuchova, CEO and co-founder at Mesh Radio. Hi, I'm Scott Herlman. I'm a data industry analyst and consultant and the host of Data Mesh Radio. We're launching a podcast called Stray Data Talk and it's all about type and data field and how this hype actually meets the reality. We invite interesting guests who are first of all data practitioners to tell us their are stories, how they are putting data into action and extracting value from it. But we also want to learn their wins and struggles as a matter of fact.
00:00:31
Speaker
And as Ilya said, we're we're talking with these really interesting folks and that a lot of people don't necessarily have access to and that these awesome conversations are typically happening behind closed doors. And so we want to take those wins and losses, those those struggles, as well as the the the big value that they're getting and bring those to light so that others can and can learn from those. And we're going to work to kind of distill those down into those insights so that you can apply these amazing learnings from these really interesting and fun people and apply them to your own organizations to drive significant value from data. Yeah, so every conversation is not scripted in a very friendly and casual way. So yes, this is us. Meet our next guest, and yeah, I'm very excited.
00:01:18
Speaker
Okay, my camera is frozen, but anyways, hello. It's us, 3D Let's Talk, Julia and Scott are back and we are recording our next episode together with Michael Toland.
Michael's Journey at Verizon
00:01:32
Speaker
And I'm very excited to have you. Thank you so much. I'm so excited to be chatting with both of you.
00:01:37
Speaker
Okay, Michael, you have so many projects and ah concurrent projects and and so many roles today, different organizations. i'm i'm I will fail to repeat that, but what rests my attention when we talk is that you were doing data product management back in 2016 when none of us even think about the term. And also, Blasto's edit was happening at Verizon and you know just thinking about the data silos and um you know system silos, whatever you know data types, the cleanliness of the data you used to work gives me goosebumps. Yeah.
00:02:22
Speaker
like she and Looking best at it, i I took for granted a lot that that we had. A lot of organizations did not have what they had at Verizon. Okay, so I want to talk today about data product management, ah what you did at Verizon, but please introduce yourself and tell us all about it.
Data Product Management at Verizon
00:02:44
Speaker
Yeah, so I'm a senior product management consultant with test double. I used to be part of a company called Pathfinder that was acquired. And um prior to that, I worked at Accenture for a little bit, but I really got my chops at Verizon, where I worked for a decade, starting in customer service on the phones, answering you know customer calls, and then ah working my way up through the company um across several different front-facing roles for direct customers and members.
00:03:09
Speaker
um to a systems auditor role doing device testing, where we concluded that everything that you did on your device, we tracked it through the network and made certain that it rated through a billing engine um on our mainframe that billed all and only what a member and customer should be billed.
00:03:23
Speaker
um From there, I did that for three years, helped um build out kind of this internal data product for our team of um being able to capture our test data outside of Excel spreadsheets. And we moved it into a web-based form. And then I got moved over to managing um a massive accounting issue um where then that led after we found about a significant balance sheet issue um that was mirrored into some very thorny pieces um that I don't want to disclose too much about.
00:03:51
Speaker
um we We moved then, I got promoted to being across all amortized products, so device payments and promotions over time. And we'll talk more about that as we go through because that was relatively new. um And I managed the shutdown of Verizon's existing reference data engine for all device promotions, which was a three and a half billion dollar annual spend for Verizon.
00:04:12
Speaker
um into a new platform. um So they built something in house they wanted to migrate that to a third party platform that was more configurable and easier to develop on. So that was like, that's my professional career. I also serve as, um as of right now the interim chair for the Board of Directors for New Leaders Council across the country.
00:04:29
Speaker
um And so that is a relatively new role that I've taken on. um I have um since, because of that, had to resign from the Columbus Symphony as a course member, and I also serve as a mentor for Venture for America. And I am one of several hosts for the Data Product Management in Action podcast. um So who knows when I sleep.
00:04:49
Speaker
Yeah, and I'm glad you also needed a pause to remember the long name in all the podcasts. It's literally why I couldn't pronounce it. I got it. Yeah. Thank you so much. And you know, for stopping by, given how busy you are. It's just, you know what, there's always time in the day.
00:05:07
Speaker
Cool. Listen, um I want to jump into this use kit that you shared with me during the prep call, which you actually did. And, um you know, can you please shape um kind of the situation, what you guys achieved and where did you start?
00:05:21
Speaker
Yeah, so um I'll start with sort of um in 2016 T-Mobile put a lot of pressure through the mid 2010s on the entire mobile ecosystem. The uncarrier model was very popular. It was causing a lot of churn amongst the other big three. So AT and&T, Verizon and Sprint before they were acquired by T-Mobile and what we we What we were challenged with is that they really upended the nature of device promotions by saying, rather than giving you this lump sum $200 to $400 credit when you purchase a device because they sort of modeled device payments as this concept and paying your device over 24 months versus a two-year contract, um they then moved their promotion cadence for cash flow over that same amount of time. So rather than being $200 out the door right up front,
00:06:07
Speaker
you can delay that cash flow over time um and split that up into 24 monthly credits. So we saw that, and I wasn't part of this project yet, but we saw that and we needed to pivot all of our promotions. And so very early, we designed an entire very sticky process that said, and this was back in like 2016, we're going to build enough to present a promotion on screen for this new type of promotion type.
00:06:30
Speaker
delay it by 90 days from actually configuring to give us time to build it and manually qualify customers for those promotions. um So we didn't have everything built. We had enough to present the offer to them.
00:06:44
Speaker
um in our promotions engine to show up online and in our point of sale systems. And then on the back end, we had really robust SQL and data analysts who would go in, identify folks who were targeted for the promotion, as well as individuals who may have fell out and didn't get qualified for the order correctly because it was rapid. And then from there, um we built within all of our reference data, the actual capability to deliver on that promise that we were able that we were marketing, right?
00:07:12
Speaker
um And so it was a piecemeal effort. We built first the present layer to qualify folks. Then we built the back end to manually qualify everyone. And then we also had to build as piecemeal um how we were going to actually apply those credits over time. um Once we did that, then we started
Automating Promotions: Challenges and Timelines
00:07:29
Speaker
automating more and more.
00:07:30
Speaker
So I want to ask a really specific question. How long you guys were experimenting before making it into the automated production? Like how long you were doing this? um more par manual Oh, probably a good six to eight months. Six to eight months. Yeah, because it was a new it was a new type of um logic in our system that we had not yet built. right um And so the way the way that we had typically done things is we have this reference data system which had all of our attributes that someone could apply to a promotion. So what's what's your device make and model purchase? um How much did you have to spend um per month in order to qualify? Those were all attributes that you could define. What were your customer type? How many lines of service? Was this a trade that you required? What were the qualifications of that trade?
00:08:16
Speaker
All of these were just configurable assets um and attributes so that we could then configure it via UI and build out promotions. yeah so So in your case reference data is like metadata for a few customers. I mean, it was a series of relational databases that two things. One, we had a UI that we could define new relational um attributes and tie them all together. And then we had the second layer, which said, OK, now that we have defined all these attributes, how do we build something that the system understands how to interpret, qualify, and then apply, and then pass to the billing system to do its work
00:08:52
Speaker
to make certain that all of those attributes are um verified to apply. There were two giant reference data systems, one for pricing, one for promotions. And scale of a company like Verizon. you know i mean at At that point, it was like 100 million subscribers or something like that. 100 million lines of service, yeah. Yeah. and And I still remember when I bought my phone from Verizon in 2015, I bought like ah one of the Google Pixels, I think probably the Google Pixel. Probably the Nexus.
00:09:19
Speaker
Yeah, that might have been the pixel background. yeah Yeah, like the initial one. And I couldn't buy it outright from them. I was like, I just want to buy the phone. I don't want to do the the pay by my I just want to pay my money upfront and have it done. And they're like, OK, we can have you pay all but 24 cents and we'll charge you one cent a month for the next two years, because that is the only way we can do it, because you can't bring a phone to the carrier. You can't do any of this.
00:09:44
Speaker
So we should talk offline about that. That is something that hurts me in my soul. you absolutely and We should talk offline about that. Okay folks, sorry, you're not going to hear i hear this. to but like i live I don't think I could talk specifically about that limitation, but also um there was ways around that that you should have been given.
00:10:03
Speaker
Well, but it was, it was even prior to you setting up the system, but yeah why don't we define a little bit more of what you mean by reference data? You know, cause some people talk about this as taxonomy, ontology, like ah semantic layer, all this stuff yeah kind of coming through, but like, what are you thinking of when you use talk about reference data?
00:10:20
Speaker
Yeah, so I'm going to talk about it in plain terms, not like data product terms, because I see only way I know reference data to me is a series of attributes that are defined by a domain. So I will say an attribute might be make and model for a phone. And then another attribute might be the memory configuration.
00:10:38
Speaker
um another attribute um about our customers, maybe things like a price plan. Well, then within a price plan, I will have different attributes around the features of that price plan. Well, when I get into features and mapping out this numeric value, is this feature has a description. Now I've got a map out in my reference data, all of the attributes on what that feature actually provides. is it you know Back in the early 2000s, it was like all of our pricing plans were minute-based. So you had like 900 minutes. You would configure an attribute to say, this is 900 minutes available in this pricing plan. Before you got an overage, you would define the overage amount. Every attribute that could be defined, we had a UI that could define the attribute or add a new attribute.
00:11:18
Speaker
And then all of our downstream systems understood because they had business logic to interpret the values that were in placed in that reference data. And so for me, when I think of reference data, I think, or what are all of the possible permutations of qualities or attributes about a given thing?
00:11:34
Speaker
And then how do they relate to one another? And at Verizon, I would say there was probably over a hundred different reference data tables that all understood how they related to one another. And there were clear teams that were domain experts over all of pricing, all of inventory, and all of promotions. I get a question.
00:11:54
Speaker
Well, I assume so. Definitely, yes. um yeah you you know You had contention around you know your demands for marketing, and if there were certain things that required net new business logic be built. um And we didn't have attributes defined for that, and we didn't have the system logic in place to handle those new attributes. That was usually the part that pushed team to shove. But when it was just like when we needed to, you know you figure Verizon launches new pricing plans all the time, and usually on an annual refresh.
00:12:21
Speaker
You can't spend four to six months waiting for engineers when you're already taking usually two to three months of market research to evaluate what the pricing plan should be. And then once you get that, you want to pivot and be able to deliver on that immediately. And so at Verizon, the power of their reference data systems and all their engineering capacity to be able to build out um opposite of hard coded, like just the business logic that could interpret how all this reference data gets rated, billed.
00:12:49
Speaker
um it you could build a brand new series of pricing plans within weeks.
Leadership and Problem-Solving Insights
00:12:55
Speaker
which for a company that huge is incredibly impressive. right um Time to market, this is what the organizations i need to work together, the department, to collaborate, to make sure as the time to market as short yeah possible and this is I'm sorry to jump in, but this is makes so much sense. you know when Once I was first time in the United States, I was impressed with the cadence of how big stores like Target, Walmart,
00:13:22
Speaker
are ready to launch new campaigns. yeah And this ah and and then like you you go with pumpkin spice season, back to school season, then there is the Halloween season and and the Christmas next day. But this is such a huge work you know to kind of to get ready for that. And that totally makes sense. You need data, especially in telecom. Can you define for me? Because what sounds to me, the reference data in your case is very much product, data product.
00:13:53
Speaker
Yeah, I would say the whole end-to-end sort of reference data engine that would feed like there was a UI where you could define all the reference data. um And then there was a UI that helped you test out like the qualification criteria of that um and make certain that it was presented. We had a robust lower set of environments. There was a lot of complaints that it would be very buggy, but it worked the most part. um And, you know, I think um
00:14:22
Speaker
the The key aspect there was probably how much people took for granted that it just all worked. You made an enterprise and you're a team. and and Yeah. and I mean, there was there was this sort of thought process that like once you... I think this is probably true for a lot of products just in general, but like the better you build it and the more reliant people become on it, the more they just kind of take it for granted that it just all works and it always works and it doesn't need like a lot of support, which reference data does. You've got to make certain things clean. You've got to have a team that understands the context of what they're building for.
00:14:56
Speaker
Um, and there was a lot of times where those teams had a lot of scrutiny over the value they delivered. Yeah. and like That's something that's been kind of frustrating for me of in the data space, people don't talk enough about business value versus yeah the data work itself has value. yeah And so, you know, I know you've got a question here, so I want to, but like, yeah and I don't want to answer the question last time because I started losing trainers.
00:15:21
Speaker
but but i But I think we want to wrap that into the conversation of how how do we treat this like an actual product? When we think about product management, product management is about delivering business value. It's not about building a product, it's about delivering business value through product, right? And that product management thing is is is I'm struggling to talk to people in data about that because people don't seem to understand that it's that data products aren't the things that deliver value. It's delivering value via product management, via data products. it's not it's it you like I always talk about the underpants gnomes of step one, steal underpants, step two, question mark, step three, profit. It's like step one, do data products, step two, question mark, step three, profit. and it's like
00:16:08
Speaker
It very much is that yeah even if that did generate profit, if we can't show that it generates profit, then people don't believe it. Yeah. Well, Julia, you had a question. Let's start a question. I have hypothesis why it worked so well in and the horizon because the Yeah, yeah, yeah, we coming back to the person who initiated all of this ah project and who made it happen. And this is exactly what, like this person have nothing to do with data. And this is really the case that you don't need necessarily to be a data person to create data products. You need to be a reasonable person. So please tell us a little bit, what are all of this person who, who lead the project?
00:16:49
Speaker
I think um I have to give credit to the executive director who um taught me everything I knew. um we We sat within a group called revenue assurance and it was within finance, but we really did way more. We were much more systems auditors um and we probably should have been rolled under the operations orgs just for anyone out there who's interested interested in the telecom space. A lot of revenue assurance groups within telecom are getting moved under the COO or CEO.
00:17:13
Speaker
functions instead of finance, um or they're getting split out. um So i want I want to caveat that. But um she was someone who understood two things. And this is what she
Autonomy in Product Management
00:17:24
Speaker
taught me. And I have to say, I have been fortunate to have a lot of really great female leaders in my in my career um who have taught me a lot. um I will say this, women leaders have had a different set of challenges than white men like myself and Scott will ever have, especially bald white men.
00:17:40
Speaker
um But there's a certain thing about us, ah Scott, that I'm sure you you know. but um which But she taught me two things. And then she told me never to use this trick on her because she was too smart for it. But she taught me, one, always say what you're going to do and don't give people an option to say no. If it's the wrong thing, they will tell you. Otherwise, just go. um And then I tried that on her. She said, no, you can't do that on me. I'm too smart.
00:18:03
Speaker
but Oh, honey, yeah, yeah you're in the court. You're too bold. You're too bad. Let's let's let's let the audience hear. But it was a great life lesson. And then the second thing she said was like, we may not be the owners of something, but if we see a problem, I want this department to go and figure out, solve for it, and then find the permanent owners who should do it. um But she's like, we can't just ignore it if it's a problem. And she really gave us a lot of agency. um I think when you think about product management, autonomy is key and trust.
00:18:31
Speaker
I've never worked for a leader—well, that's not true. I think she was the first main leader that I worked under who trusted implicitly all of her individual contributors. um And I think she had a knack for hiring people that were just vastly different than her.
00:18:48
Speaker
um She taught me to always hire people who thought differently because you have to hire people who balance you out. And so that was this is the two criteria that she taught me. But I think from her perspective on that second piece, when they realized there was a problem and no one would solve it, she had built this department from scratch from the 90s.
00:19:05
Speaker
Her thing was how do we build faster and deliver to the business what they need at scale within weeks rather than months. So all of reconciliation for billing products. um She co-created with her teams and hired people who could help build those out.
00:19:21
Speaker
um She helped the accounting teams, even though it was outside of our purview, understand the impacts to their ledgers. And then she helped co-create all of these big reference data systems with her teams to say, what are we doing on a recurring basis? What are the common things about that that we continue to do? And how do we build something that will automate that over time or allow us to configure that without having to involve engineering? She hated, she absolutely hated.
00:19:45
Speaker
the delays that engineering would have, not because engineering was poor, but because um it was just too long to get things to go. And you know one of the things that Verizon's motto internally was is that we operate um as faster than companies a fraction of our size. And to some extent, I truly believe that because I've worked with a couple other large scale enterprises and Verizon could move very, very quickly when all of the energy was mustered towards a specific effort.
00:20:10
Speaker
um And so she just had this vision for understanding how all of these common attributes worked. She hired the right people who could foresee um the need, and there was just so much volatility, particularly in the 2010s, but we had these systems well before I even started. um In the early 2000s, the systems were being built. um She just had this vision of like, we need to make certain that we can move at the speed the business needs us to without getting in the way and being a bottleneck.
00:20:39
Speaker
um And it also allowed us to be a control point for all audits, which is a huge, wonderful place to be ah for stability of your employee base. um That said, we are the ones who are responsible to make certain that any issues that arise for like customer billing or something is usually going to be something we catch, either because we built ah pricing wrong or we found a bug within our rating engine on the mainframe and we built entire reconciliation processes to recycle and spit out those records and correct them.
00:21:07
Speaker
um there was not a single part of our operation internally that could not be automated over time. there's There's a couple of really, I think, useful points for data people in there. um Because I i was the finance person that was embedded in engineering when I was at Tenable. it And that audit, ah if you really want finance to love you and to not bug you about funding and things like that, the more ah that you can take off their their shoulders for audit, the more their
00:21:37
Speaker
Cause you know, our, uh, our auditors came in cause we were going public and they asked me a question and they got off the phone after five minutes and said to the CFO, don't ever put us on the phone with that guy again. Um, and, and, and they were like, Oh, why it was he rude or anything? He was like, no, he just, he answered all of our questions perfectly within like three seconds and also offered like seven layers deeper. Your, your audit capability or your, your ability to, to survive an audit for your AWS costs are.
00:22:08
Speaker
bigger or better than any company we've ever dealt with because I was like, oh, this is my methodology. Here's how I think about it. Here's here's how the information flows. You know, sometimes you can't have perfect to dollar for dollar, but we have this ah method of tolerance. That's where it's about three, four percent tolerance. And we can we can show you we can do this, but sometimes AWS moves things after the fact. And that changes the numbers a little bit. But this is how we do it. This is the exact methodology. This is how we allocate um are prepaid, you know that you talked about amortization. we talked to you know How I amortized the prepaid reserved instances, all of this stuff. And that ability to drill down into what somebody cares about instead of, again, the data work. You haven't talked about the data work. You've talked about having clean flows of information so that people can do business.
00:23:03
Speaker
That's where, where people start to go, oh, this data team knows what they're doing instead of doing data work for the
System Migrations and Technical Debt
00:23:10
Speaker
sake of data work. And so I i think a lot of what you're talking about there is how do you accomplish things as you think about, you have to have somebody focused on what matters to the business and why. And like this, yeah there was a bunch of pressure from a competitor that was coming in and disrupting everything for what you were doing. So you had to change the way you were doing business.
00:23:33
Speaker
So you had a way where you were able to put that in kind of, I just had a conversation with somebody this week about band-aid data and that they're like, this is something that, you know, we have a a cut. We're put throwing a band-aid on it, but if it needs stitches, if it needs, like, this is something that's going to have to get ripped off and replaced.
00:23:50
Speaker
um So maybe Band-Aid isn't the exact perfect analogy, but it still and gets people thinking that this is not a permanent solution. This is a something that we're sticking on for a boo-boo. But if we if we don't fix it, it's going to it's going to be bad. And so I think a lot of what you're talking about there is moving at the speed of of what business matters instead of trying to get to perfect data. Yeah. And i they try to be perfect.
00:24:14
Speaker
The other thing is like we didn't need data engineers to build our data. The UI, I know, right? It's crazy. The UI was so configured that we could just create new attributes on the fly.
00:24:27
Speaker
And they just saved. It was a data platform. I mean, reference data platforms are critical because I could not only build new reference data um based on all the attributes that I needed to define. And listen, there were hundreds of attributes that went into a single pricing plan because you had to build a plan criteria, just a base placeholder for it. Then you had to map either existing or new features you had to go build into that pricing plan. and and then how each of those features rated if they were not previously defined. So there was some mix and match and reuse of prior data. We could copy and make changes to that copy so that we could take from what we are already know worked and just update the configurations for it. But all of that was then created as net new reference data attributes that could just be defined by a business user in the UI based on requirements that we would get from our marketing and pricing team.
00:25:19
Speaker
um So there was sort of like this big sort of push around understanding from the marketing and pricing team what the domain context of pricing data needed to be. What was the attributes for the features? How would it need to be rated? What was the allowance we were going to give people, whether it was a unlimited or tiered pricing, particularly when we were in the the awful times of the 20 times where we got rid of unlimited data. Once I like metered data, I will say this to unbloom the face. That was a ripoff for consumers in the 2010s.
00:25:43
Speaker
um It absolutely was the amount of money that people made off data over just alone was insane. And I don't care. but That was a universal practice across the entire globe. um And now that we're back to unlimited data, right? um That was all just configured on the fly within this reference data platform by a business user following requirements from our pricing analysis team. And then we could schedule for deployment.
00:26:08
Speaker
and then test it in the lower environments and make certain it would work. And then we could push it directly into production with the click of a button. There was no release window that you needed. It was just, we could make it go live within minutes if we wanted to. It's it's super impressive. I'm like, um as a product manager, ah the way you test it, the way you build out and and make your hypothesis work,
00:26:31
Speaker
um at the scale of number of users and, then you know, the risk implied. Yeah. And you had the ability, if you wanted to target, and we didn't do this probably as much as we could have or should have because product wasn't super formalized then, um particularly in the data work. But if we wanted to, we probably could have created subsets of customers to target for clear pricing. And there were times where we would do a little bit of that, particularly on device promotions, we would do a little bit more experimentations to evaluate.
00:26:57
Speaker
as opposed to like large scale national pricing because keep in mind our our group not only handled and consumer mass pricing where you only have a few price bands in the building, but also every custom price plan for businesses. um That's fine. Yeah. And businesses would have another reference data engine that housed all their contractual information, all the specific discounts that their contractual obligations got them outside of the system that I ended up rebuilding and replatforming, which was device promotions. So all of this was built before my time, but when I took on the role in 2018 through 2021 of migrating a existing reference data decision engine that housed all reference data and the qualification logic for device promotions, boy oh boy, that was a lot.
00:27:43
Speaker
Especially and how long have this came out in production? Crazy. 13 years. 13 years. Right? like that Hundreds of millions of people have across this, right? Yeah. I get a question from Greg. Michael, tell me, what was your role back in 2016?
00:27:59
Speaker
Back in 2016, I had just finished an accounting nightmare that had wrapped up where I had built a subledger to allow the accountants to um actually do reconciliation on an individual um journal entry layer um through a clearing account. It was a nightmare. I don't recommend anyone to get into accounting, but if you want to learn how money even moves, um it's the best place to figure out where all of your system issues lie.
00:28:22
Speaker
um So I got promoted to be a product and program manager and I started off just really helping out with deploying and understanding from an accounting perspective as a business user what requirements we needed to make certain we had because we were missing a lot of our requirements because we didn't engage the accounting team at all. We just thought it would work. 2018 came around um and we decided we were gonna do this re-platforming and I kind of became the head of the entire re-platforming for the business side in leading it from business architecture, working with lead engineers on understanding our current system. And then specifically, what were we going to kill? If you're going to migrate to a net new system, we needed to really evaluate what we were no longer going to do in the existing business processes that we needed to modernize, update, and stop doing. So that happened at the beginning of 2018. They said, oh, it's going to take us 18 months. This was a system that had been built up since 2008.
00:29:17
Speaker
um So it was already a decade in and was kind of configured at times with bubblegum and Band-Aid as our executive director liked to joke. um And so we had to understand all of the context and then also what marketing's future plans were. Because we had a lot of things we just never deprecated that we had stopped doing. So we had to evaluate when was the last time these prior these types of promotions that just lived in perpetuity were qualified. Why were they qualified? Were they part of like this other system?
00:29:42
Speaker
um How did we present to the front end that information here? And one of the things we realized we had to do is we have this middleware that existed between all of our systems. So there's a point of sale, there's this reference data, there's this IT system ah for our e-commerce site that really mapped how all of this data worked. And we had to rebuild for this new engine all of the APIs.
00:30:05
Speaker
to handle the news. mode Yeah, which didn't happen, right? It did not happen. um I actually left at the end of 2018 because there was a voluntary severance. um I left for nine months. They called me back as a contractor to come back and take over the role for a lot more money.
00:30:19
Speaker
it excuse yeah know right and It's exactly that thing where you're like, Oh, you see my value now. Okay, by the way, Yeah. And so I went back um and they hadn't actually, they made some progress. I'm just like building out sort of the key platform, but they thought like, Oh, we've only got six more months left and we'll be able to build it. Well, that was the end of 2019. I was there. So 2021 when we launched our initial pilot offers of just simple accessory discounts. That's how far we got to. And um once we got to the point where it was like, you guys are going to be good. You guys now have a clear indication. This thing does work.
00:30:55
Speaker
um And I led the entire team, I had an entire team of onshore developers with about six leads and then 50 offshore developers I met with every morning. um I was coordinating all of these requirements with um a group of about my boss at the time and then all of our marketing counterparts and then working explicitly with our mainframe team to understand what did we need to pass through the point of sale.
00:31:17
Speaker
through the middleware layer, I mapped out all of the qualification logic and how it moved because not to mention because we had in 2016 handled that strange um amortized promotion from scratch, we had two different paths that promotions could qualify through and two different pricing reference data sets on different ways you could amortize promotions once they were on the bill.
00:31:38
Speaker
that were all configured based on the logic of this decision engine. um There was a lot of technical debt that was built because of that that we were accommodating for. But my my role then in 2016 was, um and really into 2018, was understanding how all that worked so that when we got to the point where we were going to replatform, I had probably the best domain expertise from end to end inception of the offer itself to qualification, to presentment on the internet site or the point of sale system, to getting it on the receipt, moving it into the customer's bill, validating then that the system was billing it correctly. And um when you think about all of the line items on your mobile bill, there's a certain hierarchy that things have to apply when you make a payment or discount first. and so Because of tax and all that fun stuff. has is amortization, and this is public, but Verizon um trump takes all of their device payments, bundles them together, and sells them to the open market like we did with mortgages. Oh, yeah, that makes sense. Yeah, well, and and one thing I want to do, yeah, I want to do ah one quick thing, Yulia, I know you've got a question here, but um just defining the word amortize, which means to recognize over time time in kind of an accounting
Financial Strategies in Promotions: Amortization
00:32:52
Speaker
sense, right? So if somebody prepay, you know, you're talking about um
00:32:57
Speaker
There's a discount. You might say, well, we're going to give you a thousand dollars because you're signing up for a two year contract. We're going to do 40 bucks a month. You know, 40 times 24 is nine 60. But like 40 bucks a month on the bill.
00:33:13
Speaker
is it each month instead of, you know, recognize a thousand dollar loss at the very beginning, we're going to say, OK, this is a credit of 40 bucks each month and that kind of thing. So it's just important for people to understand that are not um that didn't have too much exposure to accounting. Yeah, like like the two of us have. My background was an English major in creative writing. I had no skills doing this until I until I got involved. it' here Yeah, my executive director is just like, you seem to like asking too many questions. Go handle this. ah but That is accounting. Like people think it's this super, super rule system. And it's like, no, a lot of it is just like, how does this work and how should it work? I've also said we had a reference data engine that also handled accounting entries. They could build a new ledger entry from scratch, define it and map it to all of the ledgers as well. Reference data was king at Verizon. Like no joke, there was a reference data said ah engine for almost every type of business logic that they could create.
00:34:08
Speaker
it's It's the pre-work to make, it's it's like instead of reacting, reacting, reacting, it's doing the pre-work so that you can do what needs to get done. And but this is and at some point that that devolves and that's why you had to replace it, but it lasted for over a decade. yeah Does it happen for systems at this scale without it being a complete and utter hot mess unless you've done a lot of work?
Ethical Business Considerations
00:34:32
Speaker
but No, no, this is really impressive case where you guys make made data products and reuse them, reuse them at scale and you know. you I mean, yeah, it's an entire reference data platform at that point. I mean, because they had a reference. Yeah.
00:34:49
Speaker
It's a data mesh. Can we count that to be a data mesh as well? I feel like it is. I mean, it was all relational databases. I'll look at you Take off your video. up No, no, no, no. ah yeah Scott's over here shaking his head on our video calendar. Yeah, I don't think we could describe it as a data mesh because we didn't have you know independent domains for everything. like There was a pricing domain and marketing owned the creation of what needed to be built, but the domain experts were within revenue assurance.
00:35:19
Speaker
um Now we had... Maybe not pure data mesh. No, know I think it was more... I think actually the socio-technical data mesh concept was probably more followed, right? like We had clear teams that were responsible from engineering. We had an entire um yeah streamlined equipment discounting is what our old system was called for promotions. There was an engineering team that owned that and there was a prior ah reference data team that managed understanding how that worked.
00:35:46
Speaker
um And then there was like backend tables that were owned by the engineering team, um separate from our reporting structures out of like a Teradata warehouse. um But like they were kind of confined to that system. It was owned by domain owners and owned by that system in its entirety. It was not owned by some centralized data authority.
00:36:02
Speaker
Same with pricing reference data that all lived within our mainframe that then exposed to this UI for kind of common pricing elements that the business on our side operation side of the business could go in and and manipulate and define all of those attributes, add and create new attributes that would just get written into those relational tables.
00:36:20
Speaker
And the engineers just knew the magic that needed to happen to understand how to take this attribute, plug it into our engineering code as a as a referential model rather than hard coding all that logic. Now there was times you needed the hard code and we always got in trouble for it because we had to move quickly and then we had to go back and refactor that.
00:36:38
Speaker
um But I mean, it was it was a substantial operation. And I think that any time, you know, all all big organizations have layoffs. And a lot of the times I think that um to Scott's point, it was really hard for us to continue to sell to executives what the bad thing that happened would be if we lost more people who manage this entire apparatus.
00:37:00
Speaker
You know, I think i've I've got a big, big question coming up, Julius, if you want to do that. but i went but But I think, you know, one thing that I've i've been ah frustrated about the last couple of weeks with ah what happened with CrowdStrike is what was the value of doing ah Canary deploys? What was the value of doing reliability engineering? What was the value of this? And, you know, they had the value, the perceived value of that was very low if they were doing this. This was a a company that was, you know,
00:37:30
Speaker
that was really, really selling as the cloud and wasn't doing Canary deploys, which, you know, when they they first got um built out, I think was in the early 2010s or whatever, Canary wasn't, you know, it wasn't really possible to do that very easily. And so I get why it wasn't built with that natively. But I guarantee there were a bunch of people internally that have been pushing for this, pushing for this. And yet people were like, no, new features, new this, new that. yeah And the cost of that is now going to be, you know,
00:38:00
Speaker
$100 billion dollars to the company, basically. The CrowdStrike thing, ah okay, i've got I've got this working theory. two Two working theories that I'm actually working on blog posts, and i'll share I'll share them here. And I share them also with like the conversations I had at Data Connect. But two things. One, restraint is a virtue, and the ability to hold yourself back and focus on what you're doing before you start something new.
00:38:25
Speaker
um Even if there's a great idea out there, being really committed to hold yourself back um is probably one of the most important business skills I've ever learned um because it stops you from spiraling into every shiny new object. That's number one. But the second one I think that you're talking about is all of these small trade-offs we make um that build up technical debt, documentation debt, operational debt, all of these different types of like debt we talk about in an organization that accrue a certain type of interest that if it gets too big, all of our systems crash.
00:38:55
Speaker
I actually think that that kind of yields over time into this sort of type of ethical debt, where we are making these trade-offs to maximize value rather than reinforcing and making certain that our systems have resiliency. um there's a There's this really delicate play between efficiency and effectivity that we we are over-optimizing for efficiency and that if any piece of that breaks, the entire system becomes ineffective.
00:39:25
Speaker
And I think that this this profit at all costs, these small trade-offs that we make over time build up and cascade into this larger ethical debt. And I'll say it, um you end up with situations like Boeing. That's literally what I was going to say was Boeing. Boeing, I mean, killed multiple, multiple people. Multiple people for the sake of fucking profit. Sorry. if You have really thought about this. um It's insane. It is absolutely insane that we don't hold back a little bit because of the pressures. I think it was Dustin Moskowitz of SANA who has this great quote when he was talking about some of the open A.I. drama. This is getting into my philosophy of philosophy about business. But um he said even the best intentions can't survive touching this economy. Yep. And we're in a ah postmodern capitalism or whatever, be you know, where where it's like a dystopian capitalism. and It's just, you know, this this this desire of how much is enough and the answer is always more.
00:40:19
Speaker
rather than there is this limit of how much is enough. um You know, I have a good friend who works at Anthropic and he said, you know, given a dial from zero or negative 10 to 10 on the scale of technological pace, where 10 is speed up, light speed of technological pace in AI versus negative 10 as we slow down in reverse, um where would you put it? And me and his wife who were best friends were like, 10, let's see where we can go. And he's like, negative two, slow it down. We're not understanding what we're building. We're building too fast.
00:40:45
Speaker
Yeah. And I take this sort of these two competing thoughts of restraint being a virtue. And if we don't have restraint, and we continue maximizing technical debt um to the point where we can no longer pay it down. And what are the trade offs for making? We get to the sense of ethical debt that just compounds as our organizations. And it causes us really big things like CrowdStrike taking down
The Role of Data Product Managers
00:41:04
Speaker
global infrastructure that the world depends on. Really? We still have astronauts. We still have astronauts stranded in space because of Boeing.
00:41:13
Speaker
like This is insane to me that we aren't talking more dutifully about the fact that like we need better indicators for an organization's success that's not just profit. but yeah everything is and but we if you If you're not focused on that, then a private equity company is going to come in and buy you out. and so yeah we need yeah yeah so conversationious these changed But that's that's so outside of the realm of, I'm sorry, we so i but I had to get it out. That's really interesting. yeah But it's it's product man it's product management 101 of what are you trying to do? What is your company trying to do? And and what problem are you trying to solve and why? And when when you talked about um you know the data work is less about data work, it's more about what problem matters to solve right now and for who.
00:41:55
Speaker
Who am I solving this problem for and then why? um This is where's product management. yeah This is pure product management. You can't build a data product if you don't know what problem you're solving, who it's for and why. And you understand the experience of that person you're building for end to end. One of the things that I'm always talking through with my clients when they're working through a big initiatives they'll break out their customers into like these proto-personas and yada yada. And something that I learned from Leo Hickman from SVPG is the necessity to create a concrete construct of a persona that is named, give them a first and last name, talk about their experience before the product and after. And I think we i think something that I failed at Verizon and have learned since then is like how to define
00:42:42
Speaker
the core user and what benefit they get on the other side of this product being built and mapping that value chain that it improves their life. They can do more. They can deploy more readily. um They can test better. um And these are for internal products. But then you know from the from the customer side, we were able to then say, we could qualify you without this 30-day wait period. We could qualify you instantly and apply the credit instantly. And then if you didn't do some things that you needed to to fulfill all the criteria, we would call that promotion back. And we built mechanisms for that that were all automated.
00:43:17
Speaker
um And so like there was so much logic that we built and we understood. And one of the things I constantly was like, why why are we putting out everyone through all these hoops? Sometimes i was creating a bad experience.
00:43:30
Speaker
Manko, but you also, I mean, the fact that you also were responsible for how, I mean, this, you know, this misconception that product manager are so only responsible for why and and who, this is a big mess, misconception stuff, because they are also partially not entirely course involved in the how part, you look at the data, you you know, there are security privacy i issues that you need to, oh you know, to be, to count Like, you know, a lot of logic and logic is actually how that is going to happen. My question to you, which I'm trying to refocus our right now, you know, for the last 30 minutes, is do we, like, how do you feel like, listen, but there is going to be a little bit of concept because I'm coming from a product manager background. Okay. So I was doing data product management as well, without knowing that I was doing data product management. Yeah.
00:44:26
Speaker
um So interesting part is... I don't believe that every team out there ah need a product management, a product manager, like in a classic, classic product manager role. And that is because lots of teams have evolution. Like they evolved, you know, they started to get more product thinking. can They, the data engineer, not data engineer, sorry, engineers asking more sophisticated questions. They start to think about users in a way that they are, want to understand them. Okay.
00:44:58
Speaker
My question to you, do we need data product management, ah data product managers today?
Delivering Business Value
00:45:05
Speaker
No, no, it's not loaded.
00:45:09
Speaker
i think I think some people would think that's a loaded question, but I'll answer it because I i like the question. um i I agree with you that not every product needs a product manager. um If it's a small product and you've got really clear engineers who are close to the customer, great. um And i I agree that in part, we are not um responsible for all the how. And I think that product managers who accept the burden that they also have to figure out how, isolate themselves.
00:45:39
Speaker
um And it is a mistake. Engineers, designers, product managers, all of your cross-functional partners, we all have to co-create these solutions together based on our collective knowledge. There's no one person that can do this job. um Otherwise, the job becomes incredibly isolating and lonely and feels overwhelming.
00:45:57
Speaker
um But the instincts of product to be able to define the right problem to solve at the right time and get everyone to focus, I think, is the most compelling piece of product management and then being able to sell that vision. um That is table stakes. Now, when it comes to like whether or not you always need a data product manager, I think so long as the folks leading the effort understand the problem, have the context of who they're solving it for,
00:46:21
Speaker
Teams, if given enough autonomy um and freedom, can co-create really, really flexible, wonderful, robust solutions um and and understand the long pole, if given the right principles and guardrails to build with it. Principles being like, we're not going to hardcode, we're going to make everything modular and flexible.
00:46:39
Speaker
um where we can. And i thought something that at Testable we like to talk through on the product side is like this mindset of principles that are rooted in this, even over that. To say, we're going to prioritize this, even over this and very other important thing.
00:46:56
Speaker
um because this is what we're going to index on. This is what we're measuring. This is what we think is more valuable than that piece. That's also very important, but we're going to prioritize this. um And that that's a really hard discipline that I think product can do. But if you have an organization that espouses those types of principles um and continuously reinforces them, then yeah, you may not always need a product manager for all of the products. um It's helpful.
00:47:20
Speaker
um but But yeah, I would say you don't. um I do think data product managers are needed for a lot of big initiatives, particularly when you were trying to figure out from the business perspective what value your data can deliver for you at scale. And again, I have never worked on a data product that's sold to the open market. Some data products are, some APIs are, yada, yada.
00:47:44
Speaker
um I have almost always worked on really thorny internal products for better or for worse. That's a harder thing to sell because I'm not necessarily driving one benefit value. I'm usually deferring cost or I am accelerating speed to scale and market. um And once you once you once you deliver that scale and it can be repeated, people take it for granted and then suddenly your cost again. yeah i Leaders need to be constantly reminded, had we not built this, we would have taken X amount of extra years to even get to this point.
00:48:18
Speaker
And that's something that I challenge for a lot of my clients and a lot of people that I work with now on the other side of having built that system and working with some really talented people is the reminder of had we not built this, your organization would be this much further behind. And so you have to not look at this as just a cost center that you're continuing to maintain.
00:48:36
Speaker
But it is a product that you need to continue to support, staff for, and continue to modernize as you flex
Strategic Focus in Projects
00:48:43
Speaker
the system. And so when when people are just making layoffs because they're trying to hit some some quota number to maximize the revenue and profit margins, um I take a lot of issue with that because the vast majority of time is not totally thought through. Everyone's a headcount as opposed to looking through and saying, if I don't have these people, what's the bad thing that happens to my product? And can I continue to self-sustain?
00:49:04
Speaker
Yeah. role Role versus responsibility. It was a lot of what you were talking about of every product needs product management, but every product doesn't need a product manager. but like now that That's the thing, right? like You to have somebody that cares about this yeah and that yeah at end and exactly the whole thing. of we you I had Sid Shaw on from Airtel on my podcast and he was talking about, I can't tell you the business exactly how much more valuable it is to go.
00:49:35
Speaker
you were three to four weeks to implement ah an experiment, like a small scale experiment, three to four weeks to get your information back. Now it's three to four hours, three to four days, right? And so within a week, you can have an answer. And you what's the value of being able to do that quicker? And what's the value of being able to do much more small scale experiments instead of grouping together six things that you have to do because it took three to four weeks anyway to implement it. So why not make it four to six weeks and implement six things at once and try and figure out Which of these actually have the impact or which not? um ah hundred but The other thing that is also something that we don't talk enough about is it tells you which initiatives to kill faster so that you don't have some cost. Stop wasting money. Stop i' wasting money. like But it's more than that. It's not just waste of money, but it also dilutes the attention of the people.
00:50:25
Speaker
Oh my gosh, right now you know my client is getting ready to undertake a rather significant program. And if they miss it by one month, it's going to be a $5 million, dollars a bit of hit to their margins because they're giving some um supplements. But because of the way the the orchestration is happening, if they miss it by one month, they will not be able to catch up for three months.
00:50:46
Speaker
And so what I've told them is every single person needs to be laser focused because if you get distracted and there's too much context switching, and this is a huge data play because they're building an entire new system in-house, you will lose $15 million dollars of a bit of margin because you're distracted.
00:51:03
Speaker
And any value that you're going to bring in needs to exceed that potential risk. Because if it doesn't, it's not worth pursuing right now. I don't care what you're what what what you need. You are staking your business on this. um And then ah know a couple of the product managers really resonated
Philosophies of Product Management
00:51:20
Speaker
with that. Someone was like, well, what about member value? We have to continue creating value for our members. I was like, sure, that value needs to exceed this potentiality of loss if we're going to context switch on it.
00:51:30
Speaker
Because can't you cannot have more than two to three clear company-wide initiatives. And if everything is not fully aligned to that, um you are going to burn out your people, you're going to burn out your engineers, and you're not going to focus enough. um This is where like I love talking strategy more than anything else. um Because I'll be honest, you guys, as a data product manager, I'm really bad at the day-to-day administrative work. It's just not my better.
00:51:50
Speaker
but But this is also the ah classic product management prioritization of things. Like you need to prioritize what matters to the business and make sure everyone is aligned on that. yeah or astoomeric It has to be vertically aligned within your department, all the way to your executive function, but also horizontally aligned. You cannot get good work done if there's not also clear horizontal alignment from the executive team that gets cascaded down about what it is that we as an organization are prioritizing.
00:52:20
Speaker
and then holding everyone accountable to that um and and being really ruthless. I mean, I'm a big fan of Christina Wojcicki's radical focus book um and the concept of ruthless prioritization to say, this is a great idea. It's not a lie. We're going to put it on the back shelf. But this is, yeah, it totally makes sense.
00:52:41
Speaker
um So i had I had two big questions here. um and And whichever one you want to go to and at one you you might be able to even an answer with a one sentence answer But I think it's too big of a question for it, but you you choose which of these you want Is data product management more about data or product management? And, and we could even get into, is it more about, is it data product product management or is it data product management? and and The difference is there. And also from everything you've learned, you know, that we've talked about from this big Verizon thing and, you know, your more recent work around data mesh and and things like that. What are kind of your big recommendations for people that are kind of.
00:53:23
Speaker
you know, whether it's individual, but probably more at the organizational level of of like, where are the things that they should focus when they're thinking about data product management? Because I see all this stuff and there's again, this whole thing of data products are the point. And it's like, that's, that's missing out all of the product management. That's data people focusing on data work. So either of those two, whichever one you want to go to.
00:53:45
Speaker
Huh? I'll tackle both. Your first question, um is data product management more about data or product management? I think 100% it is about product management. Data is data is data is data. And um something that I constantly tell folks is that no matter what your facts are, they'll never be compelling. Facts are just not compelling. It doesn't matter until you wrap it up into a story.
00:54:10
Speaker
Stories and narrative are what sell. And stories and narrative are what motivate people. We're a social creature. where're we're we're We're rallied around the concept of the bonfire and sharing stories. And so if you're going to, you're going to build out these really robust practices around data product management, you've got to know how to tell the story that the facts of your data are sharing and, and data is interpretive. Um, there's multiple different ways to slice data. And so I think the key point is really around that storytelling and the narrative and understanding here's what the data says. What's the narrative that I'm trying to get to on why this is a compelling thing that we should work toward.
00:54:47
Speaker
um So I think that is product management to a T. I think that is one of the most important pieces to answer your first question. The second one, from everything I've learned. Now, I might answer your second question before you have the chance to dive in. You've got her finger. um That's the big recommendation for people about data product management as they move into this work. um I think what the big lesson I think I've learned is that you can't do it alone and not to try to think you can.
00:55:14
Speaker
um I'm certainly not an expert at data engineering. I can barely write SQL statements. Now, I've gotten better. um I've also gotten worse. But like that's not the point. The point is have is to have good contextual understanding of work processes and flows, how people move through their day, and why this data matters to them, um what happens when data is not accurate. And if you start getting those sort of bigger, thornier pictures, other people are going to help you with the day-to-day tasks.
00:55:42
Speaker
and the day-to-day work. But you have to help them piece it together as a product manager and outline what the problem is that they're struggling to identify and then figure out solutions for. Because 99% of our problems as organizations from my perspective is people not articulating clearly enough and reiterating what problem is being solved by this work. And then laser focusing on that problem to identify more and more context about the problem as they build out a solution and being willing to pivot based on the results of what they're building, whether it is, yeah, we're building this thing and and we're finding good signal that
Iterative Development Approaches
00:56:18
Speaker
it's the right thing or we're finding good signal that it's the wrong to build or just kill it. Please kill it for being willing to do that. Yeah. And constantly referring to why does this matter? yeah Right. Does this matter? And andd when you find out that it doesn't matter that, you know, you if if there's no baby in the bathwater, you don't have to worry about throwing out the like
00:56:41
Speaker
you know Yeah, and also what we'll learn is you could take the baby out of bathwater and there's so there's stuff there that might be useful in the future for different initiatives. There's little toys and stuff in the in the bathwater for the kids for their the future. Any better without first out there? any any Any better things to compare you folks?
00:57:01
Speaker
I don't know. It's like an American thing where you don't throw the baby up bath water. Yeah, it's okay. It's something I didn't know. Yeah. Yeah. Yeah. I was gonna say without without understanding that analogy of you are following that right you throw out what you don't need, which is the ba water after it's served its purpose or it's no good, but you don't throw the baby out with the bath water. Right. It can seem like when people would like go into the well and get water and then have to dispose of it outside.
00:57:27
Speaker
I'm glad this is a cultural, something you know important you know it from the from the childhood. Yeah. It really could. um But yeah, you know that's I think the lesson that I've learned more and more in my career is like I don't have to be the expert. And they're they're all technical product management experts. And I consider myself kind of technical, even though I'm not like a developer. I don't have consistent background. You're just dangerous enough. I'm just dangerous. And I'm willing to ask a lot of questions. And there's the other lesson that I think every product manager needs to realize.
00:57:55
Speaker
acknowledge what you don't know publicly often and sit in humility in your stupidity I am why is humility no no no I think it's being brave enough to tell you to know something and and people normally don't do that at like when you all wish yeah so um you know this concept of like I don't know the answer to that question being unacceptable as a response in business, I think is toxic. I think it yields into ethical debt because then you make something up, you aren't factual or truthful. If you don't know an answer to any level, and I don't care, if the CEO asks me a question and I say, actually, I don't know the answer, that's a great question. Let me take some time to think on it and get back to you. How, when do you need the answer? Now, granted, CEOs don't like that.
00:58:37
Speaker
because they want certainty. But see bye we've got to move away. And I've written an entire blog post about this on TestableSight. But this idea of certainty versus clarity is really important to me. It goes along with this whole concept of restraint. It's like the work of product management is not about certainty. It is about uncovering and being more clear about what things are and recognizing when things still aren't clear. um And moving towards clarity versus certainty is, I think, just tantamount to what this entire career path for anyone is necessary to understand.
00:59:06
Speaker
And it's also very much resonates with me with making things done rather than making things perfect. Correct. Don't let perfect be the enemy of good or good enough. Yeah, but you can't. Incremental of government. Yeah, and this is the same. You clarify something step by step. You cannot build a system perfect from from the scratch and because you uncover more and more use skills. This is exactly the same.
00:59:28
Speaker
right lo you know When I re-platformed this decisioning engine, I had to scale down and consistently remind people that like yeah we're going to get all these attributes defined, but we're not going to build the most robust offer first.
Reflections and Future of AI
00:59:39
Speaker
That's insane to me. and We're going to prove out this thing works. We were operating in both systems for probably two years.
00:59:46
Speaker
yeah um And that was a nightmare for people because not only were we educating folks for a system on building that, but they were also just swamped with other things. Like around the holidays, Verizon would have to build um sometimes five to 600 different individual offers to get all the permutations of an individual promotion um campaign to run correctly. In the new system we were building, we were promising to shrink that by factors of 10, where they didn't have to build hundreds of offers. They could build 50 to 60 into all of the same qualification criteria done. That was one of the things that I wish I could have sold better.
01:00:22
Speaker
um about the day to day and like how much we've taken for granted, how quickly we can build all these promotions at scale, but how much quicker we could in this new system. um That was huge in the ability of like being able to just maximize time and again, shorten the amount of leave time that marketing needed to provide us with what their campaigns that they wanted to run and how quickly we could turn those around.
01:00:45
Speaker
Okay, folks, let's wrap up. Michael, thank you so much for stopping by. Yeah, thanks for having me. It was an extraordinary talk and I feel like, it you know. We covered a lot of ground. Yeah, that also like, which was saying resonates with me so much and it's just about being sane and and I don't know. I was approaching things from a a reasonable perspective instead of a, I have all the answers or everything will be done perfectly. Or it's it's like this iterative human understanding instead of treating the data people like they're, you know, a data system.
01:01:26
Speaker
were not think of the scene yeah I think the real model doesn't apply to technology, and we've got to get out of it. I'll talk about that too someday. But, um you know, yeah, this this human model, how do we approach things and understand new models? How difficult it is not to take for granted what we built to meet the needs and that we can continue building things that are more I think AI is going to be revolutionary. I know a lot of people are really starting to use cases. We're focusing too much on the consumer side and not want internal business processes to automate and be coefficient-affected on tasks. Really think of where startups can really accelerate to enterprise-level scalability. um If they do, there are a few shop-prompting vials for a lot of the reconciliation audits. I'm thinking a lot about this and trying to espouse this and understand what one would sell. But it's huge. And so, yeah, but thank you for your ramble.
01:02:19
Speaker
That feels like another episode for us. Another episode for us. Thank you so much. yeah That was a blast. okay