Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Scott Hirleman - Cost Managment in Data
35 Plays18 days ago
Scott Hirleman, founder of Data Mesh Radio, has a very diverse career, including a role as FinOps Manager. In this podcast, we sit down together to discuss his FinOps philosophy: cost containment, cost attribution, and cost forecasting. It's an insightful conversation on how to balance costs for data teams with both data and business considerations in mind.
Recommended
Transcript
Introduction and Focus on Scott
00:00:01
Speaker
Hi everyone, it's Trinae the talk and today it's two of us, me and Scott. This is because Scott was complaining that it's not enough time for him to talk when we have guests over. So today we decided, you know, Tejoses wanted be like very unique. um podcast when Scott talks a lot.
Understanding Cost Management
00:00:21
Speaker
But anyways, jokes aside, ah together with Scott, we were chatting a lot about delivering value, using data, delivering data with, delivering value with data, um and in
00:00:37
Speaker
What else? but but We're going to be talking about cost management in data because I think this has been something that's that been a challenge that I hear a lot of people talking about when people talk about what's your return on investment? What's this? You have to think about how are you limiting your costs, but I think We also in in data think a little too um specifically about what is what is a cost, right? Like, oh, it's the exact dollar on the um on the income statement or whatever.
00:01:13
Speaker
And so we're going to be talking about that and and and I'll give a little little bit of background on where I'm coming from on this so that it helps people to understand why we're we're jumping into this. So I was you know I've done a whole bunch of different things you know I've done like stock stuff I've been I've done.
00:01:31
Speaker
ah community management and tech and data and um even was a recruiter, a tech recruiter for a while. But um I was brought into a company um about six, ah eight months before they went public to help them manage their spiraling AWS costs. So they're just general cloud costs that included their data costs, although they weren't as focused on data then as they are now. um But I was brought in to help them understand where they were spending money, why they were spending money, and stop kind of the exponential growth. Now, but they had just moved towards launching their service publicly um as on the cloud because they had been much more of a um download the software kind of company before. So the company is called Tenable and they they do security management stuff.
00:02:26
Speaker
But I was brought in. and And so naturally the growth was going to slow somewhat. But I came in and I helped them really manage that cost. And about you know after about four months in our fully loaded costs, including our reserved instances and and like how you have to amortize those over time.
00:02:46
Speaker
over the next 15 months, our number of client assets, that that's what they call them, um were up over 400%, but our costs were up 21% or maybe the bus. Oh, this is impressive. It wasn't to all me, but I learned a lot of philosophy because this was even before people were really talking about FinOps.
Introduction to FinOps
00:03:13
Speaker
We're going to be talking about FinOps, but we're going to be talking as well just about general business philosophy around cost containment, cost management.
00:03:25
Speaker
how you as the data person can, you know, manage your costs, but also prevent yourself from getting slapped with the, your costs are too high, cut your your costs by 25%, right? You know, every once in a while you're going to get that no matter what, because the people that you're working with just suck. The people in finance, if they come to you with that and just say that with no context or no reasoning, they suck, right? They're not doing their job correctly. And sometimes that that happens.
00:03:54
Speaker
But the best way to prevent that is by building up a relationship. but I will talk about that. um I know data people want only data stuff. In a lot of cases, we've talked about this. of they They only want very, very solid answers versus go and talk to people, but but we'll talk about that.
00:04:13
Speaker
so um But, you know, I managed to really, really stop the bleed and that we did some major cost cutting. But also there were a lot of things where I suggested, hey, we could cut costs by doing this and they just didn't do it. um And so understanding prioritization as well and communicating that of like, hey, if we do this, then this happens. Right. And and putting those that optionality in front of people. so Again, especially people in finance, especially especially people who control the budget, if they don't feel like they have control, they're very likely to do something so they assert control. right They say you have to cut your budget by 25% because they have no idea
00:05:03
Speaker
where you're spending your money and why you're spending. So of course they're not comfortable with you spending more, you know, with your, your spend going up five, 10% month over months because they don't know why you're spending that. And you're like, well, you know, data is the future of the company. Get specific, right? So, so i we're, we're going to talk about a lot of these different things, but I'm happy to go into my general philosophy and we can start there or wherever you want to start with this.
Cost Frameworks and Financial Implications
00:05:31
Speaker
Okay, so first of all, shout out to you for putting it in this way. I love it. and um But yeah, you mentioned, let's go into specifics. So ah we can, of course, touch base on how hard it is to work with finance guys, but um I think what Slack and on the market is, we're needed people to understand that.
00:05:57
Speaker
um, computing data storage, not happening in cloud. As you know, new generation of data engineers, uh, thinking is actually happening on the service, physical service somewhere, which company is slow to be, need to pay for. And, um, was this in mind? Um, also what I realized, um, building had that what distinguished a good engineer from a great engineer.
00:06:27
Speaker
It's understanding what they are doing, not not just why they are doing, but how much it costs to do that. They have in laying out the options how to do that in kind of a lean perspective or where to push and to innovate faster with this margin of having it a little bit more expensive, but at least knowing about it.
00:06:50
Speaker
Okay, so you mentioned that you have this framework, like, as you you you told me, you have this framework, how to think about the cost and how to think about on working with data costs. You mentioned the cost attribution, cost remediation, cost containment. Yeah, cost cutting, cost containment, so cost prevention and kind of managing as you're continuing to spend. And then there's- Cost distribution.
00:07:26
Speaker
Yeah, cost attribution, which the is the accounting thing, and then um as well the cost forecasting. Because again, those people that that they need to understand where, you know, hey, what what are the levers we can pull in and what's going forward. And and I actually did want to search circle back to one thing that FinOps is is great and FinOps is terrible.
00:07:49
Speaker
So, so FinOps is great in that we can get more data about where we're spending and why, and that it's not just this kind of the cloud thing is, you know, the cloud bill is this black box, right? Hey, we're spending this much on BigQuery. Well, blah, you know, and mass ed obviously helps with but people understanding where they're spending and all that stuff. That's why, why I'm literally an advisor, why I reached out and said, I want to, I want to get involved with this company because you're doing great things with that. But,
00:08:20
Speaker
For data people, FinOps can put you down a bad path of trying to focus too much on the data side of this. On the exact, what is our exact cost per query or what is our this? We had this where we were trying to figure out our cost per query when I was at Tenable and we had this we had this customer that was relatively large that had the most inefficient query of all time. And so when we, we looked at our, our overall cost on a query basis, it looked like their costs because they were only running this query once a day or once every hour, it looked like they were actually pretty low, except for what they did was pull down all of their data every hour.
00:09:08
Speaker
for all time. And so you know we had this other customer that was 15 times larger, maybe even more than that. you know It was like 70,000 assets versus over a million. And this um this other customer really, really used their APIs incredibly efficiently. They ran this stuff ah very, very well. And so they ran way more queries. So it looked like if we just did our average cost per query,
00:09:39
Speaker
You know, I just felt it's the best metric ever. Yeah. Yeah. but But I mean, even when we'd say like, if somebody called this API. if somebody ran it with the proper filters, it was a very, very cost efficient API. But if somebody didn't run their filters appropriately, this thing cost, you know, ah every single time they were downloading a couple of gigs, right? And so, ah or more like, you know, I think it was like 15 gigabytes every hour.
Balancing Costs with Business Needs
00:10:13
Speaker
So you know you start it starts to add up and that that customer, just because they weren't running effective filters, were costing us $1,000, $2,000 two thousand plus dollars extra a month for no reason other than nobody went in into their stuff.
00:10:30
Speaker
and so FinOps gives us certainty around things, but in a perspective that's often off instead of, why are we doing this? right that customer The reason we didn't go in and fix it was because that customer was a real pain to work with, but they were willing to renew as long as we didn't go in and harass them and tell them to do things correctly.
00:10:56
Speaker
so you know, ah hey, you get you get this big chunk of money. And so, yeah, they're eating into our margin. But if we didn't harass them, they were very, very difficult. I think this was the one they were very, very difficult to deal with. And so, you know, when you look at that from a FinOps perspective, this customer is bad. When you look at it from a business perspective, bre yeah Well, and they're going to renew if we don't go in and fix their stuff. So do we go in and fix their stuff and make them more efficient or not? i That's where you start to get into trying to dig too far into the data without understanding what you're doing. Context. Context. The business reason of this.
00:11:39
Speaker
And this is also, you know, which was saying being reasonable, what we're trying to achieve with it. Are we trying to optimize and, and you know, and ah achieve phenomenometrics or are we trying to get paid?
00:11:54
Speaker
like
00:11:57
Speaker
And I think I think that's so I want to make sure that we also Don't just talk about the frustrations, but we get into real advice for people But my my my framework again has has these four different pieces and and I want to talk about the last two That are the least sexy the least interesting but why they're important to and also that they're not that hard in a lot of cases and but why they will make you your finance team's best friends. Just to clarify, so this was the, I guess that was an attribution part, right? when we attention and and Yeah, yeah so so the actual ah accounting allocation and the forecasting. So let's talk about a accounting allocation and what that that actually is and what that means.
00:12:44
Speaker
so every month, um your company has to run through their actual accounting processes and say, where are we spending money and why? and you know There's all these fun accounting rules. do not Don't look up 606 and how much fun that was around revenue attribution and all that stuff. There's so many fun things that they have to deal with where the accounting profession just creates problems for your accountants, right? And your accountants are generally going to be under your finance team. Every month they have to figure out, okay, this cloud bill
00:13:24
Speaker
Where do we actually put that into? you know ah Was this cost of sale? So was this something that was actually against are what we sold? Or was this actual just sales cost? you know It was our demo cost.
Communication and Forecasting
00:13:38
Speaker
Or was this for marketing spend? Or was this for, you know our our you know again, the product would be cost of goods sold? And so like you end up with these these different buckets that they have to put the money into. And if you take care of that for them, your accounting team is going to love you. And it's not that hard to develop a
00:14:03
Speaker
a framework for ah taking your costs and and and bucketing it into this. you know um My team at Tenable wanted this done by the fourth business day, and i I negotiated them. They wanted it done by the third business day, but we used Cloud Health, um which is a ah thing for tracking costs that was bought by VMware, I think.
00:14:26
Speaker
and anyway and they um ah which has now been brought by Broadcom. But ah anyway, they... AWS costs, because of the way AWS does stuff, doesn't actually settle until the the third um day of the month. So if they asked for that day, things might actually change if we went back and recalculated it. Because where they apply a reserved instance can change. Because you know if you've got this pool of 100 reserved instances and you were using 150 instances in that you know yeah one time frame,
00:15:02
Speaker
they will just randomly apply them wherever they want to apply them and all that. And so I came up with a framework. It wasn't that difficult to do this. you know You just have to think about asking the questions and then has anything changed? But I quickly did that for them. So every every month, they didn't have to ask me for this. It was delivered to them. And it was delivered down to the cent and said, hey, here is here's how we allocate it to each of these buckets.
00:15:30
Speaker
And then, you know, their auditors came and asked me a couple of questions and then ran away and said, don't ever have us talk to that guy again, because his framework is the most comprehensive we've seen. It's like it really wasn't, it wasn't that difficult. It was like this account is for.
00:15:46
Speaker
R and&D. So therefore we put it into R&D and this this account has our canary deployments. So that's still R and&D, but the rest of the account is for cost of goods sold. So how do I allocate this cost out? And again, that made me best friends with them and it made me understand what are their problems, right? Their problems are they don't want to have to go in and understand this bill. So if you solve that problem for them, that's one aspect of it, right? That's the aspect of just the accounting side. I wanna highlight this idea. Basically, you are acting as sort of labeling person with the data, okay? You are labeling the billing, okay? You are understanding where it's going. So you need the communication part
00:16:38
Speaker
We went to people, asked where these clusters used, ABCD, and then we went to the finance people and told them, sent them reports which speak their language. Yeah, they didn't have to care that, you know, well it was like, what is our elastic surge cost?
00:16:56
Speaker
They don't care. What is our Kubernetes cluster cost? Inside, they don't care. they They care about why costs are changing. And and that's that's the forecasting side. Now, our company was relatively easy because we didn't have really elastic up and down costs. We just had a Kubernetes cluster that was just a stable number of nodes, which is kind of not how you're supposed to do it, but it was also very useful because we just had this very, very stable um cost ah thing. And we knew when we were going to be upping the number of nodes, same thing with our Elasticsearch clusters and things like that. And generally you don't want to be tearing up and down here or your ah permanent database infrastructure. But anyway, we can get into the philosophy of containerization of of database infrastructure. but um But so I knew
00:17:44
Speaker
where we were spending money and why, and I had everybody come and talk to me about if we were going to be increasing our spend. And so I could talk to the people and tell them, hey, ah you know I would do this report every week as to what is our spend looking like for this month?
00:18:04
Speaker
What else are spent to date in this month? How many days have been in this month? And yeah there's always that fun of um February has less spend almost a minute. What? Because there are some services that are billed on a per hour basis, but there are many services that are billed on a per month basis. So on a per day basis, your February costs actually look higher for those services versus anything you have to like kind of figure out, OK,
00:18:32
Speaker
You know, i it was, it kind of what came out in the wash, but I would, you know, kind of make sure there wasn't anything that was too crazy on that.
Justifying Increased Spending
00:18:39
Speaker
But I would just tell them every week, this is what our costs are tracking to this month. And this is what we've spent to date. And and it made me also catch things of, you know, we had this issue with our Kubernetes cluster where when we were doing a blue-green deploy, when we were spinning, you know, ah changing um ah one set of nodes from production to kind of non-production and you know putting those nodes on pause, we were bringing in the other nodes. that This is what anybody who's done Kubernetes stuff knows blue-green deploy. It's not that important for data people to understand. But anyway, we were switching um our instances, but when we unattached the storage, we accidentally made a copy. So we just had
00:19:25
Speaker
50,000 a month of extra cost, but I caught it on day four. So yeah, we spent an extra 10,000 bucks, but I caught it pretty early and we figured out the the the issue. And so as well, communicating those things of, hey, our spend is this much higher because we had this blip.
00:19:49
Speaker
If you're constantly in contact with your with your finance people, again, some people suck, but if you find if you have the right people in your company, if you're keeping them informed and you're letting them know this is where we're spending and why, they're going to be much more okay with the spending because they can understand why you're doing it, right? it wasn't you we We had this problem where um our Elasticsearch costs were going up, but I couldn't really understand why. And it looked like we didn't actually need to expand our our cluster from the metrics I was looking at. So I went and I poked at our our SRE team, the site reliability engineering team, who was in charge of kind of the infrastructure and all that. And I said,
00:20:35
Speaker
Why is our spend going up? Can you help me understand this? And they said, well, we have to spend a little bit more than we planned this month because our time to query in our user interface went from half a second to one and a half seconds. And we we have to get it we want to keep it at that one and a half seconds. And I was like, well, why do we want to keep it at one and a half versus half a second?
00:20:59
Speaker
And and we we dug into that a little bit. I said, tell me how much it's going to cost for us to get back to half a second. And I went and and I talked to product management quickly and said, this is something that's a problem. Why didn't anybody flag this? And they said, well, we thought you were the guy whose job was to prevent us from spending money. And it's like, no, I'm the guy who's making us spend effectively. So I immediately went to them and said, we need to spend more money because our our actual product sucks right now. The actual user experience is bad. And so they immediately went and and allocated another like 4% a month immediately. And then it caused our growth rate on a monthly basis to increase as well. Because I could say this is why we are spending. It's not. We are spending more on BigQuery to do better data or to do better pipelines. it's
00:21:55
Speaker
our user experience sucked. And again, that's that's, we can go back to um kind of my my points about cost um containment and cost prevention and or cutting, but this is that forecasting of understanding why you're doing it as a business instead of understanding what you're doing with the cloud infrastructure. yeah and And why? and And giving them that lever and saying, hey, we're at 1.5 seconds for a query resolution. Do you want us to go back to half a second? Here's the cost.
00:22:29
Speaker
Oh, yes, do it. like they immediately i got I got a message from the CFO saying, thank you for alerting us to this. Yes, do the spend. Within an hour of me figuring this out, that this was happening. Is every company going to be like that? Is every company going to be like spend, spend, spend? No, but they understood that cost. But ah this is also justification of the spend. And it wasn't the FinOps justification of like, this drives, you know, $18 more of revenue. They just realized we couldn't sell the product and we couldn't demo the product for people and have it be this bad and still close business.
00:23:14
Speaker
So like that was just understood and that's that's that exactly what you're talking about of cost management within the data world. It can be difficult because sometimes yeah end users don't tell you what they're doing. But from my perspective, that's when you block them off from getting work.
00:23:33
Speaker
where you go, if you're not telling me how this is going, like, I need to know this so that I can understand where we're spending and why, because that is my job. And if you have somebody that's that obnoxious and isn't going to tell you, you know, you're in a bad spot, but 98 to 99 percent of people will give you that information if you're asking and telling them, like, I need this to be able to do my job and to serve you better, because if you tell me that this is the most crucial infrastructure,
00:24:03
Speaker
Then i will I'm going to be putting far more alerts on it. I'm going to be doing that. like You need to tell me why this is so we can understand where we're spending and why. and Then you can communicate that again proactively.
00:24:16
Speaker
to people so that you're you're you're not having that person come in and go cut your budget 25%. Because you've already said, we're spending here for this reason. We're spending here for this reason. Here is here's the target outcome. OK, this is not going well in this space. Like, hey, we're we're spending 20% of our budget on product line XYZ. Product line XYZ is not going well, and we're de-emphasizing this product, and we're looking into shutting it down.
00:24:43
Speaker
I know that earlier because I'm talking to the finance people and I start to figure out how we can cut our budget there because I don't care that my performance is is worsening there because we're de-emphasized. We're getting rid of that business. we're go like that you start to have those conversations and that you're asking people, why are you spending this money?
Transparency and Financial Trust
00:25:05
Speaker
Why are you doing this? And not in a justify it to me. It's I need to understand so I can justify it to the people that hold the purse strings.
00:25:16
Speaker
to keep getting you your funding. like Do you want to continue to get funding? Or when they come in and say, we're cutting 25%, I'm cutting your stuff because you won't tell me what's going on? They have that human to human conversation. So i know I know I keep interrupting you when you're going for a question, which which has never happened before on this podcast. It has never happened once. Never. Never. Never. Especially in the presence of guests.
00:25:41
Speaker
So, I mean, I will use this talk. You have experience in it. But here's a challenge. I can agree one major thing that you keep emphasizing is being transparent.
00:25:55
Speaker
It's just creating this transparency. And even if you cannot forecast something, you know, precisely, you can at least show things which you're expecting. Nobody is expecting from you, you know, Penny, like being accurate and not short on painting, whatever, right? So just create the transparent environment. How do you see, how is where is going and what is expense? This is clear and very much related to data space as well.
00:26:25
Speaker
But there is one bug. So the cost for the infrastructure and when it comes to the product that companies are investing in to get the revenue, which is not necessarily the data product itself. Sometimes it has a UI and there are different spendings to actual product of the company compared to data products. We need to distinguish those. So when you're talking about the spans,
00:26:52
Speaker
that you need the clusters, you need elastic storage, whatever, right? This is a different spans compared to data spans. So what I saw from talking to the companies, the data teams sometimes take from 10% to like max sortie. In edge cases, it could be 70%. What I'm trying to say that is still Majority of the companies invest heavily in software infrastructure to run their business, not actual data. So data is going to be the smaller piece of it. Okay. And also the data is very fragmented and data phenoms practices are
00:27:37
Speaker
How do you put it? Not garbage. They're hot garbage. No, no, no, no, no, no. No, they do not exist. That's what I mean. Hot garbage. It's not even garbage yet. It's before that phase. We're not even dealing with garbage in, garbage out. We don't even have the garbage creed. Oh, shift left. Shift left. Shift left that garbage. I'm sorry if I have rendered anyone's feelings here.
00:28:05
Speaker
i I did want to circle back on one quick thing that I've said before because yeah we were talking about. um so When you're in in the know about what's happening with your spend, nobody is, no sane person is expecting exactly what you said, like super, super accurate forecast. yeah But the thing that I tell people that the finance team hates the most are first, surprises. Second, surprises. Third, surprises. Fourth, spending too much money. So like their job is to
00:28:42
Speaker
as accurately as possible forecast the spend. And so that is something that at and to justify why are we spending spending in appropriate areas, all that stuff.
00:28:53
Speaker
But if you are and yeah empathizing with them, if you really think about what you can do to help their job, it is helping them forecast better. And so when you start to have those conversations with them, you say, like what can I do to help you out? That's really important. Because exactly what you're talking about, these hes FinOps practices within data are really difficult. so at least helping them understand, hey, we're we're going to spend a little bit more this month because we're helping on this project that's going to dry you know that is targeted to drive revenue in this way, or that is this. and that you know You can start to say, well, how much did the data contribute versus how much did the software team contribute? and it's like
00:29:42
Speaker
if you have somebody that's trying to dig into that specific, they don't understand finance. Because quite honestly, finance is extremely squishy, right? When you when you say, hey, we need 3x pipeline coverage to to be able to meet our sales targets, there's a reason for people saying that. That's just kind of a rule of thumb.
Handling Unexpected Costs
00:30:01
Speaker
But it's like,
00:30:02
Speaker
How much did marketing generator pipeline that's always a fight right when you say how much did sales generate versus ah marketing how how how do you do marketing attribution it's always a fight but if you're sharing a flow of information about.
00:30:20
Speaker
Here's what we're seeing. you know Again, that surprise factor is is your worst enemy. If if my if my FP and&A team, the financial planning and analysis team, had just gotten the cloud bill and then had to come to me and say, why did we spend 10% more of this month?
00:30:40
Speaker
that would have been a terrible, terrible relationship instead of me telling them ahead of time, hey, we're going to be spending this much. Or you know we had one we had one month where we actually came in well below expectations because we were planning on deploying this new project. And we were planning on deploying it on like the 10th or the 12th or something like that. And we ended up deploying it on the 29th.
00:31:05
Speaker
So we spent way less, we didn't spend any of that money really in that, on that project and in that month. So if they just took their forecast and said, Oh, we actually spent 10 below this month. I'm going to keep rolling that forward. And then the next month we would have way overspent and they would have been pissed. But instead I told them, Hey,
00:31:30
Speaker
The deployment date for this got moved out and therefore keep your numbers for for next month, but this month is going to come in below expectations. And they're just that constant flow of information. yeah it's yeah it's a you know you I know you how much you love my relationship analogies, but it's like, you know you you you if you want to but build a healthy relationship in any sense,
00:31:57
Speaker
The flow of information back and forth is is important because you know human beings literally evolved to not like surprises. that is you know I know there's some people that agree or don't disagree with the lizard brain thing, but all of that. Our brains react poorly.
00:32:17
Speaker
in general to surprises, especially negative surprises. you know Some people love like getting surprised, but they don't love these types of surprises that we're talking about. and so yeah Hey, we're spending 10% more than you thought. We're spending an extra $100,000 this month that you didn't budget for. The earlier that they know that,
00:32:38
Speaker
the less bad it is. right And yeah, you can have some really awkward conversations, especially as you're figuring this stuff out. yeah I was brought in and and the finance team thought my job was to come in and to cut costs. And I did a little bit of that, but most of my job was on preventing future spend and making sure we were spending appropriately and helping them stay informed as to where we were spending money and why, and and smoothing the process for them. right you know i I did take some accounting classes when I was in college. I was a finance major, so I understand a lot of that accounting stuff.
00:33:19
Speaker
but it's not that hard. You can take a course they or a saundries and a but on this stuff and like understanding what are the decision points and you make that process much smoother for them. And you know if if you've got a really crappy team,
00:33:35
Speaker
They're going to come down on you more for more information. That's really rare. If you've got a reasonable team, they're going to see that you're putting in the effort to make their job easier. And you might say, I'm
Improving Cost Efficiency
00:33:47
Speaker
a data person. I got into data to do data work, not to do this business work. But that this is this is, again, that that they like what is concrete about this?
00:33:58
Speaker
going in and yo say you're if you're on GCP just in install masthead and it'll tell you where to cut a bunch of your costs but if you're not a lot of this is just going in and saying like trying to backtrack and figure out and and if you're entire data production process is nothing but spaghetti, where you just kind of have data comes in, a bunch of stuff happens, data comes out, and you're not really sure of how the costs allocate to any one thing, then you need to start probably decomposing that. You need to start to say like, where where are our levers, understand where that is. And that can be fun data work. That can be like, i'm I'm following how this all works, but
00:34:44
Speaker
If you're only wanting to do the data work, then you're not gonna be a data leader, right? You can be a data engineer, maybe, especially in a larger company, you can get lost in the the sea of data engineers. But if you wanna be a data leader, you have to understand, you know, Benny Benford says this a lot. I love this this quote where he says, are you a data leader or are you a business leader? Are you a data executive or are you a business executive who's in the data department?
00:35:12
Speaker
right? And that's for kind of the head of the data or the, you know, top few spots because they really have to be focused on business and how data impacts the business. But if you even just want to be a data leader, you have to understand how budgeting happens. how does this at youle It's not just that somebody says they want data work done and you go and spend whatever needs to get spend. We this um ah new project that was going to be extremely large for us, um presumably from a revenue standpoint. I i wasn't still there when when it rolled out, but from a cost standpoint, building out the infrastructure.
00:35:54
Speaker
and They were just going to deploy it without forecasting what that infrastructure spent was going to be. And I pushed back on that heavily leap and said, i will I will go up to the CFO and I will block this. um I am in the product organization. I am in the engineering organization. And I don't care if I get fired. I will block this because you can't just do this.
00:36:19
Speaker
I'm not necessarily agree with being this rough, but anyways. I just said, you you can't do this business in this way. And so one forecast that they gave me was zero dollars of incremental spent. And I said, OK, what is this going to do? it's good Well, it's going to completely trash our Elasticsearch clusters. So all of our other product interactions and this interactions will be gridlocked and we can't stop it. Yeah.
00:36:48
Speaker
OK, so no. OK, well, OK, we're just going to do this the way that he is going to make it the most performant. And it was going to increase our spend by.
00:37:00
Speaker
40 percent euro. or you like Yeah, it was going to be massive. And so we started to go in and say, OK, we were looking at DynamoDB versus like Apache Cassandra and the DynamoDB costs because they're on a per query basis or even if you allocated.
00:37:19
Speaker
extremely expensive, comparably to having these big clusters of Apache Cassandra and just say, okay, like if we do it this way, we have to have overhead of these clusters. They're kind of a pain, but our costs are reduced from, if we did DynamoDB, our costs would be 2 million a year versus if we do Apache Cassandra, it's 400,000. And we go, okay, so if we allocate one full-time engineer,
00:37:46
Speaker
to that even you know and and Cassandra especially at that point was still a big pain in the butt to to manage it's gotten better but um you know my background is around Apache Cassandra specifically but um one, you know, heavy cost engineer, maybe $250,000 fully loaded. Okay. 400,000 plus 250,000. That's 650,000 versus 2 million. Right. And as that scales up, the cost goes ever higher. And so we even ran out of trials to see like what would happen with DynamoDB. And we had really bad costs or we had really bad performance as well. And so even went like, we can't justify doing this because
00:38:28
Speaker
it's not built for the scale that we're trying to do. It can't do this many transactions per second. I think it's gotten far better than it was. But it literally can't handle what we're trying to do. Oh, wow, this is really great that we we did this performance. But we constantly went back and forth. And again, I'm i'm rambling a little bit. But my so my the point of my story is I went in there and said, like why are we making this architectural decision?
00:38:57
Speaker
and then went back. healthker and and Yeah. And then was able to tell the finance people and they were able, like the CFO was able to to be like, Hey, we don't have a forecast for this very important project. yeah And that came to the head of the the product department.
00:39:17
Speaker
And he could have been mad at me, but he wasn't like because we had a good relationship. And I said, like this is this is a career ender for half of the people in the product org if this goes wrong. So like we have to figure this out. And so we pushed out our launch by two weeks for this this thing that had been kind of a long time coming, for a total of two weeks to really make better architectural decisions. But it meant that we could scale it out far faster because we had really, really pushed through that.
00:39:47
Speaker
you you need to be asking those same questions on the data
Engineer Involvement in Cost Decisions
00:39:50
Speaker
side. and As a data engineer, you need to be thinking about, you know am I just building this pipeline for this one use case, or can I build one for multiple use cases? and like Thinking about what's going on and and why, and should I be shutting this stuff down? yeah We had multiple things where we went,
00:40:11
Speaker
yeah I found out that have we had somebody who had misarchitected a few things and we had all these read replicas for our databases that were literally never accessed. We had you know a leader and a read replica. This has never happened in data.
00:40:26
Speaker
Like, you know, never yeah just it's, it's, it's specifically the use case for software engineering. They are doing this. This is, you know, the worst practice, none of data engineer. So, you know, uh, data analyst, like but I can test properties. I mean, we, we literally had read replicas and our reads were being sent to the the leader. And so exactly I tried to get us to to shut all these down and then they went, no, we're going to rewrite our software. But like.
00:40:55
Speaker
the number of times things are still running that shouldn't be running, do you know on you how much you can go in and go like, hey, we we detected that this thing was no longer being used and therefore went and talked to the people and were able to shut it down. That is such a massive win where people are like, you are on top of what you're doing. You're focused on containing the overall cost.
00:41:20
Speaker
and not just the cost of what is running, you're focused on the business perspective and saying, does this still matter? And that you're proactively asking those questions. If anybody's watching the video, I'm using a whole lot of hand signals.
00:41:36
Speaker
but like let me let me Let me just sum up really important points that you mentioned. This is what I was saying at the beginning. and This is what makes you a great convert good engineer when you actually understand the cost of your architecture decisions. Same implies for data. When you're making the choice in favor of some solution or you know even going the other road,
00:42:01
Speaker
This is just to reinforce your point about all the duplication and and the same stuff running software. Yesterday I was on a call with a full client and we were looking through their pipelines, you know, and I was like, I look at this. So this is moving to another to the enterprise editions will reduce your cost by 50%. And she was like, do you know what? We re-varited all of those pipelines into dbt need, like it's not supposed to be here. And I'm like, oh, sorry, because it ran like two seconds ago. I can tell, I can show you the job. I can, I was like, no, no, no, it's not new run. It's not method run. It's you folks forgot to switch off the old pipelines that are still running and contributed to your compute.
00:42:50
Speaker
Avsi, you guys finished your project, how did your goal migrate into DBTU? We had these ah things that were called a site, and that was like a full software stack so that we could have R and&D testing and and we had these these easy deploys. You push a button and it would deploy it, and then they had another button that they would run only at the CLI that would tear it down.
00:43:14
Speaker
except for it only worked about 60% of the time. so I'd be asking our SREs, you know hey, I see half a site up in this, can we tear this down? and mean These things were 10,000 a month. and so I would catch it. I'd look at least once a week, but usually on a daily basis, I'd just quickly jump through the actual user interface, you know, web interface for AWS because everybody was only doing it at the CLI because we did this thing. It was supposed to have worked. Did I test that it worked and it didn't work. And so, um, you know, it's one of those things where, but you, you, I mean, you've, you've kind of talked about this as well, where you had one client that was like, Hey, we found a whole lot of cost savings. We're gonna save those.
00:44:02
Speaker
for when somebody comes in and tells us we have to cut our costs. This is my favorite. I don't recommend it. But at the same point, it is one of those things that you're going to find. And and the other thing is, if you're talking, I mean, you can play this for them, this clip for them. If you are a finance person and you're trying to say everybody has to be 100 percent efficient on their usage of cloud, you are 100 percent incorrect. And here is why.
00:44:32
Speaker
there is a time value cost to those engineering hours. We're always going to have certain things that are running that shouldn't be. You're really telling me all of the reports that you're running as a finance team are being fully used or are being you know fully leveraged. Are you really telling me that all the the reports that you're sending to all of these um you know business leaders that... nan statehold ever sam Yeah, Yeah.
00:44:59
Speaker
that they're using all that stuff? No. So you have to have that that little bit of cushion, but you're going to find as an engineer. Yo, I went in when I was cleaning up um some of the stuff in our AWS. I cleaned it up so that it so that it was cleaned up and so that I showed everybody best practices. But I was going and manually shutting down DynamoDB tables that were costing us a dollar a month.
00:45:24
Speaker
So it took me about two minutes to do and so like recouping that cost was like, you know, or whatever. I i worked out the the math and there were multiple ones that I was shutting down that the recoup period was two years.
00:45:43
Speaker
But I did it so that it was it was shutting all of them down and just made very, very clean, so anybody who looked at it. But I also had somebody that came to me and said, hey, I'm going to be running this thing. And you know I'm going to run it on an M4X large back when M4, I think M6s are now the thing. But anyway, M4X large, but i can with two weeks of work, I can i could bring it down to an M4 large.
00:46:07
Speaker
I was like, thank you for coming to me. Let's talk about this a little bit. OK, how many of these are you going to run? He said, one. And I was like, OK. So the cost for an M4 x-large, if you're running with reserved instances, is about like $3,000 a year, I think it was. So cutting that in half is $1,500, right? If I go about even a cheap engineer,
00:46:32
Speaker
um that is ah at $100 fully loaded cost per hour, which you're not. You're a principal engineer, so your cost is significantly higher. yeah That takes three years before I recoup the costs, right? If I save $1,500 a year, that's going to take me, oh, that'd be three years per week. So it's going to take me six years to recoup the cost of you working for that two hours. Thank you so much for coming to me. I'm going to to put in a good note about you to a few other people because I sincerely appreciate that you were thinking about this, but we've already spent more time than we should have because it's just not worth the effort. But thank you for thinking about this because I need you to be talking to others when you're talking to others about thinking about their architectural decisions.
00:47:25
Speaker
But there is no way that's worth it. Even though it would have lowered my cloud bill, it would have been bad for the company. And and having those discussions with the finance people about that back and forth, the most perfectly optimized um pipeline, and it takes you yeah know six weeks to develop this pipeline, every single pipeline. How many freaking data engineers do you need? And if there's any changes, you know you have to do another three weeks of optimization on those changes.
00:47:55
Speaker
Yeah, the cause benefit and the value that you're going to bring in over and and the efforts you're going to put into it. So just, you know, we almost out of time, the thing that I wanted to touch base is really, again, we were talking about um software phenopes a lot. and As we stated, the um data phenopes practices and are not there. Here's the interesting thing, you know, like,
00:48:20
Speaker
Again, as I mentioned, it could be 30% of companies check, it could be 20%. I mean, the whole data infrastructure and data team alone. So here's what's happening. You mentioned this, that sales team and marketing team are sometimes getting into arguments, you know.
00:48:39
Speaker
who is doing what, okay? But when it comes to Data Team, there is a clear spend behind Data Stack. There is a clear spend on Data Team, but not necessarily a clear revenue from Data Team because Data Team serves horizontally to every department in the organization.
00:49:01
Speaker
Okay. And the chunk of their data infrastructure spans, if you think about this one team, it's actually big. So overall, the data team spans, uh, proportionally to other teams within the company is quite big, if you think of it. Yeah. Yeah. So what, ah what I'm heading into. So data phenoms, um,
00:49:26
Speaker
need to need to appear. One of the reasons for that is because of the, when it comes to letting go of people, for some reason there were lesser data people let go. And I think part of that is not declared justification of data team spends. And I think that's part of Colossal Management, right? Like that's, if we're talking about what are clear recommendations for people. what What can they take from this? One is to go out and actually figure out um where you're spending and and and then start to ask why and then start to ask, is it worth it? um One thing that I i
00:50:09
Speaker
I talk about is the data field of dreams and and justification of of data. that you know There's this movie, Field of Dreams, and and it's this guy. and you it's a it's I don't really recommend it. I've watched like 10
Evaluating Data Work's ROI
00:50:22
Speaker
minutes of it. I'm supposed to have watched it because I'm from Iowa and it's set in Iowa. and so It's like one of the like five movies that are set in Iowa.
00:50:29
Speaker
Is it about relationships? No, it's it's about baseball. But it actually is all about relationships. Anyway, these ghosts are are whispering to him, if you build it, they will come. And if you if you build this baseball stadium, people will come and watch baseball. And it's it's it's a very weird. Anyway, it's an 80s movie with Kevin Cost.
00:50:49
Speaker
but In data, there's this idea that if you do data work, value will come because there is value in data automatically. and and i would I would actually say yes. I would say there is no matter what value in data, but a lot i heck of a lot of the time it does not justify the cost.
00:51:10
Speaker
and so get out of your head that data work will have inherent value because there there is a return but not necessarily a return on investment. And so you have to think about, you know you can't do everything, right? So yeah you can't be friends with everybody. um no How are you spending your time to build relationships that matter to you so that you have the friends that you want?
00:51:39
Speaker
And like, how, how are you doing those things? How are you thinking about is what we talking about here? Like we suggest in data, people just making a decent human relationship with our colleagues. It's to stop only focusing on the data work. The data work is important, but it's, it's kind of like, um, what Daniil said on, on our, our last report of like, do you want to be somebody that's just doing the data work?
00:52:05
Speaker
you're You're at risk of being the ones that are like, oh, right? Your your entire department is like all of that. If you're just doing the data work, it can be okay for your career because other people are hiring for people that just do data work. But the second you start to get a business focused leader, those are the ones that change people's careers, change people's perspectives on things and make you a business leader instead of just a data leader.
00:52:31
Speaker
And if you even wanted to graduate to a data leader, you can't just be doing the data work. You have to be thinking about how does this integrate into what we do as a company and where we should be focusing. And so like when you think about cost management, we haven't talked even much about cost cutting or cost prevention, but it is those like, why are we doing this? Could we be doing this better? Could we be doing this in a more cost effective way? But also,
00:52:57
Speaker
You don't have to optimize to such a degree where you get to perfect. You have to optimize based on multiple factors, including your own time. Your time is worth money. you know Unless you're really, really bad at your job, your time is worth money to your company.
00:53:15
Speaker
like more concretely is to dig into where are you spending, why, and start to communicate with people about, should we be doing this? like Not, this is interesting work. It's, this should we be doing this, and how much should we be spending on it? and and Should we continue doing this? And interrupting you here it doesn't mean necessarily do you need to monitor the cost of a job or who is running the most expensive job. like it It's not the point. The point is to have understanding.
00:53:52
Speaker
like Where does it bring value? and and who uses it and and how do you make sure it brings value and being used it as well. This is a part of its job, like not just ship it and wait, you know, if it's like, not just ship it, like, so this is not, this, i think I think the problem is that people sometimes think that when they move the ticket in JIRA, from in progress to done, the task is done, but it's not necessarily done, especially when you ship like a,
00:54:28
Speaker
um you know, maybe dashboard, maybe dataset that was requested. It's still continuous work where you need to understand if it's still used, there are any best practices should be used to optimize the cost and so on and so forth. I don't know. I guess it's also been going into details, but I'm trying to put it more tangibly. So that makes sense. Project thinking versus product thinking, right? If you're thinking about just doing, getting the work done instead of why are we doing the work?
00:55:02
Speaker
and And you might be in a company that really, really values that project thinking. And you can start to add value by moving out of that. But you can also get in trouble. There are certain organizations that are just backwards thinking. And and I agree that yeah you should yeah you should start looking for a new job. It's not easy that you could just, especially in this environment, that you could just find one. But if you want to become somebody that is going to significantly change and feel more satisfaction in their job, unless you only get satisfaction from just doing the data work, which is is fine if you want to do
Aligning Data with Business Goals
00:55:39
Speaker
that. Their organization, I do think they're going to go away more and more, but if you don't, if you're not feeling satisfaction in just checking the ticket done,
00:55:49
Speaker
look for those organizations that think in that productized way because it's not just about my job is to move the JIRA ticket forward. My job is to complete what we're trying to accomplish and the JIRA ticket is just a way of measuring what we're trying to accomplish and moving that forward and saying, okay, we have moved forward on what we're trying to accomplish and keeping that that kind of ongoing mentality Rather than, you know, i I had this when I was at Datastax, I was helping manage their startup program and I told them specifically, my job is to maximize signups. That is what I'm doing, because then we will have additional users. Nobody else really paid attention to the metrics I was using. When I was saying signups, signups, these are the number of people that have completed the contract, not have installed the software, and that caused a bunch of issues.
00:56:44
Speaker
because I wasn't communicating clearly enough and they also just weren't paying any attention when I'm saying this. But like again, thinking about that, like What are you trying to accomplish? Are you trying to move the ticket or are you trying to solve the business problem? Are you trying to make something better for your customers? Are you trying to cut costs? Are you trying to expand revenue? like what What is the point of this work and and why is it a priority and and what's going to make this last so I don't get another ticket on it next week?
00:57:18
Speaker
Also, what is important in mind, it also, because of this work, might be not the success in the world. It could be boring. I find it so interesting because I love talking to people about this stuff. I i love talking to them about what they care about. ya Yeah, okay. We were about to wrap up here, so we don't get started on that, okay? But what I'm trying to say is that Generating value is much more important than playing with new technologies or whatever. You need to make sure that there is a value and there is optimal decision for your ah time that you deploy into this task and the solution used, like you have to be intelligently honest about it. For God's sake, you know, not inventing something to to have time to play with it. Yeah.
00:58:10
Speaker
i I have one wrap up thought on kind of what we've been talking about. And I think a lot of it is go out there and understand more about what your company is doing and why. And and and always be asking a little bit, digging into that why without being obnoxious about it. But like you for most people, I can't say for everybody, for most people, you'll get more satisfaction in the long run. You'll find it you'll find your job more engaging Rather than it's fun to play with the tools, it's it's fun to interact with with the business because you're going to learn so much and you're going to be so much more valuable and more more kind of i've been complete as a human being because you're going to understand
00:58:54
Speaker
how to communicate with other people about other things. I have a friend that gets really upset at me because I always say, like what are we trying to accomplish? She's like, no, what is the end goal? And I'm like, product thinking versus project thinking. Not just what is the end goal, but like what are we actually trying to have happen? Now, what are we trying to complete? But what do we want to have happen? And then, again, coming back to this on cost management,
00:59:18
Speaker
asking if that's still happening.
Gaining Recognition through Cost Management
00:59:21
Speaker
Like keeping in touch with people and understanding what you're doing and why will mean that you find these great areas to contain costs, to manage costs, prevent costs, to do all these things because you've asked the questions and you understand how to ask those questions. And then, you know, that you're constantly kind of you also The one thing I didn't do that well, but was seen, but a lot of people in data think value is inherent and is inherently seen. If you're creating value through your cost management, no you got to be talking about it. you got be You got to be putting together your metrics. You got to be telling people, yeah am I doing this? and And that will give you the visibility because data people are often unsung heroes and because data people think the data just shows that the the value I'm doing more and
01:00:13
Speaker
people are in their own world, right? Like people aren't exactly like that. You know, we can do another relationship analogy. If you went and you, you did a really deep clean of the bathroom today, would you think anybody would really notice? I mean, depends on how clean your bathroom already is, but you know what I mean? Like if, if, or do you be like, yeah, I don't, I mean, we're going to my eyes and not right now. Just the question. Yeah. No, not yet. No. Okay. You have to go all the way to the back of your head. But like,
01:00:41
Speaker
It is that communicating things to each other and be like, I did this thing for you. Not give me credit, but like I want you to stay informed that I'm also on your side. I'm doing things for you. And then you have that kind of back and forth communication. Those people are going to be on your side too.
01:00:57
Speaker
so it's it's Again, we're we're laughing about how in concrete some of this stuff is, but a lot of it is just like going out there, finding where these things are. It's not about the Finopsy side as much as it is about understanding, and then you can dig into the Finops. If you try and be like, this cost per query is this, and it's like, yeah, but again,
01:01:19
Speaker
that we optimize this pipeline. I i significantly decreased that pipeline. That pipeline shouldn't have been running. We moved to everything to dbt six months ago. So rate that you optimize it, you saved us 10% on it, but it should be, you saved us 100% because we should have shut it down.
01:01:36
Speaker
Yeah. Okay, folks, thank you so much for staying with us this long and listening to your all the wisdom from, uh, scored about the relationships and hear us, um, in two weeks, like.