Zencastr
00:00:00
00:00:01
Speed1x
Format▸
Share
Embed
Report

Is Your Backup Infrastructure Really Secure? Veeam Patching, Tier Zero & Supply Chain Attacks

The Rebel Devs
The Rebel Devs

8 plays · Nov 10, 2026

Transcript

Speaker: Rebel Devs podcast, a show that challenges conventional thinking in the tech industry. Each episode features conversations with solutionists who have taken unconventional approaches to problem solving, pushing the boundaries of what's possible in their respective fields.

Speaker: Join the devolution.

Speaker: Hi, everyone, and welcome to another episode of the Rebel Devs podcast. I'm your host, Jonah May, and I've got Tom here with me today. How are you doing, Tom? I'm doing great. It's great to be here, and I'm excited to talk about some current events with you.

Speaker: I am as well, and I think we should probably start with the one that pays the bills first. Get that out of the way so we can get into the fun. So did you see, what was it, I think, as of recording two days ago now, something somewhere in here this week, Beam released another security patch.

Speaker: Oh, yeah. Well, I did actually see that. And that's always an exciting time. I'm so excited to have more security and I'm so plagued by the how do I patch it sort of thing. So love hate.

Speaker: Well, hey, good news this time. It's only for version 12. If you're on the latest version of 13, you're already patched. So I think that's 1303 if you're not yet or the latest patch.

Speaker: both arty received fixes Well, that's good news. Most of our customers are on 13, so that's good news. We have a couple holdovers, but this isn't a bad patch to install. It's just, as you know, Enterprise Manager, Veeam 1, what components does it cover? you know That's typically what you're looking at and doing it in the right sequencing. Something also to keep our listeners just aware of, remember if you're running Veeam Recovery Orchestrator, There's the VBR that's oh over there. Beware of your patching cycle. So don't just go apply it or you're going to spend a lot of time unapplying things sometimes. Don't know if it applies to this one right now, but just keep that in mind. A lot of folks forget and they think VBR patches VBR. But remember with Veeam Recovery Orchestrator, there's also a VBR sitting over there. So I don't know. What do you have to say about this, Jonah?

Speaker: Uh, yeah. You know, luckily for me, in my case, our company upgraded 13.1 earlier this month. So we're already patched. Yay. I checked the builds. we We do have a mix of customers, both still on 12 and 13. So we are going to have to reach out to those customers on 12. Regardless, they're going to have to upgrade here soon.

Speaker: I'd say it's pretty even split so far. A lot of customers have held off on 13 because just there's so many changes under the hood with it. Although 13.1 has improved a lot on the initial 13 release. But for those of you listening who aren't aware, version 12 reaches end of support at the end of February. So you've only got about, what is that?

Speaker: Four more months, I think, because we're in early October now. I'm not doing math on that, but up we'll go with it. The end of February. So March, get ready. You're going to make the move. There's plenty of features that you want to. Something I think the listeners should be aware of. You'll notice where I'm speaking about my customers, Jonah's talking about his customers. So I work for a company called Different Dev. We do a lot of professional services. We're on hand with our customers. We do maintenance and monitoring. So we're actually on the customer side of the environment. and help land it internally, other locations, maybe off to a cloud provider. And then Jonah, you live in cloud provider world. Tell us a little bit about that as to why we're talking differently.

Speaker: Yeah, so we have so so I work at CyberFortress. We have historically been a Beam cloud and service provider. So we have existed since the days before these things like AWS S3 and Wasabi and Backblaze existed. And since probably 2016, even earlier than that, have been around to receive your offsite backups, receive your offsite replicas and offer you a way to comply with the 3-2-1 rule and recover your workloads, whether they be back to your production environment, to a temporary location in our data center or elsewhere. And then i'd be remiss if I didn't mention, we do have a little bit of maintenance and monitoring too, mainly on the monitoring side. And if you're needing cybersecurity, where your data resilience experts, I think,

Speaker: is the new tagline or something along that line. So we've got great folks like me at Cyber Fortress who have been in the BCSP space. We have people who have a lot of experience with cybersecurity. So we help you hit your three pillars of data resilience, prevent, detect, and recover through your backups, your recovery testing, and your security and monitoring like your SOC, SIEM, and MDR solutions.

Speaker: No, that's good to know. And basically what I wanted our listeners to realize is we kind of live in different areas. So when you're celebrating, you have upgraded to whatever, you're talking about your cloud infrastructure. You're not talking about the customer side. On my side, I really don't have a cloud infrastructure. I use different vendors out there. So for me, I'm talking about the endpoint side when we say this. So that was kind of the main differentiator there that I wanted folks to realize. So definitely just to let you know if you are using a cloud provider, sometimes Veeam patch will come out. Don't ever just apply that without talking to your provider first. Sometimes they're minor versions, sometimes they're major. And really your cloud provider needs to be ahead of you or equal to you. Don't ever jump ahead of them or bad things will happen. When in doubt, if you have a service provider, definitely take the time and go ahead to speak to them before we go. But enough of talking about that. So I don't know, Jonah, what do we want to think about? Since we're talking about security, it sounds like it's a good time to be able to talk a little bit about some Veeam security. Wouldn't you agree? I mean, we're talking about a patch. Maybe we just... I was going to say, we have some discussion questions, you know, basically along the lines of should

Speaker: Veeam be considered a tier zero infrastructure, mission critical, essential, like your domain controller? And if so, how do you expect, and I'm using you more so to refer to you and me, you know how should someone like us or a service provider handle emergency patching, especially when they could be managing hundreds or thousands of customer environments and Cloud Connect servers?

Speaker: Because I will say for any service providers out there listening real quick, this patch that we're talking about specifically, it's very rare, does have a vulnerability fixed for Cloud Connect.

Speaker: So well if you're not on 13 and you're a Cloud Connect provider, you do want this patch. Well, think about it this way. Why don't we frame things like this? Since most of my customer base is on-prem, most of your customer base you're catching in the cloud, why don't you speak to the cloud side on these? And I'll speak to the you know customer side, if you will. yeah I think you were talking about the rating of the backup infrastructure. you know How mission critical do we think about this?

Speaker: I would say tier zero. It is the most important thing in the environment. So when we talk about patching with it, I like to think of it as tier zero, but I don't know that I want to be an early adopter to something like this sometimes because these are running my backups. Where I look to the customer is if they're writing somewhere else, I can be more aggressive with my patch.

Speaker: So if it drops on a Tuesday, I can do it on a Tuesday or a Wednesday, depending on the vulnerability. If I am one of those people that say, hey, I'm in a community edition or hey, I have a licensed edition, but I'm not writing it anywhere except my infrastructure.

Speaker: I'm going to hold back and make sure that as it's widely adopted, a patch of version, say,.1 doesn't quickly become version.2 because there's some sort of bug in there. I'm going to be a little more conservative on it. What's your take on that from the cloud provider side?

Speaker: ah My take is it's a little, it's prioritization and it's case to case. Like in the case of being a cloud provider, Nine times out of 10, this being one of the few exceptions, the vulnerabilities fixed tend to not really affect us on the Cloud Connect side. A lot of them are like, hey, and a local unauthenticated user can do X, Y, Z. Well, our customers are storing their backups encrypted with us. We don't have those encryption keys, so those things can't happen. Or, hey, if your domain can join, this can happen, and we're not really domain joined.

Speaker: Again, this is one of the rare cases where it would affect Cloud Connect. So if we weren't already patched for it, there would be a little more priority. and But yeah, in our case, we typically like to let it bake a little bit, maybe you know see if anything happens for a little bit and try to include it in our monthly patch cycles that we already do for other components like Windows, you know our other security softwares, our other backup products.

Speaker: And I'd say it the same for the handful of customers that we do manage as well. you know Typically, like you said, if it's their only copy, we'll be a little more leery of it. On the other hand, if it's something we know directly impacts them because of how they're configured or how their infrastructure is set up, we might be a little faster to get their systems patched and Historically, that has been a pain in the rear end.

Speaker: However, recent improvements to things like Veeam service provider have gone a long way in helping with that, wherein we can now much more easily push out updates for customers who at least check that management box and allow us the remote access to push out patches, not just for VBR, but for some of the other products too now. Like I think Veeam 1 was added recently and 365 might be in the works.

Speaker: I know one of the challenges when I was in the cloud world, and I don't think it exists anymore, was we always had to watch the versioning that the customer was on. And I think nowadays with the backward compatibilities, it's a little safer on the cloud side to go ahead and update these because the customers will in fact be compliant. But I think you can agree with me, there's one critical time that you have to watch applying these patches as a cloud provider. and I do remember having to monitor the servers and say, listen, we need to make sure that we are watching inbound rights from the field and disabling the tenants to allow them to drain off, patch it, bring it up and re-enable. Is that still the case on your side or do you find that the reconnects seem to be okay?

Speaker: if The reconnects. They seem to be a lot better. Thema's definitely got better. you know We haven't seen things like we saw at our last provider, right? When we worked together where, you know hey, we decided to reboot and this REFS file was in the middle of generating and doing a merge and oh no, now it's been detected as corrupted and it deleted the entire VBK.

Speaker: yeah you know we haven't We haven't seen that. That being said, we still do kind of like we used to do at our last provider. I see that our patches tend to be during business hours because counterintuitively, most people write their offsite backups overnight after their business hours. So they're not using their bandwidth and they're getting their data at the end of the day, which means the best time for us to patch and do maintenance is during business hours where you know a lot of companies will wait till after business hours for impacting changes.

Speaker: Well, you know, you threw a loaded question in there. You talked about domain joined and, um you know, I will say that the first thing I do when I meet with a customer, that backup server is out of the domain all day long.

Speaker: Absolutely. Anything related to the backup is out of the domain. In rare cases for very large environments, we've actually created a secondary domain which is kind of like a DR domain. And we've put all the components in there. So if there's multiple beam servers, it makes it a little bit more transparent.

Speaker: And this is where somebody asks the question, right? And they say, wait a minute, this latest vulnerability said, if you have a domain join system, you have to put this patch on. And I'll get this question from customers all the time. Wait a minute, Tom, it's not in my production domain, but it's in my DR domain. What does this mean?

Speaker: Ah, loaded question, right? Do we need to patch? So the technical answer is yes, you should patch it. Of course, there's a vulnerability, but let's talk about the vector. When you're in a DR domain, the concept is I'm in an isolated domain.

Speaker: When I design them, I'm not doing a trust model. So there's no one-way trust or two-way trust in place. It's an isolated domain. We're typically not allowing internet and other activity to happen on these boxes. So we are really locked down to where you log in. It's one, you know, two factor into the Veeam server. It's coming into Veeam. It's two factor with Veeam. It might even be four eyes. It's really locked down and specific. So the threat matrix is lower than somebody that comes to us and says, okay, this is domain joined in my domain.

Speaker: there's impersonator problems, I have to put the patch. That is an absolute mission critical patch at that point, but it's so mission critical that after you patch it, it's time to get those machines off the domain.

Speaker: They need to be standalone or out of the production domain all day long. Jonah, what's your thoughts? I was gonna actually take it the opposite direction, believe it or not. So one of my gripes, Veeam has been better about this in recent years, which I'll get into more in a second is I feel like the Veeam server itself gets granted too many permissions, even if it's one way, because yes, you can have a separate domain or its own local work group, but what happens when someone reuses a password or something, right? Same username, same password, even though it's separate, you know, you live in a world where if the Veeam server gets compromised, I mean, we've seen this firsthand with some partners like cloud IVR, right? Like, especially

Speaker: in the days before we had the Linux appliance where things are a little more locked down. A bad actor would get into the Veeam server. They can get in separately from Veeam itself into the Postgres database or the Microsoft SQL database, and they can figure out ways to decrypt and exfiltrate credentials. One of those being S3 keys where they can then go mess with versioning and stuff. But on the flip side, and I don't know if this is the case, but Hypothetically, if they can exfiltrate S3 keys, there's nothing stopping them from exfiltrating your domain admin for your VSS backup, your application aware processing, or your VMware administrator or root user that you have to have to connect to ESXi or vCenter.

Speaker: But one of the things i enjoyed seeing, and I hope maybe it goes this way more and more as Veeam evolves, is when Proxmox support released, they said, hey, you can add the Proxmox host as a root user or alternatively,

Speaker: Here's a set of commands you can run to give a user least privilege access to the Proxmox cluster in order to still do backups and restores. Okay, but we're talking about domain credentials getting into the Veeam server, right?

Speaker: If we're locking that server down, it's off domain, If someone's not using the same credentials, which they shouldn't, if it's got two factor authentication, so you're gonna have to have Duo or something to get into it, then when you get into it, now you can start doing SQL. And if Postgres is on the same boxes as the Veeam server, as long as those ports are locked down to local host only, you're not getting remote calls to it.

Speaker: Is there really vulnerability at that point? And I think that's why they're saying it's so important to be off domain. Right. I mean, yeah I'm sure there's ways into it, right? There could be some vulnerability within Beam itself through a REST API or a Windows zero day, right? I mean, I have seen in some cases, albeit rarely, where something like Duo has been bypassed.

Speaker: Absolutely. But those are low lower attack vectors. So this is why get it off the domain, do a Duo on it, That's gonna lock the machine down. Only have your backup administrators right with access. they I mean, they shouldn't even really be using the server anymore. They should be using the remote console in all purposes, but they can come in, they can use the server,

Speaker: They're not going to get on the Internet. They're not going to browse. They're not going to chat GPT. You're going to lock that right on down to where it's only accessing the infrastructure and the right points, if you will. So lockdown, I say it's off.

Speaker: It's not a domain admin. It's locked. It's safe. and And to the point of these patches, I'm making this number up. But I think about 90 percent or more. are typically driven for that. This is a concern if your domain joined. So just don't domain join. It takes the threat matrix down. Now your patch isn't really a security patch because it doesn't apply to you. It's really more of a bug fix, if you will, at that point. so You know, off domain, we're done. Yes, there's vectors, Joan. If you get into the, I mean, if if you ever get- Well, that's why you have security through layers, right? Because- Right. And why you why regardless of whatever you put in front of the Beam server, you still want immutability as your last line of defense. Right, i just if we're gonna talk about vulnerabilities in the database and being you know exfiltration and whatnot, I think that's a different conversation. That's a, hey, I got in. And and any threat level goes up when somebody is in, right? And insider threats are very real. What we're trying to do is Joni sits at a keyboard, Joni's browsing the internet, Joni puts her credentials in somewhere, doesn't realize it. Somebody comes in, uses Joni's credentials, finds the Veeam server, gotcha.

Speaker: Well, if it's on a different domain, as long, at first they're gonna have to find the domain, which maybe they can. Then they're gonna have to find the Veeam server, maybe they can, right? Now they have to hope that Joni used the same username.

Speaker: Well, that's controlled from an administrative standpoint, right? And then is it the same password? So really you've knocked out a lot of the things versus in domain, Joni put out her password, Joni got in, they destroyed everything as Joni.

Speaker: So that's the big differentiator there. So yeah, and I guess you do have the added layer too, if you're using like an object first appliance or some of the community projects out there, like, are they even hitting your real Veeam server? Cause there are Veeam server honeypot projects out there now.

Speaker: Right, absolutely. And I mean, and then if they even do get in there, that's why you have immutability because they can't delete things and da-da-da-da-da. So, all right. I like this concept and I'm going to direct it at you is how much customer access is too much for a service provider?

Speaker: It's a great question. And I think it really varies based on the specific customer. So, I mean, so we've had discussions as we get more into managed services at my company of,

Speaker: not being necessarily all or nothing like, hey, you give us no access or you give us full access. We're almost trying to work on like a bet good, better, best approach, right? Like three tiers of managed service of what expectations are on us versus the customer, like a shared respect like a shared responsibility model that you see with the cloud providers, but a little bit of compromise where Maybe there's a company that wants that advisor, but doesn't want us in there all the time, as opposed to maybe it's the mom and pop shop or the MSP servicing them that doesn't have a backup expert on staff who essentially wants us to be fractional backup experts or however you want to phrase it. Right.

Speaker: Give us administrator credentials. Let us, you know, hey, VMware had a snapshot issue. We'll go in and we'll proactively fix the snapshot issue so your backups start working again. now Whereas some customer yeah some customers, and this is unfortunately a fault with Veeam. I know it is in their feature requests as service providers have asked for it. Unfortunately, while we've defined that, it's very much all or nothing with Veeam I've come to find, you know. Once you check that management box in Veeam, the service provider can see and manage just about everything through your Veeam server, especially now that Veeam is building in some of the remote REST API proxying and stuff. There's no real way to do like RBAC or even like read-only access between your Veeam server and your service provider.

Speaker: It's very black and white still. I think one of the challenges is so different dev, we we do this, except we kind we also expand into more of the, um I'll call it executive function. So when different devs in there, we're,

Speaker: basically part of the change advisory board. We're meeting with the backup team or the server team or whatever team that is, even if we're functioning as the backup team to say, if we make this change, what else ripples in the environment? So when there's a break fix going on, we're not necessarily just going, hey, we're going to go in and do something where, hey, customer, raise this issue. Do you give approval for doing X, Y, Z? Kind of from a change management perspective, because it is a shared responsibility model. But ultimately, you have to be cognizant of if I make a change, like i'm like you said, go in and delete a snapshot.

Speaker: Man, that's pretty cowboy to do without letting somebody know what you're doing because you could crash out their environment. Why is the snapshot there? Is there a root cause analysis? There's so many things that ripple. So yeah, if I go to a generic cloud provider, and I'm not saying cyber fortress, and they're just given keys to the kingdom.

Speaker: Do you know what's going on? Do you have an SOW? And beyond that, you have to think about just because you have an agreement with somebody, what prevents them from overstepping those bounds? What controls are in place? Insert in.

Speaker: for eyes protection to Veeam. You know, that's why that can be so critical. So many times when I deal with a customer, we make a change and it doesn't happen because we have to go tell the other person say, hey, check your email. It sent you an alert. Hey, go approve this change. So if we stay within the confines of Veeam, I think is important to the discussion because snapshot hunting is really outside the scope of a lot of providers. Even I wouldn't put my team on that. I'd be like, well, you're VMware a guy, I can advise you, but you must do this thing.

Speaker: That's fair. Yeah. And we've we've had discussions around it, too, even just from our side of like, hey, how is this going to work? hypothetically, if we were doing a managed Veeam data cloud offering like we have, and we were going to go in and help with that, how do we balance the need for auditing and license costs? Like, are we really gonna ask them to give every support engineer of like a licensed Microsoft account on their tenant? Or are we gonna come up with some kind of shared administrator? And then how do we save that and audit it on our side? So we know every time someone accesses those credentials, you know, who did it and when.

Speaker: Well, that's where I think you literally do need For best practices, it has to be a ah Windows account or some sort of account that has logging on the machine. And then you're doing the same logging to make sure, hey, who did what, where, what changed where, and you have that background control to pull it. But right, that's only ah forensic at that point. Somebody did something, let me pull a log. What prevents it from doing it sort of thing. So yeah, you're definitely talking about shared responsibility. You're getting into your SLAs and really at this point, sometimes OLAs between vendors. to see who can do what and in our environment. It's, you know, when we get into a customer environment, we're coming in usually through their secure methods. Sometimes we do come in through the service, ah you know, through a cloud connect port, but we're always using a unique account to come in per person to know what's going on with auditing. So that's my recommendation. And just remember, prevention is the easiest cure. It's not necessarily the aftermath and root cause. So just think about that. I think it's good food for thought. um

Speaker: Jonah, I think you were talking something about vulnerabilities or something you you wanted to touch upon. I guess trying to stump me on this one. I mean, speaking of patching, we've been talking about patching quickly and bringing it into change control boards and different things. There is evidently some, a thing as patching too quickly, especially with package managers. If you maintain your own code basis, I feel like every other day we're seeing a supply chain attack to different package managers where yes, there are cases where AI or someone tries to hack you specifically, but I feel like everyone's trying to cast a wider net these days.

Speaker: you know, go in and hack an upstream package that a lot of people use. So now if they patch immediately, you know, 15 million systems are compromised instead of one company. So a lot of it falls on open source.

Speaker: So i I want to ask you something, Tom, because I think we're going to have interesting takes on both sides of this. At what point does open source become random code from the internet, in your opinion?

Speaker: This is a circle of life that I've lived in my whole career. At one point, you don't have the budget for something or somebody does a cool project and you're like, this is the best thing ever.

Speaker: Right. And at that point. I think it's random code from the Internet, man, like it does, but I think it starts there and then it becomes open source.

Speaker: And then I think it basically becomes an attack vector. So when you first start writing code, you're subject to the person that wrote it, right? what it like post Let's use Postgres, for example. and And this is a very stable platform, so I'm not discounting them. But I'm sure when they birthed, maybe it was a little more cowboy, right? At some point it becomes stable and we're like, open source is amazing. This is great. They have some good methodologies in there. They're working, they're doing great. But now they've grown up enough to where somebody looks at them and goes, wait a minute, let's look at an attack vector by using them. So I'm going to go with open source is random code to mature code to attack vector. What what do you think?

Speaker: I think that's fair. I mean, I was going to say there's always the argument that some people make that open so source is more secure because it's had more time to be reviewed by people and for issues to be found.

Speaker: But at the same time, I think because it's a dependency, it becomes a vulnerability. And it can be very hard to separate the quality open source from the not quality, for lack of a better term, especially these days. You know, there's always been bad code on the Internet.

Speaker: And there's always been good code, right? Like look at the Linux kernel. The Linux kernel is an open source project that enterprises adopt and you know run their own feature sets on and sell. But these days you don't just have bad code out there that someone may have wrote, maybe maintained it may not. You have AI code. And in some cases there's developers. you know i am a developer. I've wrote code for years. I use AI.

Speaker: But there's nothing stopping someone who's not familiar with development processes and security practices from going and using ai and introducing vulnerabilities into that code. No, that's true. And so, you know, that's one of the things of change control all day long. What's being installed, where, what inventory and patch management, right? What are we doing to patch what's out there? So um I think that's a little bit of why WordPress went a little wild west too back in the day, because if this was great. You had a baseline system. You want to add a feature. You get a plugin. You install it. Life is good, right?

Speaker: How good is that code? So all these plugins you're putting in place, where's the quality control on it? And I think that's why it went crazy. And now they've fixed some of those vulnerabilities. But for a while, man, I saw entire sites getting nuked from something like that. So, yeah, I don't know, Jonah. I mean, if you were hiring a developer, I mean, how would you handle kind of like the NPM packages? Like, would would you just let them put anything on a system? What would you do?

Speaker: I mean, i would want reviews at bare minimum. I mean, you need to know, is it the best package you're using just not even from a security perspective, but from a performance perspective, especially talking NPM specifically. I mean, that's your front end. Just that can get out of hand very quickly with inefficient code and make your page size 10 times larger than it needs to be and slow you down. But there's licensing you have to factor in there like Especially, you know, in a service provider commercial side, reselling services to other businesses. Are we adopting something with a specific license that could get us in trouble legally?

Speaker: And then, you know, it it it needs to be a mix of fast and slow. You know, sometimes you have to go slow to go fast, as I think you've said before, like You don't necessarily just want to sit on a four-year-old build of something because obviously there's going to be vulnerabilities, but at the same time, you don't want to adopt something the day it drops. You know you want to... you know Patching, you know you don't necessarily update your Windows server on a patch Tuesday. You might deploy it to a couple test servers a couple days later. You might let it run for like a week, and then you might upgrade unless there's a critical vulnerability.

Speaker: It should be really the same for your dependencies and packages, not just your software or your OS. I think anybody that's done software development has a love-hate relationship with patches.

Speaker: They're out there. You patch because you know you need to be secure, but you hate patching because it's going to break everything. you know It's definitely a love-hate situation. So I know that I've been on some older softwares in my career and it was like, man, I don't want to do this update because it's going to be brutal if it doesn't go well.

Speaker: Well, shout out to Veeam. What can we do now? We can do some data labs. We can go ahead and... you know bring some things up containerized, we can go ahead and apply patch and see if it works and we can push to prod. So, you know, if you're not using that feature of Veeam, I highly recommend it. If you're not using it, well, make sure you're backed up and ready to go. So if it goes sideways, you can undo the patch. So I love that we talk patching, especially adjacent to Veeam.

Speaker: And it's not even just a sysadmin. I feel like I hold my breath every time we deploy a code release for the customer portal I help with because it's like, great. I fixed 10 bugs, but how many more am I going to add?

Speaker: Yeah, no, definitely. It's kind of like, where does the sun touch, as they say? you know You just don't know till you do. But that's where you get more mature in your software development cycles and change management and teams and documentation and, you know, prod tests to wherever. I know one of the biggest challenges in my career was especially with software delivery was back in the day, is as a sysadmin, you'd have a server that's running a prod system.

Speaker: And then you will look to put out a new change or whatnot. So what would happen? You you' build out a new server. You would install this application. You do say a database backup ah and then you do a restore and look, it's running. It looks the same. And then you do all these patches, changes, fixtures, everything, document everything. And you're like, look, it passed testing. Go to prod, try to put all those changes in place.

Speaker: Did it really work? Well, you don't have the same server. It's totally a different install. Why isn't it working? Your demo account is backing up five users while you have customers backing up 4,000 users. Well, it's not even that. It's not even that. It's just maybe I was on Windows 2022 and I installed Windows 2022 today. Well, my today is different than the previous one that has 15 different layered patches on it, right? Does it have everything? So that's where I really like the idea of like, hey, let me go ahead and instant recover a server, maybe shut off the networking or I'll put it in a lab, right? And I'll have a little NAT appliance there with Veeam. And I'm like, this is actually the real prod server.

Speaker: Let me go ahead and put all the testing and fixes and do everything. Look, look, look, it's working great. I've at least now done an exact clone of what I'm working on. Now, if I commit it to to prod, I'm worried about data change, right? Did something go in place while I was in this containerized area?

Speaker: Okay, that could be a thing. And maybe it's just getting the app people to do a, you know, database restore on this version or, hey, did I document everything right? And I know that it's a mirror of where I was at. So now I can click these five things instead of just coming out new. So that's kind of where I i think it helps in those regards. But um I don't know, you wanted to talk something about lock files. So you're probably punching above my wait class here, but what were you talking about with lock lock files before we started? I know we were talking.

Speaker: Yeah. it's so It's again, it's the go slow to go fast. Do you immediately update something and or do you wait? Like there are, I mean, there's GitHub bots out there that they go to your lock file and you're pinned to a specific version and they go, Hey, there's an update available and it can run your CI pipelines. You know, you can merge it with a click and now you're on the new version of the package as opposed to, Hey, I haven't updated this since 2022, but like, what's the best mix of the two?

Speaker: I don't know, man. I think you have to regularly update. So I've always kind of been to the standpoint of if it's features being added, i don't know that I need more features. But if it's secure, i do need to be secure. But at a certain point, I have to keep a current version. So let's say something drops...

Speaker: in October of this year and it's not a security thing. Well, I think by November, December, I'm probably thinking about bringing it in so I stay current. Whereas if I waited three years, like you said, oh, I'm secure, but at some point I hit end of life. And that's the biggest jolt to any company in my experience is when something goes end of life, what do I do? A colonel has officially retired, but now I'm against the wall instead of just kind of keeping pace at times.

Speaker: Yeah, well, and especially, you know, back in the open source world, it's not even something going end of life, but someone decides tomorrow, hey, I'm not going to maintain this anymore. yeah What do you do? You know, are you going to fork it and maintain it yourself? Has another company decided to do that? Like, we've actually it's actually happened this year, believe it or not, I think on the Python side,

Speaker: Yeah, the Python side, there's a major library that major systems use, including Home Assistant, by the way, which itself is open source, but I think we've covered this on a previous podcast. The largest active repository on GitHub.

Speaker: They use a they use an a Python package called HTTPX. which the maintainer decided I'm not going to maintain it anymore. Thankfully, there's a big foundation out there that for Python, because Python is so large that, you know, brought it in, they forked it, they're going to keep maintaining it. You just have to cut over to their package version, but you have to then live in the world of like, can I trust the people that are taking this over?

Speaker: Is it legitimately being taken over or has someone compromised it, you know, and is there a, what's the migration path? Is it a seamless migration path? Well, I think that's the whole lesson we learn in the Linux world, right? Is CentOS is good to a certain level, then we get forks, what are we doing? And that's where IBM and Red Hat really came together. And it's like, here's Red Hat Enterprise Linux, you know?

Speaker: Guess what? You're paying for it, however, we're going to say that this is a real system. and And that's one of the things that I used to gripe about Linux back in the day was this is great until it's not.

Speaker: So at some point in order for somebody to really have the management of something, they have to get paid. I mean, let's face it, it takes man hours, it takes staff, there's a business model here. And unless you're selling someone's data, how do you for free maintain some sort of package?

Speaker: you have to be monetized for it. So I've always been leering back to the open source of, you know, when it's free, how long does it stay free?

Speaker: Right. that the best ones The best ones that have done it are like Linux themselves, Home Assistant, or even um I think it's Pydantic, the one I was talking about with Python a few minutes ago, you know, and it's kind of the approach WordPress has taken too, I want to say, is you go and you establish a nonprofit foundation And you get donations or you get sponsorship from companies using your software. And in a perfect world, you're not beholden to shareholder expectations or any of that where you constantly need to be raising prices and turning a profit. It's just making the money for you to pay your employees and survive as that nonprofit entity.

Speaker: No, that makes sense. But I think from an enterprise, that's always been the scary part of me of, oh, this is open source till it doesn't exist till they charge money. Like it really is a nice little lore to be like, look at how great my software is.

Speaker: Right. And then all of a sudden it's not so great when they charge you an exorbitant amount of money and you're like, what do I do? so At some point you have to pay. you You can't just use free in my mind. So certain open sources are great until they're not. That's that's my lesson. So always. Well, and even even with open sources, you might be paying because you might be dealing with so many bugs yourself that you're forking it or committing work upstream and spending your developer hours fixing issues or making it work to your needs.

Speaker: Well, and that's where I like vendors that have been open source. Like there's some firewall companies that do this, that you can pay support and hey, at least I can get some support. There's some storage vendors. It's like, here's the free version of our software and you get no support, right? And then you move into I want support and it's like 12,000 times the cost. And you're like, well, wait a minute.

Speaker: I want the the functionality of the free. I'm willing to pay you support, but I'm not going to pay you, you know, $1,000 a terabyte or something, you know, like, come on guys. Like there needs to be some homostasis in there. And so it'll be interesting to see long-term how this goes because I love open source.

Speaker: However, from an enterprise, I get weary of it. So my home lab loves it when I'm putting things into prod. I'm like, ah, Well, and it's I guess it's true of enterprise software too, but I've definitely seen...

Speaker: I've definitely started noticing a trend developing more and more to that open source wall can be great because it tries to be a one size fits all solution. And a lot of cases it can get very bloated very quickly. So yes, there's a positive in something being open source that, Hey, you can wire it up in a couple hours and have a working proof of concept or a working system or feature on the flip side.

Speaker: Again, back to the performance side of things. Well, number one, obviously the security side, it's harder to audit that code and make sure it's not compromised, but like, I don't know how many packages I have found.

Speaker: Some of this is just tech debt, right? So our portal is however many years old, there weren't necessarily like native functions within a web browser to support some of these features back then. So these packages existed, but some of these packages are taking up tens or hundreds of kilobytes, even compressed and inflating the page size when you can replace it with something relatively custom that takes up almost no space or uses a native browser function and takes up no space.

Speaker: Well, I think that's why when people are in web development, you run a course and then eventually you do a version, right? It's like, okay, we had version A it landed three, four years. Let's build something net new, not because we want to go through the hassle of redesigning all the page content, but let's build the engine. You know, the latest WordPress, let's say whatever plugins, or if you're doing Node.js, React, those things, you're going to build it up. You're going to do it in the latest flavors of whatever, and now you're going to move your content over. It's interesting because to design it, you don't care about content, but once it's up, you care about content. So to me, if I have to fork it and I have an existing website, at least I know I can design and then port content over. Even if it's a manual process of moving that content, man, does it run so much better.

Speaker: So definitely something to to keep look. mind on, but I think we're kind of coming to the end of our episode here and I'm going to give you some closing thoughts. Like I like to do, I'll throw a closing thought and, uh, you know, this is your time to shout out to your company as well. So go for it, Jonah. Last closing thoughts.

Speaker: Yeah. Um, if you're an open source developer, please, please think about how others might use it, how it'll survive long-term, you know, succession planning.

Speaker: Don't just put something out there. I mean, I would hope someone would fork it, but it's never a guarantee. Oh, absolutely. And how about paying the bills? Give a shout out. yeah ah If you need offsite data resilience, MDR, SOC, SEAM, cybersecurity, backup as a service, offsite disaster recovery, come talk to us at Cyber Fortress.

Speaker: I guess my closing thoughts are I think we've kind of covered it all. it's What you're finding is there's no easy answers to this. A lot of it is gut sometimes. we We can argue all day long people process technology, methodology. There's not really a hard concrete Bible to follow. So I think you look at standards, NIST, you know DevOps,

Speaker: whatever you want to work on. And always remember this. It's what makes sense for your organization. So I always joke if I had three different people who were all running agile and I put them in a room and I said, who runs agile? They'll all raise their hand. And you're like, great. Now talk. They would each have different flavors of that. So that's not a bad thing.

Speaker: right there's guidelines and what makes sense so there you go paying my bills different dev come see us for all your veeam needs where it comes to consultation think professional services implementation hands-on monitoring uh fractional backup support um think of us as a hired person on your team so You don't have to hire yet a backup person. You're getting the expertise of a backup person that is dedicated so many hours and it's a team.

Speaker: So come check us out at differentdev.com, cyberfortress.com. All right, folks, we're going to be out of here and play some cool music. We'll see you next time.

Speaker: Whether you're a seasoned solutionist or just starting out, the Rebel Devs podcast is a must listen. So join us, comment, share and subscribe. Be part of the Rebel Devs.

Speaker

Speaker

Speaker

Speaker

Speaker

Speaker

Speaker

Speaker

Speaker

Recommended