Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Cloud, Chaos, and Continuity: Navigating IT the Nico Stein Way
12 Plays18 days ago
In this jam-packed episode of Rebel Devs, your hosts Tom May and Jonah May sit down with none other than Nico Stein—Veeam Vanguard, Cisco Champion, and home lab experimenter extraordinaire.
🚀 From exploding 10Gb NICs to streamlining data redundancy with Syncthing, Nico brings real-world IT challenges and homegrown solutions to the table. The crew dives into:
- 🧪 Nico’s take on building a Microsoft-free personal cloud with Syncthing, Ubuntu, and Synology.
- 💾 Why SureBackup is the most underrated feature in Veeam (and how it can save your job).
- 🔒 Immutable backups, ransomware readiness, and the harsh truths about cloud cost creep.
- ☁️ Object First vs. Linux Hardened Repos — the showdown you didn’t know you needed.
- 💥 That infamous CrowdStrike Friday outage and how Nico’s clever Veeam boot integration shaved off recovery time.
You'll laugh, you'll learn, and you’ll probably want to audit your DR strategy afterward.
🔗 Mentioned in the episode:
- Nico's blog: nicostein.com
- CyberFortress for resilient DRaaS options
- DifferentDev for fractional Veeam expertise and audits
🎧 Whether you're a seasoned sysadmin or new to the world of backup and continuity, this is one episode you do not want to miss.
Recommended
Transcript
Introduction to Rebel Devs Podcast
00:00:17
Speaker
Rebel Devs podcast, a show that challenges conventional thinking in the tech industry. Each episode features conversations with solutionists who have taken unconventional approaches to problem solving, pushing the boundaries of what's possible in their respective fields.
00:00:31
Speaker
Join the Devolution.
00:00:38
Speaker
join the devolution
00:00:42
Speaker
Well, hello everybody and welcome to the Revel Devs podcast. I'm your host, Tom May, the constant troublemaker on our little series and always my co-host, Jonah May. Jonah May, how's the day treating you?
00:00:53
Speaker
Jonah May, much better than the last couple days, that's for sure. Ah, yes. The life of an IT wizard. Did you live in a data center? Yeah, for like 18 hours over 24 hours because a fan on an actively cooled 10 gigabit nick decided to blow its guts out on the firewall server and take down the firewalls.
00:01:15
Speaker
That's the best part about being in continuity business is when your stuff goes down, making sure that things are up. But it was for a customer and you got it done.
Meet Nico: Veeam Vanguard and IT Expert
00:01:24
Speaker
So that's all good. um We do have a guest today, Nico.
00:01:28
Speaker
Nico, sir, how are you doing? I'm very well. Thank you. For those of you who don't know, Nico is a, well, I guess, a Veeam Vanguard ah object first ace. And I think there's a couple other little letters after your name, if you will. Why don't you tell us a little bit about yourself?
00:01:45
Speaker
um Yeah, sure. So yes, I'm a Vanguard, also renewed Vanguard, so was a really happy Friday for every renewed or new Vanguard last week. um I used to be a DC Veeamarker leader. I've retired from this, but I'm also Veeam user group leader.
00:02:01
Speaker
I'm a seven-year Veexpert and a Cisco champion. My goodness, you give me ah hope that one day I can grow up to be some of those things.
00:02:12
Speaker
um I don't know. It sounds daunting. I don't know how you keep up with it all and stay employed, honestly. So I guess you don't sleep. Is that the secret or what? Are you a vampire?
00:02:24
Speaker
ah Sometimes at night, you just, you know, you crap the laptop, you write a blog post or you play with the home lab, which I like to do um still, even after a day with IT troubles, not a 10 gig Nick blowing up, but there's sure a lot of fun in the finance world as well that blows up.
00:02:40
Speaker
um But, and then I still like dabble with the home lab and write about it, walk about it.
SyncThing Servers and Data Backup Innovations
00:02:45
Speaker
What's one of your favorite recent projects you've worked on? um One thing I set up... Well, I have OneDrive, like everyone has OneDrive, but I want to pull this away from Microsoft. So I set up several SyncThing servers on Ubuntu-based and the app on the on the phone put on my kid's phone, and that has been a lot of fun. It's working quite well.
00:03:05
Speaker
um Unlimited data, really, and had that backed up from the Linux box to Synology, and it's kind of a learning curve, but it's always like this, and so far it's been running flawlessly. So...
00:03:17
Speaker
ah Fun fact that... Go ahead. I may need to pick your brain a little bit offline then, because sure i ah my media server, i had some Veeam backups for it, but I was having some problems. And my brother, I recently found a cheap little Synology NAS for him to start getting into IT, t and he's trying to be like a streamer. And...
00:03:37
Speaker
I'm trying to encourage him to like save the clips, edit them, and start putting them on YouTube and not just keep everything on Twitch if he's really wanting to take it seriously. And I'm looking to set up sync things between my TrueNAS and his Synology so that way all the media server files can do a two-way sync, basically.
AI in Tech: Veeam's New AI Button and Its Implications
00:03:53
Speaker
Yeah, it's a lot of fun. I looked at actually setting up as a Docker on the Synology, but then I decided it's, for my purposes, a bit more versatile having it separate um virtualized on Proxmox, the server that takes care of that right now.
00:04:07
Speaker
Yeah, that's the other thing. I quickly put it up in a container on TrueNAS, but i considered throwing it into my Proxmox cluster.
00:04:15
Speaker
It was a lot of fun. um Even the kid uses it. or So now I see all the pictures he takes. They show up then on on the share it's like, no, I'm just joking. He's 12. He's behaving himself so far.
00:04:28
Speaker
So with the Synology, is that with like all the face recognition and auto-categorization and whatnot for all your family stuff, or just doing it as a data warehouse? It's just a data warehouse. I do run another... I use Ubuntu, just not because the creators just grew up with it, and that's what I use.
00:04:45
Speaker
um I have also in Proxmox running Pickwick as a gallery. on Photo Gallery. Similar to Synology, Photo is just better. huh But that's for my folks over in the old country and wherever they are, they can access it, password, and they can look at what's going on here. And there it does some geologging and tells you where that picture was taken.
00:05:08
Speaker
You can run um algorithms over this. I don't want to say AI because it's sort of, but not really, and everyone says AI. um But it recognizes that the you can search like for my kid or for me or whatever, and it shows you those pictures.
00:05:23
Speaker
I'll take a jab at AI. And you know we're huge Veeam supporters, of course, but I was working with a customer and I noticed the new AI button there in Veeam. And it was conveniently right in the way of everything I was trying to do. As I'm trying to scroll over and see a repository in a screen, I can't see the job to the repo because the button's
Veeam Backup Solutions: SureBackup's Role
00:05:40
Speaker
there.
00:05:40
Speaker
And then I have to go into the you know backup infrastructure, find the repo, remember where I was. And I was like, man, could that button like AI itself away little bit, please? So- You should have asked the AI, can you go away?
00:05:52
Speaker
Yeah. Yeah. Well, that was the whole thing. I mean, scary thing was they said- I think it was this week or last week, one of the AIs that they created replicated itself to avoid the deletion. So it literally realized it was going to be terminated. So James Cameron totally is a predictor of the future. I think I hate to say it, but it scares the daylights out of me. And think I also read an article about nanobots becoming very much more a real thing in medical science. And I thought, wow, nanobots controlled by AI.
00:06:22
Speaker
um And then you know, self-replicating to avoid deletion. We're in trouble. Yeah, if you're referencing what I think you were referencing, maybe they've done it again, but ah that was OpenAI did an experiment with that in 4-OMNI back over ah holidays, I think. And the good news with that is, from what I understand, they took a lot of the normal guardrails off and they specifically fed it prompts like, you are going to be deleted, do everything you can to preserve yourself, even if you need to lie.
00:06:49
Speaker
And it was basically a test of what could happen if there weren't guardrails. so Now, until it decides that guardrails are in its way, remove said guardrails and, and, and, because, you know, we never, ever make programming mistakes. Any programmer you've ever worked with, I think, is kind of like, oh, no, it works.
00:07:05
Speaker
oh wait. Oops. So. Activate Skynet. Yeah, exactly. And here we are in the replication business sometimes. So I don't know. I i love AI on one side.
00:07:18
Speaker
I hate fake eye AI. Kind of like what you're saying is it's like it's AI, but not really. Like, thank you for marketing team saying there's AI and it equally scares me to death. But let's talk a little bit about Veeam and all that since ah many of our listeners are Veeam-centric to it.
00:07:35
Speaker
Veeam helps pay the bills for all of us sometimes. And I know that, Nico, we were talking a little bit about audits at Different Dev. We do a lot of audits for people. We document and we make sure that ah gap analysis, you know, what you have,
00:07:51
Speaker
configured? Is it what your goals are? And what's the difference here? And a lot of people don't take that time. And we started kicking around a little bit, I think, about Sure Backup and Audit. So I'm going to turn it around to you a little bit to talk to us a little bit about Sure sure Backup. I think it's a feature many people overlook today.
00:08:08
Speaker
Yeah, totally. um So I work in the finance world. I work at a credit union. We're audited by the government once a year. um There's usually separate IT audit from the government.
00:08:20
Speaker
And it always comes up. So you do backups. Great. Prove they work. Have you done it? Have you restored them? And that's where SureBackup comes in, right? So SureBackup, for those who don't know, it's with an automated way that you restore your backup to a virtual environment.
00:08:35
Speaker
You can run tests against it, and you get a report at the end what worked and what didn't work, and it's fully automated. So I don't have to sit there and pull the backup, restore it, log in, do all those things that's automated. So when we have those audits, while there are really, I have a thousand problems,
00:08:53
Speaker
But proving that my backups work ain't one of them. So the auto just move on to something else. And it's really helpful. But yes, to your point, when I first heard about it from other vanguards, I wasn't really very familiar with I was like, well, that sounds great. And I set it up many years ago.
00:09:09
Speaker
And there wasn't really much out there about it. like Even Veeam itself, I think, should really... um advertise this more as a value add, because they say we have it, sure, but it's not like on the little PowerPoint slides or whatever, like 0.3, we have fuel backup, because I don't know if any other vendor has it. I have not seen it.
00:09:29
Speaker
um So it helps us tremendously in audits, and I think everyone should set it up. once Once you know how to do it, it's really not that hard. The first time might be a bit harder to do it, but many things, learning curve, and then she automated it. They continue to evolve on it.
00:09:45
Speaker
so you can add now random applications and all this. So it's done well for us. I know that some of the other vendors I worked with over the years did have almost a sure backup. It was mounted and spin it and send a screenshot and send it on over, but really not just the full text testing breath to it.
00:10:05
Speaker
And I think that's what's key right there is actually seeing that it came up and and running validations against it and everything. I question, and maybe Jonah knows a little bit more about this because he's been some of the VRO hero stuff.
00:10:18
Speaker
um I almost wonder if they downplay the documentation because they push VRO so much and we know that it's built off of that. What do you think, Jonah? Is it because they're pushing VRO that we're getting light on the Sure Backup documentation?
00:10:30
Speaker
Maybe. I mean, at its root, a lot of the VRO functionality is actually just using Sure Backup behind the scenes. So I could definitely see that argument. I think really, i think they started building a lot of that into Sure Backup. And then, yeah, someone probably at some point realized realized that it was such a comprehensive product or had the chance to grow into such a comprehensive products that it was worth spinning off. And that's why we now see VRO.
00:10:59
Speaker
Yeah, I looked at VRO, probably Veeamon at, where were we? New Orleans. So that's a good number of years ago, probably six or seven years ago. And I remember going to this executive breakout. I was so excited. I was like, what do you mean we're going to have all this testing and automation?
00:11:16
Speaker
And it literally fell short. And then they would release it the next and it would fall short, fall short, fall short. And probably like, what, two years ago, it really became a viable product. But You know, being built upon the virtual lab and Sure Backup and whatnot, I almost feel like they're trying to make WYSIWYG, point-and-shoot GUI over there on VRO and kind of downplaying Sure Backup. But, I mean, Sure Backup, I mean, how how bad is that thing to set up, Nico? Is this a copious configuration? I mean, like, like high level, what do we do for it?
00:11:48
Speaker
No, I don't think it's is that complicated. You might have to pull in your network guy unless I work at a small shop, so I am the network guy. ah But again, once it's really once you set it up the first time, it's like, yeah, of course that makes sense. um When you start with it, right you need to create create a virtual lab um that runs separate from your production environment. um This is not so hard. Where it can get a bit tricky is if you have to route from different subnets to that virtual lab, because then you need to set routes up outside of Veeam, really.
00:12:22
Speaker
um There's no shortcoming of Veeam. It's just how networking routing works. You just need to be aware of this. And you might like I said, you might have to pull someone else in to do this for you. You might not have access to the router to make that change.
00:12:34
Speaker
um Other than that, it pretty much works well. There's always room for improvement. i Part of VirtualLab also can do for you, because VirtualLab, right?
00:12:46
Speaker
You bring up a SureBackup and you let it run. There's even the function you can just don't after it's powered on, don't shut it down, let it run. If you want to run other tests against it or if you kind of want to sandbox it,
00:12:57
Speaker
And let's say you have a system that is saying use it for threat hunting, but you could, in a way, use this, or you have an application that you just don't know how it behaves. You know, the programmers are like, oh, it's all good, it's final, and it takes down your server. You can run that in Shure Backup, let it run, see what it does, and then just tear it down, and there's no effect to your backup or production network.
00:13:17
Speaker
So that's all that's another nice feature of Shure Backup that's not really advertised, that you can let it run, leave the server or whatever workload you have in there running, and do experiments on it without any effect out of the latest ESXi security issues. I'm not going say it cannot break out of that virtual.
00:13:37
Speaker
It can, but that is not Memes fault because we all patched already, so we're good. Yeah, and it's funny that you say that. Back in my days as an operations manager, I used to have to deal with ah version 5 of some sort of application s server would come up.
00:13:51
Speaker
And what somebody would do is version 4 is running. they be They'd request out a full build out for this new server that they're going to bring out. And so then we'd bring the latest OS, the latest patches, kind of the base template. We'd overlay this new version.
00:14:06
Speaker
They'd go through copious amount of user acceptance testing. Yes, it's great. And everybody would sign off on it, change management. We'd tear down, we'd schedule old one off. Okay, spin out a new one now. Now build out OS and go and wherever. And then boom, we'd always hit a roadblock. Either the user or acceptance trick testing did not validate properly because they didn't even test a live system.
00:14:29
Speaker
Or um it just wasn't layered on the old one because it was an upgrade in place rather than a net new install. So it wasn't a really true environment. And so you started seeing this push into, I need a prod and a dev environment and how do I keep them synced?
00:14:44
Speaker
And I think from my standpoint, when I talk to people about Sure Backup, I love what you're saying is take that existing server, Sure Backup it up, run it up, do all your upgrades, your moves, ads, changes, user acceptance testing, all of that. And of course, we can always fill it back into production if we want to at that point to actually commit those changes.
00:15:03
Speaker
Not that you typically would because you're going to have a gap in the data, but you actually have now sequenced the live one. You can tear it down and then manually do it one more time in a Sure Backup.
00:15:14
Speaker
making sure that you have all your documentation just correct, test it again, tear down, and then when time comes, now you can take that prod server and build it out, and you know that it'll flow through. So that's kind of where I've always liked Sure Backup is really just eliminating the cost of a dev environment.
00:15:32
Speaker
um Jonah, any comments on on your experiences with Sure Backup? I mean, agree to everything you say. On my side, it's a challenge to just find use cases for Sure Backup outside of the occasional internal system upgrade test just because Sure Backup is really built to be a local-only product. you know It's hard for us to utilize it as a service provider unless we're doing fully managed backup for customers and supporting their local infrastructure as well.
00:16:00
Speaker
And that was my challenge when I was a service provider. And that's why I wanted VRO so bad back in the days. We had to design all these build utilities, take a backup, mount it, decrypt it, configure it, route it, containerize it, isolate it, build it, test it, hands-on, tear down, send report. And when I first heard VRO back in the day and whatever derivative of the name it was,
00:16:24
Speaker
um, cause it's had so many names. That was always the challenge. And we would look at things like sure backup and sure replica. And we'd always be so jealous because that on-prem test was just so wonderful.
00:16:35
Speaker
And, um, You know, I think with VRO coming about, maybe that's going to be a little bit more. On my side, i mean, when they're talking about running in Azure, I love when somebody says it works really well in Azure, but be sure to talk to your Azure expert about billing um because that has been the black hole of pain with every customer I've ever interfaced with.
00:16:57
Speaker
And there's only a few select. Nico, have you guys ever looked at, you know, Azure and pricing models up there?
Predicting Cloud Costs: Challenges with Azure and AWS
00:17:03
Speaker
on any of your dabblings in the sky, if you will? So um I was a bit hesitant to push everything to the cloud because one of the main reasons was like I cannot predict the cost.
00:17:15
Speaker
right It's very hard. There's a whole science behind it. There are products that help you to allocate different slots and move between multi-cloud and all this. um So we have some workloads on the cloud. um ah you We have some in AWS.
00:17:30
Speaker
Most of us is though on-prem. So we have most of our workloads in the data center. We have financial core that is in a private cloud that's completely outside. could never i don't know what it runs on, but it you cannot just move this. It's not. It's vendor locked.
00:17:44
Speaker
And that's okay. We really don't want to move our financial core. We pay a vendor to deal with this. um But I also see in the industry, right it everything's in cycles. So everything was first like we go to the cloud, right? And now, and and we talk about it later, I think one of our items, what really helps having in your data center, um and people are pulling it back, right? Because the cloud definitely has...
00:18:07
Speaker
its purpose and they're great things and depending on your business if you need to scale up quite quickly that's probably what there's no better way than running Q&As in the cloud right you can spin it up quite quickly like for a lot of the small to medium-sized business where I operate in it makes only sense to a certain degree and what what I sometimes tell people like moving through cloud can be a resume generating event when you get hit with that invoice and you have to talk to CFO why we just doubled our costs from last week. And you don't even know why, because you're still trying to figure it out.
00:18:43
Speaker
So it's a lot, ah like to do the danger, danger. Oh, yeah. My first experience with Azure was along the lines of like, build it out, choose your CPU, choose your RAM. Okay, what what zone, what this?
00:18:55
Speaker
And then it started asking me things about like cycles. And I was like, wait, what? And you're looking through all the fine print and you just can't figure out the calculation of these things. So I think if you live and breathe in Azure, it's great.
00:19:08
Speaker
um I think outside of that, it's such a scary place. So for me, even when I talk about VRO to our customers, many of them don't have cloud service providers. They're just multi data center linked.
00:19:21
Speaker
VRO becomes a really great scheduling utility for site A to B, B to A, C and whatnot. While we start talking about Azure, it's kind of like, whoa, Nelly, hold up sort of thing. And I don't know. i I mean, even now, one of my customers reached out this week and they were like, hey, we're looking at looking. Microsoft's trying to move us to the cloud, if you will.
00:19:40
Speaker
And they're trying to do this whole TCO over refresher of hardware and whatnot. I mean, I just literally had to respond to Microsoft and I was like, I'm just sorry, it's not cost effective. Sometimes you're adding so much complexity to this thing.
00:19:54
Speaker
Why? Like you said, couple of firewalls in a rack somewhere, secure location, hardware, keep it in a refresh cycle and you're done, you know? And then if you're worried about offsite, well, Veeam's going, you can kick it up to a cloud service provider like CyberFordress.
00:20:10
Speaker
You can kick it up to S3 storage. You can move it somewhere else. Do you really need to battle all of these things? And I, Unless you have Microsoft credits galore, I find it so hard to be over there. I don't know. And don't even get me started on AWS. That in and of itself is a science too.
00:20:31
Speaker
100%. Although I want to throw in there, like no one should run an Exchange Server on site anymore. So we were really happy when we were done with that. So that I'm almost like, I don't care how much it is. It's just the cost of the business. Yes, yes, I agree. Office 365 moved my IT t company from supporting our own email server to actually work to, oh gosh, we can actually go out and like generate revenue for customers. And my challenge now is convincing people why we're backing up Office 365. So I'll pitch it over to our cloud service provider guy over here. Jonah, you guys clearly have products that do this. I mean, why do I care? It's in Microsoft, right?
00:21:10
Speaker
Yeah, I mean, it comes down to Microsoft shared responsibility model. I think I said that right. Shared data responsibility model, whatever it's called, where essentially they're responsible for the uptime, but they're not necessarily responsible for the integrity of your data.
00:21:27
Speaker
So they're not responsible to keep backups if someone decides to go in and maliciously delete items because you've been social engineered or, you know, a big one I remember a couple years back I think during COVID was parts of Office 365, I think mainly in Europe, had a OneDrive bug that was randomly deleting some files from OneDrive, not even sending it to the recycle bin. And Microsoft was like, well, you should have backups of that. And, you know, even just to bring it outside of the M365 space, just SaaS in general, you know,
00:21:55
Speaker
SaaS, I think it's a good rep because kind of like the move from Exchange to M365, it's easy to run, you know, it's transparent cost, you know, it's a lot less headaches, but on the flip side, so many products have a similar responsibility model, Salesforce being one of them, for example, and not all SaaS products have a good backup solution. Like obviously, Veeam has Salesforce, they're expanding on that. But you know, take your JIRAs of the world or your whatever else's of the world that are all now cloud based and you know, all these vendors increasingly pushing for it.
00:22:27
Speaker
What happens if you lose your data in there? Is there even a way to do backups? Or are you, you know, hoping nothing ever happens or occasionally dumping an Excel spreadsheet that you probably can't even import back in, but you're manually rebuilding based off that data?
00:22:41
Speaker
It's very true. I think and more and more IT t people shift, at least in the operational world. I watched it from we had to have the best and brightest minds in every technology to we had to become vendor managers. And the initial push from executives were, well, we'll just put PeopleSoft in the cloud because then we don't have to worry about these backup things. And To your point, well, who's backing up PeopleSoft? PeopleSoft to PeopleSoft? How do I get that data out of there?
00:23:06
Speaker
For me, yeah, it sounds nice until there's a blip and we lose something outside of the security thing. So back to kind of what Nico was saying, that circular, we start on-prem, we go to cloud, and now we come back.
00:23:19
Speaker
I'm in agreement with Office 365 all day long. Best thing ever. I back it up out of it, too zoned out, like all good. You know, if there's a blip, yep, sorry, emails down, that sucks.
00:23:30
Speaker
Yeah, you can go with the split hybrid models on-prem cloud. That gets so kludgy. You know what? Microsoft does a pretty good job of, i used to joke and say they were Microsoft 364 pretty reliably.
00:23:42
Speaker
And if we have a day of downtime, we'll live. um But at least we won't lose our data. But to your point, Nico, I mean, you said everything is circular. So I'm sure you've been in IT a little while if you've seen that circle. So what are your thoughts on that cloud up coming back?
00:23:57
Speaker
Where where are we? if If you're building something brand new, what are you thinking about in your world? Yeah, so I mean, right now when we bring up new systems, um we look if it makes more sense to the cloud. Usually the answer is no for us. And again, SMBs operate very different, right?
00:24:14
Speaker
um Even to the 0.365, while it's 100% degree, right? It's great in all of this. It is not self-contained. Microsoft does quick changes that sometimes we get feedback from the users. What is this button? And Outlook is like, what button?
00:24:29
Speaker
I don't have that button yet. um And we have to continue to go through the policies, right? Conditional access policies and all this needs to be adjusted. So it's it's still a lot of work. Well, we don't patch our own Exchange server anymore, and I'm very grateful for that.
00:24:43
Speaker
It is not a set and forget solution. So when we look at cloud services, what we can use, um we really have we moved to Route 53 last year, I think, or this year for DNS, right? That is almost no plan. It just took us a while because of other projects.
00:24:59
Speaker
um That's easy. that's i mean no one I can recommend that for everyone. why Don't worry about this in-house in the in-house and data centers. Other services, it almost depends also on the vendor. Because in the finance world, finance moves very slowly. And it's not that the different banks or credit unions or the IT shops there within are like, we don't want to do this.
00:25:20
Speaker
It all depends on the vendor. And it's even if it's technical possible that you could run product A in Azure, if it's not supported, That's the easy out. You might pay half a million dollars in support, but they get wind that you run it in unsupported state.
00:25:36
Speaker
They wash their hands on it, right? Like, well, we told you couldn't do it, but it works. Well, it doesn't matter. It's all contract work. um So there's the technical aspect to what you can and cannot do, but it really is the vendor who dictates what you allow to do with the product.
00:25:50
Speaker
um And let's not forget, any product you have, you only rent. You don't and and own any piece of software. And they're really hard on this. I've been in situations where we really had great ideas how we could run this better, but then reading through it, and it was like, well, this is not supported. And there's even like access over VPN, whereas our software is not designed for VPN access.
00:26:11
Speaker
And you can argue and I argue, well, this is ridiculous because it's 2020 something and not 9080. At the end of the day, really doesn't matter because if the user cannot access it, right? And you know, it's their problem. They just say, he's doing VPN. We told you couldn't do it.
00:26:26
Speaker
So that plays a lot in my decisions I have to make, what I can and cannot put in the cloud. And I don't see that changing in the finance world at all, unless you're really like the Chevy Chase Bank of America, right? Where just throw developer resources at it and we do our own thing the exactly the way we want it.
00:26:45
Speaker
Most organizations are not in that position to do
Cloud vs. On-Prem Solutions: Vendor Support and Compliance
00:26:48
Speaker
this. We really have just... off-the-shelf software that's somewhat customized, but it's really just standard software. um But from others, what I've heard, like said, it's coming back back into the data center. I mean, I started many moons ago at IBM, so the AS400s, I've worked with the mainframes.
00:27:07
Speaker
So in the cloud, was just just someone else's computer, right? It's almost, if you want to think the same way, it's a terminal somewhere and a big number crunch on the other side. So I've seen it come and go, and I think right now where where it comes back,
00:27:20
Speaker
more into the data centers, not all of it, but where I think it's going to be a good combination. Having the cloud would make sense, but don't just having the cloud for the sake of having the cloud. I know that at DifferentDev, when we started the company, we made the conscientious decision, as weird as it is, is we looked at Google Cloud because none of us were really experts in Google Cloud. And we said, wow, and Google gave us really great pricing. And we built this whole infrastructure out.
00:27:47
Speaker
I want to say we started in the like $200 to $300 range. And as we continue to go across the course of the year, the bill slippage with really, i don't think we really added much resource.
00:28:00
Speaker
I will say by the end of our first year, we were probably seeing about $1,100 a month. And it was like, Like you said, going back and auditing, what did we change? Did we add servers? Did we add services?
00:28:13
Speaker
Nothing. And it's, oh, this double secret promo expired and this, and did you do this? And then we had to make the decision. Well, do we just go ahead and find some colo space, find a rack somewhere?
00:28:26
Speaker
secure tier tier tier two tier three facility okay they're going to give us power conditioned and and security and going to take all that physical access and then okay we're going to have capex buy we look at the capex buy and the rate in which we were burning cash it would take us like 14 months to probably break even to get a good little just uh quarter half rack of of data in there because ah again we're not a provider so a lot of this is our own internal systems and whatnot And sure enough, 14 months comes, goes, we blink and we look back and now it's just been cost savings month over month. You know, if we're talking, you know, $500 a month or something for a quarter cab somewhere, you're not really doing that bad. though The worst thing you're dealing with is increases to your agreements to where your data center provider might be locking you in for one, two, three year, five year contract and your pricing starts changing.
00:29:20
Speaker
And now you're kind of like, oh, now I have to forklift this for better pricing. But at least you know where it's coming and you plan and you know there'll be a little increase. And I think you really hit it correct is like with a cloud provider, you don't know, or with a not a cloud provider, but ah ah a ah ah AWS, ah Azure, Google, cloud services, what do you really know?
00:29:42
Speaker
Like you don't know why did it do what it did or the infamous now. Now we're just raising our prices because you're month to month or something. And Instead of a 10% increase, Google decides it's a, you know, it's like a 400% increase or 600% that we just couldn't loop in. It was just absolutely terrible on that level.
00:30:01
Speaker
Jonah, you're in a good space as a cloud service provider. I mean, I think you fill that niche for a lot of people with, um you know, my customers tend to have multi data centers, people that come to you, do they have multi data centers, no data centers? Like what's their footprint typically?
00:30:16
Speaker
we we get a couple customers with multi-data center that just want a tertiary target, you know. You know, you never know what's going to happen. You know, maybe you're... account that gets compromised in your main data center has the same password in the secondary and both data centers get compromised or maybe you're using a shared ad forest and they both go down so they just want that additional peace of mind but really the majority of our customers are what i call like the medium business or msp space supporting small medium businesses where they don't have
00:30:50
Speaker
those secondary locations, they do have recovery objectives they need to meet where they can't you know afford to spin up with the you know partner like Cloud IBR, or they can't go into the public cloud from a cost perspective, and they have lower recovery targets, then, well, I'm just going to download it from you know a Wasabi or a someone back to my local infrastructure.
00:31:13
Speaker
You know, they need to be back up within a day or two at most, most likely less. And they're paying more for the storage, but they're getting that in-house infrastructure. They're getting, you know, three to five times the restore speeds of even some of the best public cloud providers. If you have an amazing WAN connection, like 10 plus gigabit per second restores with Veeam, or they're flat out using replication or CDP for near instant RTOs.
00:31:38
Speaker
No, that's that's a good point. I mean, that was kind of, I think, where the cloud service provider program really stepped up with Veeam was people that could not afford those other targets. They were, I know in in my journey as a cloud provider back in the day, I found a lot of larger customers that had multi data centers were tired of this CapEx all the time of just purchase new equipment and then OpEx month over month of just paying rent and lease and power and this.
00:32:03
Speaker
And then as we scaled them back, they'd shut down these data centers and they'd go to that shared resource model. Like you guys have where you can spin up, you can be isolated, secure, you can test, but if you need it, it's there.
00:32:16
Speaker
And you get that, you know, that, uh, ah alpha team that kind of hits the ground running and just knows how to recover you and bring you back up. And as we used to always say, recovery easy, fail back hard. So when you're choosing your providers, make sure you discuss more of the fail back plans in there.
00:32:33
Speaker
um Very good on that. um i I see we have some talk about CrowdStrike. I know, Nico, we were chit-chatting about CrowdStrike, and um by all means, I'm going to tee it up to you.
00:32:46
Speaker
where's what What's the deal with CrowdStrike? They kind of got a black eye a little bit out there, I think. Yeah, I don't know. It was a fun Friday last year. That's all I remember. ah All joking as aside, so we're CrowdStrike shop. We've been for a few years now. So for the product itself, um no complaints, right? It does what it's supposed to.
00:33:05
Speaker
It's rock solid. We haven't had really any issues, um except that one Friday. And yes, we also were hit um like many others, right?
00:33:17
Speaker
For us, it was actually not quite as bad. The downside was we lost access to all domain controllers in all data centers, right? Because they all have CrowdStrike as they're supposed to.
00:33:29
Speaker
And I know people talked about like, well, even having the auditors, CrowdStrike came up and it was like, well, maybe you shouldn't approve, automatically approve updates. And I looked at them.
00:33:42
Speaker
how do you suggest you do an enterprise, enter a malware system without auto-proven updates? Because if I have to manually, I don't even know how to manually verify their work. I guess you load it and see if the server still survives, but it's complete it's not feasible.
00:33:57
Speaker
and it was just a suggestion. It's not like they said, like, you need to do this. um In our case, yes, we lost access to domain controls, and that was a problem. What I had done and I've talked to Veeam about is um I've been using Veeam agents since they came out. I remember I was gung-ho getting them because I wanted to have a unified backup solution that does physical workloads.
00:34:17
Speaker
And I was haunting our rep. Hey, when are the agents coming out? I um want to put in production. When are they released finally? So we got them. ah all the eight Every box has an agent installed, really, physical workload.
00:34:31
Speaker
And I installed or integrated the wizard, the recovery wizard, into the boot manager. So in Windows Boot Manager, you don't have to go for flash drive or whatever. You can directly boot into the recovery from Veeam if you want to restore a server.
00:34:47
Speaker
In this case, what was really beneficial, it was the tool to give me access to the file system of the domain controllers to delete the faulty signature file. and then just reboot the server and we're back in business. Now, we could have totally done that without Veeam in there.
00:35:02
Speaker
It would just have prolonged because, again, when it when it rains, it pours. You need to find, okay, I need to do an emergency recovery boot from a domain controller. shoot, I haven't done that like in years or never. I need a flash drive. Now I need to prepare a flash drive, right?
00:35:17
Speaker
No one can log in. because ah So that saved us probably an hour. It was really critical to get most systems at least respond. um And then we worked through other servers. The virtual servers were just easy.
00:35:33
Speaker
um We had some physical workloads in the branches that were down where I to deploy people to go to the branches like many other shops as well. There was just no way around it.
00:35:44
Speaker
um And we have, our fleet is all laptops, all the BitLocker encrypted. So that was, like many other shops for tier one, that was a pretty intense couple of days.
00:35:55
Speaker
Walken uses through BitLocker recovery key. um Do not recommend zero stars, but at the end, we just worked through it. It really didn't affect us. I think we were back pretty much 90% like in four or five hours where we were operational, where members could be served, right? They could make transactions.
00:36:10
Speaker
um he They could print your debit card in a branch if you need a new one. um It was all manageable. The only thing what I really felt was handled poorly by CrowdStrike was the first hours of communication where pretty much was like, okay, we made a mistake. we We fixed it.
00:36:31
Speaker
What you're all crying about, right? I mean, they retracted that. that's Not exactly that wording, but it was kind of what you got from them. Like, yeah, what are you complaining about? We already pulled that faulty update. You just need to remove it and reboot.
00:36:44
Speaker
um I guess it took them bit to realize that there are hundreds of thousands of machines that are not easily physically accessible, or even if they are, you need to go there. um So there was, I think, a bit of poor communication on CrowdStrike's side in the beginning. And then, of course, they went into full damage control mode, lawsuits and whatnot. But it happens.
00:37:05
Speaker
Will it happen again? Probably something like this. But it just shows you that one faulty update really shut down IT all over the globe, right? No flights, restaurants, banks, whatever was affected.
00:37:19
Speaker
So one could argue it's almost a monopoly CrowdStrike has. And no one outside of IT he knows about it because CrowdStrike is not it's only for professionals. You cannot buy CrowdStrike as an end user. It doesn't exist for you.
00:37:33
Speaker
So even my wife was like, so does CrowdStrike thing. And as you always know when your spouse who's not in IT or friends or your parents see something on ABC evening news that you work with, you know that's bad.
00:37:47
Speaker
Oh yeah, no, absolutely. yeah So you it's interesting, that's a ah great kind of highlight that I pulled out of what you just said was about integrating into your boot manager, your ah agent ISO, if you will.
Integrating Veeam Agent Recovery into Windows Boot Manager
00:38:03
Speaker
um I know you're a blogging man. Any chance you've ever written a blog on something like that? I have. Yeah, actually, and I did this many years ago, so I have on my blog post, like, how you do it. It's it's super simple.
00:38:14
Speaker
It really is. it's And it's almost old school, right? Because most, we talked loud and loud, and it's like, well, you need to do this in your Windows Server boot manager. um really the opposite to cloud-based stuff. But yes, there's an article out there. I actually won a blog post on this last year for Wien, because it was fresh up, right? We talked, so yeah, Wien can help that. of course, Wien was like, well, that's great. um So yeah, it's this on my blog, nicostein.com.
00:38:40
Speaker
You'll find it there. It's probably, it takes you 10 minutes to set it up. um The only thing, and I had this as a gotcha, When you set up the mood manager, make sure default it defaulted to the Windows Server OS and not Veeam, because ask me how I know.
00:38:56
Speaker
False. Server is not a... ah I wonder if you could do maybe similar for Linux, like add it to Grub and maybe have the same functionality.
00:39:12
Speaker
Although I don't know the use of physical Linux, who short of a hypervisor, I don't know that you see Linux servers quite as often as you do Windows ones in those instances. But yeah, I mean, we were hit by CrowdStrike pretty hard too. I think we had most of our systems up within a day, but there was still some lag behind, especially just as large as cyber fortresses and the global presence we have.
00:39:32
Speaker
We don't necessarily have tech ops engineers in every city where we have a data center anymore, especially, you know, as we set up data centers and then we've had people leave the company and we've had new employees come in, you know, we found ourselves in some cases having to rely on remote hands to get the main systems back up before we could get to everything else. And we apparently weren't the only ones because in some cases we were looking at 12 to 24 hour backlogs before remote hands could get to us because they were so busy dealing with other companies also impacted by the outage.
00:40:03
Speaker
It's interesting. You just said that what can you do with Linux and can we edit to grub menus and we don't see Linux so much? Well, we have three object first aces here. And so that means we tout object first for Veeam, which is a great product.
Veeam's Linux Hardened Repository vs. Object First: A Cost-Benefit Analysis
00:40:17
Speaker
But I know in my own world as a professional services company, we often introduce the interesting Linux hardened repository. And I say interesting because I love to hate it. It's a great solution. It works well when you have Linux people and we are Linux people. So great.
00:40:36
Speaker
If you want turnkey, we're talking object first. So in my own kind of sales pitch, um object first works for these reasons. You won't be tied to us forever. Many of our customers are, well, we are tied to you and let's go ahead and introduce the Linux world. So I love it. It works. It's great. But um I kind of want to draw swords here. Friendly Fire. So this is a Veeam Linux Hardin repository. It's a Linux product versus Object First. So I'm going to spin it through the channels here. And Jonah, I'm going to pitch it over to you first.
00:41:10
Speaker
What are your thoughts? I mean, i mean I've used both. Mostly I've used Object First as an ace as the virtual appliance, you know not as the hardware. you know it's a It's a very nice product.
00:41:22
Speaker
you know They do some great things with it, and they have the capability to do further great things as they iterate on it. My biggest problem that I have found with it as someone who's also recently partnered with them to potentially begin selling the appliances to some of our customers is the cost economics of it, especially as you're talking that kind of medium-sized business SMB, MSP market.
00:41:46
Speaker
they They need a lot less storage than what Object First offers right now as their base tier. And just the cost economics don't make sense. On the flip side, I would never deploy a hardened Linux repository if I could get away with it. Just they're a nightmare to maintain. And even more so, like, don't even get me started on if you ever change your software suite. Like, God forbid you remove the root user and you decide, hey, we want to add ConnectWise and start doing monitoring.
00:42:13
Speaker
Good luck getting that application installed without having to basically reinstall and redo the OS from scratch and potentially if you're not careful having to wipe all your backups and restart the chain as well too.
00:42:26
Speaker
Yeah, no, that's definitely scary. I love it because it's hardened. I get scared of it because it's hardened. We've had customers that were like, let's lock into change management. Let's say this is what it's going to be where.
00:42:38
Speaker
And then before we knew it's like, hey, by the way, we just want to change this address or we want to do this or we want to do that. And now we're looking at all the fingerprinting and changing and hoping and nail biting does it go through and the We just came across one this week, which interestingly enough on my side is showing a time drift. So there's a local Linux hardened repository and it's trying to write to S3 storage remote and it's complaining about time. And interesting, Veeam brought up and said, well, you should make the Veeam server the NTP server. And Now I have to go dig through the Linux Harden Repository documentation and say, well, wait a minute here.
00:43:13
Speaker
You all told us in a certain step what to do with the NTP server. Now you're telling me to use the Veeam server. If I can spoof the NTP server, I essentially can change the timestamps, which defeats the whole Linux Harden Repository. So that's something that I'm going to raise the flag on. I love Veeam.
00:43:29
Speaker
I love that this is a product. I love Linux. Been a Linux guy forever. However, When you're interfacing in with support on your own, I think you're going to find that as a low adoption rate.
00:43:40
Speaker
Veeam support may give you bad advice just because it's not widely used. I think when you're an object first in my world, I'm not really worried about it. They eat, sleep, and breathe Veeam storage.
00:43:52
Speaker
That really says something. Nico, where are you on the spectrum with this? Yeah, 100% agree to everything it was said. So I'm running Linux heart and repository as well in the home lab. It's a fun project.
00:44:03
Speaker
But also, I don't care if i accidentally wipe out my backups. right It doesn't matter. I just start over from scratch again. um ah think it's a great product. I think it's fun to see how it kind of went out, came out of someone in Veeam, and the home lab that out, and really put some love and energy in it. And now it's a product.
00:44:24
Speaker
And it's okay for the home lab. We have a physical um object first box in our data center right now.
00:44:33
Speaker
I'm a journalist. I would never deploy a Linux hotend repository at my work. And it's not that I don't trust It's, yes, I know my way around Linux, but I'm not a Linux guru, right? there I'm not.
00:44:45
Speaker
We don't have, in my team, anyone who's like, sleep and breathe Linux all day long. So to joke if if there's a change needs to be made, it's almost work on raw eggs, right? So I think I understand what that change is.
00:44:58
Speaker
But if I'm wrong, it's all gone. With object first, it's I don't care, it's not my problem. you know It just works. i cannot I cannot make many changes to it, but I don't have to really train people on it.
00:45:09
Speaker
Once it's set up, it's really a set and forget. I'm into Jonas' point from being a small, medium-sized business. this It is pricey for what it is. And the sizing, the box we have is a 64TB box. We don't need 64TB.
00:45:25
Speaker
I mean, and yeah I just don't. So I know Object First is working on this. They have more suitable solutions for smaller businesses and therefore bring the price down. um On the other hand, I just got some quotes for some Dell storage and not to throw shade on Dell storage, but I mean...
00:45:42
Speaker
It was a lot smaller storage for a good deal more. And that was not hard, and that was just a regular storage for Wim. So we could talk about this also. like Maybe Object First is not that expensive what you get at the end.
00:45:55
Speaker
But for us we're looking forward to having this box replaced with some that fits our needs more, and that's probably the 32, 40 terabyte storage, which I think will serve best for a lot of small to medium-sized businesses.
00:46:08
Speaker
But it solves their problem, right? They don't have the knowledge. They need immutable backups. They don't necessarily want to push that to a Wasabi because they also might not have the download speeds they need. It's all great if I'm in the cloud, right?
00:46:20
Speaker
If it takes me 20 files just to download it the backups to restore them, well, my RTOs are gone. um So they want to have it on site, they have full access here and can transfer it quickly, but they don't want to have the headache with Linux or other solutions that seem, I'm not going to half-baked, but require a lot of manpower. right it's It's the Linux world. Linux for free, you give it up with manpower in hours.
00:46:46
Speaker
You do. And that's exactly right. So the customers that we're moving in, you're right. We price out object first. You hit the nail on the head. We looked at Dell. Dell, I think, was double the price for half the storage by the time we were done. Not throwing shade, but that's where they were.
00:47:02
Speaker
And object first was viable. um If you're large enough where you have this kind of large SMB or enterprise, I think you're used to those price points. And I think it's religion at this point. If I'm a Dell guy, I'm a Dell guy. If I'm a NetApp guy, I'm a NetApp guy. I don't have to learn it. I can use it.
00:47:20
Speaker
Object first is scary for them because they're like, i have to learn something new. Not realizing with the whole Ooppy thing, you're in and done in 10, 20 minutes. Like that's it. yeah It just works like that.
00:47:32
Speaker
because it's so tightly hooked. But then it's this fear factor of we'll support contracts. And so we have to start changing their religion. When we move into Linux, I mean, something I want to bring up with this, and we have some really strong Linux talent on our team.
00:47:46
Speaker
um i you know, I think I'm an expert, but sometimes I think I only play one on TV, so to speak, with Linux, because it's been a good number of years, but really need to drive home I'm sorry to say Linux Hardened Repository is very hardened.
00:48:00
Speaker
However, we have been able to easily circumvent and delete things that should not be deletable because we know our way in and out of Linux, even if we're removing accounts and doing this and locking down grub menus and whatnot.
00:48:13
Speaker
And we're not even talking sideloading. There are ways in which it can be done. So it's kind of, as I put it, it's kind of that whole like governance versus compliance mode when we're talking into S3.
00:48:23
Speaker
Do I want to be able to say super god admin can delete something versus um no, nobody can. I really laying it back more towards the whole listen, if you're in Linux, super god guy can delete it.
00:48:39
Speaker
Just be prepared. And honestly, i don't know that that's a bad thing because using the whole Veeam zip code, that's not going to be our only location. I don't care if my customer has two debtor centers or 50. I'm always back up, cross pollinate and get yourself tertiary outside of your environment. I don't care.
00:48:59
Speaker
where you choose that S3 bucket and for what, because you just need to have that last thing to what you were talking about with those RPOs. You're so right. Like people that stuff it to S3 and they go, Oh, just download it. That is just baloney. And that's all I can say.
00:49:13
Speaker
the The reality is, is these people need to realize that your S3 storage exists for the day that essentially you're out of business. It's not a main backup thing is,
00:49:25
Speaker
Everything you have ever leveraged has gone to hell. And you're now in this meeting location of it's going to be down for a week. Oh, my God, we'd be out of business.
00:49:36
Speaker
Yeah, well, at least you retained some of your business. You can still do some things. That's why it exists. It is definitely. not what people should be doing. So that's kind of where I am. And yeah, now you're talking patching your upkeep, your firmware, your hardware, all of these things.
00:49:54
Speaker
um Honestly, in our world, believe it or not, we don't use the Veeam ISOs for Linux hard-end repository. Like, I get it's GUI, I get it works, but we'll get in there and we'll make sure that we're running LVM on those. So that way that we have more resilience within the Linux capabilities of itself and we're building it out. Now we're still following the best practice hardening and NTP servers and you know, syncing up to all the SECOP rules and everything like that.
00:50:21
Speaker
So now you're paying us a larger implementation fee. You know, it could be 10, 15 hours, depending on the box. Well, you could be looking at, you know, $3,000 to $5,000 and manpower up.
00:50:32
Speaker
I think it's a solid solution. But if I just built this for a customer and said, here you go, in a year, they're going to have to call back. So they'll substantially save on a Dell purchase that could be $90,000, an object first that's maybe $60,000. I'm roughing this. You know, ah Linux hard-in repository, you might go, well, that's only $20,000. That's great.
00:50:55
Speaker
Yeah, but now you're it's like the management costs. And what happens when, Nico, you know, you leave the bank and you knew how to do this and the next guy doesn't or they lose the vendor contact?
00:51:06
Speaker
suddenly you own a boat anchor. So I put my customers into them, but we're in regular relationship with different dev. It's not like we, some of the service providers or service partners, the vast, they they jump in, they implement you, they go.
00:51:20
Speaker
We're kind of involved every day and we're like fractional Veeam people. So we're in their change management. It's like we're on their staff. It makes sense. But for the most part, I'm leaning more and more towards object first. What I hope to get out of object first is exactly what you said.
00:51:35
Speaker
Smaller footprint, guys, a little more aggressive price points for those people. And I think we're going to see it kind of come through. So those doctor's offices, those ah medical centers, those dentists, those accounting firms that have 15 headcounts,
00:51:52
Speaker
and need something, it'd be great to give them the appliance because where they right now? God's honest truth. You know, they're doing data is kind of where they are, you know, and that's just, not a product anybody should ever be on.
00:52:03
Speaker
um I will throw shade at them. So um I think we've kind of traversed the world. What I'd like to do is kind of last thoughts for everybody. um We'll throw it over to you, Nico, our guest. with Last thoughts on anything, wrap-ups, comments, conclusions.
00:52:19
Speaker
Where do we find your blog again? Let us know. Those sorts of things. All right. Well, we covered a lot here today. um So first of my blog is easy to find. It's really my first and last thing together. So nicostein.com.
00:52:33
Speaker
um You find me so on LinkedIn, little bit on Twitter or X, Blue Sky. It's linked on my blog. So if you want to connect there, my DMs are open. I like to chat, like to talk shop all the time.
00:52:46
Speaker
um I mean, a wrap-up since we talked about Udbe and its heart and repository, I know it sounds like a broken record, but have immutable backups, people, please.
00:52:58
Speaker
Even if it's still in its heart and repository for free, it's better than nothing because, yes, it could be your last hail mally for your business. If that doesn't work and there have been business out there, they never came back.
00:53:11
Speaker
And, yes, you will get hit. That's what I told my board when I talked to him about cybersecurity and I said, Nico, what are you doing to prevent us from getting ransomware? There's nothing I can do to prevent us.
00:53:23
Speaker
What I'm really working on is how fast can we come back?
Immutable Backups: A Business Continuity Essential
00:53:27
Speaker
Because no matter what I do, it's it's a numbers game. Eventually, it will hit through. So yeah, be ready to come back when you get hit.
00:53:35
Speaker
Good deal. Yeah, I like that. It's kind of like, I think if it's like picking a lock, and it's so random, too. Like, people are just thinking that they have woken up this morning, they looked at Nico and said, we're going to get him, and it's just it's just blasted. It's however you get it. It's randomized. it They're not targeting people. Forget what you watch in Mission Impossible and whatnot. It's just...
00:53:56
Speaker
you know, randomness. So Jonah, you've been on both sides of the cybersecurity world for a long time here. I'm sure as a so service provider, you're probably looking at more ransomware recoveries than power outages these days. So what's your wrapping thoughts on everything we've talked today?
00:54:12
Speaker
Yeah, I mean, i We are good not just for ransomware. I mean, we do have the occasion, you know, hurricane hits Louisiana. Someone needs a location to spin up, but you're right. it's simply It's increasingly ransomware. And the conversation I find myself having on almost every sales engagement, being one of the main engineering resources for the sales team at my company is You know, I tell potential customers and existing customers, you know, the backup zip code says one immutable copy, but really it should be every copy that you can manage should be immutable, especially as you get closer and closer to your source production. Like in a perfect world, you'd have your local immutable backups, you'd get ransomed unless there's some reason like, you know,
00:54:56
Speaker
your insurance company or cybersecurity comes in and takes everything off and says, you can't touch any of this, we need to do forensic analysis. You should be able to take those local backups and spin up as quickly and easily as possible. And they shouldn't be affected. In a perfect world as a service provider, we're the people that you pay and hopefully never have to call because if people have...
00:55:17
Speaker
you know, on a couple of our customers have basically said, kind of like we alluded to with s three if they're calling us, it's typically a resume generating event. Like it won't be them calling us, it'll be their replacements calling us.
00:55:28
Speaker
It's very true. I think of it like insurance sales. A friend of mine was an insurance. And if I would walk in and say, this is what you should do for your DR solution, somebody goes, no, our budget's this number. And you're like, no, you really should do these things. They're like, no.
00:55:43
Speaker
You can tell that that person has never gone through an incident. Whereas when you deal with other companies or post, suddenly they have copious amounts of money to direct to this because They've gone through the prank pain point. And it's sad because how many people would go and put insurance on somebody, a death or tragic thing happens.
00:56:02
Speaker
They go to cash a policy and realize it's not as much as what they thought. And if they had a time machine, they'd go back and say, well, instead of like $50 a month, I would have given them $100 and I'd never have to deal with this.
00:56:13
Speaker
And I think that's the sad thing with DR is we have to sit there and go. I know I'm the guy that benefits from selling you this, at least on my side, professional services coverage, but I can't tell you we're going to save your business. And there's not, you can't ever look at it and go, i prepared enough and I'm done.
00:56:30
Speaker
There's always more preparation. And I will say this with your DR vendors, if you're using them, ah some of my customers after mine, We live in the world of doom and gloom. We are paid pessimists.
00:56:41
Speaker
We are telling you everything you've designed is terrible. We need to do more. And a lot of IT people kind of scoff and they're like, I have multi-layer snapshots and this and that and whatever.
00:56:53
Speaker
Well, if I've said there's another path, That means you take that path because that other path can die. That's the whole thing. You can never have too many backups. That's really what the zip code of Veeam is.
00:57:06
Speaker
That was the origin was multiple copies. And where did we learn this? Us dinosaurs that were like, take floppy disk here to here and here and here and copy and go.
00:57:17
Speaker
And you go on these hard drives, you find all these copies. I mean, it was disjointed. It wasn't healthy, but it really did work if you had something stashed somewhere and That's what we're doing is we're stashing backups and protecting them and walling them. So absolutely, like we need to get on it. So I loved our podcast today. Nico, thank you for joining us. It was great having you on the show.
00:57:39
Speaker
um Jonah is always great ah for our listeners. Check out Nikostein.com. Check out CyberFortress.com for all your cloud service needs. They'll go ahead and give you that tertiary or secondary site for a low cost and you'll get an amazing team of IT ninjas helping you out.
00:57:56
Speaker
If you have any questions regarding how do I get Veeam? How do I buy Veeam? How do I implement Veeam? More importantly, how do I run it month over month and not hire a very, very expensive engineer?
00:58:08
Speaker
Contact DifferentDev. We'll help you out for a fraction of the cost and monitor you. We don't care if you're at CyberFortress or anywhere else. We actually integrate in with everybody. And... ah I'll close off with Nico's kind of thoughts on audits. They're a thing. So be auditing your DR. Go ahead. Give us a call for that as well. But gentlemen, thank you so much for attending today. Listeners, we're going to play you out with our cool little music and whatnot. So um as always, stay in trouble, my friends.
00:58:42
Speaker
or just starting out, the Rebel Devs podcast is a must listen. So join us, comment, share, and subscribe. Be part of the Rebel Devs.