Stop Using VSS as a Backup Before Ransomware Deletes Your Shadow Copies

Stop Using VSS as a Backup Before Ransomware Deletes Your Shadow Copies
Ransomware deletes shadow copies using your own built-in Windows tools against you — and if VSS was your backup plan, you just found out the hard way that it wasn't. In this episode, W. Curtis Preston (Mr. Backup), Prasanna Malaiyandi, and Dr. Mike Saylor break down exactly what shadow copies are, why they don't qualify as a real backup, and how attackers are weaponizing vssadmin to wipe your recovery options before you even know you're under attack.
If you've got Windows systems and you've been thinking "eh, we've got shadow copies," this episode is for you. We cover the history of VSS — what it was actually designed for, why it became a crutch, and why using it as your primary backup strategy is a bad idea on multiple levels. Performance, the 3-2-1 rule, and the fact that one attacker with admin rights can delete every single copy in seconds. We also get into the living off the land angle: how attackers do recon on your shadow copies, how they use them to scope out valuable data before going full ransomware, and what you can actually do to detect and respond to this behavior using EDR tools.
The bottom line: VSS is a great tool. It was just never meant to be your backup. Get a real one.
Chapters:
0:00 — Intro
1:39 — Welcome & Book Talk
3:26 — What Are Shadow Copies and Why Do People Use Them as Backups?
9:14 — Performance Problems with VSS as a Backup
10:19 — Living Off the Land: How Ransomware Uses VSS Against You
12:36 — Can You Monitor or Lock Down VSS Admin?
14:26 — Why Shadow Copies Fail the 3-2-1 Rule (They're Not a Backup)
18:01 — How to Protect Yourself: Configuring Your EDR
21:31 — The Local Admin Problem and Security Culture
27:00 — Virtualization, Snapshots, and Shadow Copies
29:00 — Final Thoughts: Just Don't Do That
You found the backup wrap up your go-to podcast for all things
Speaker:backup recovery and cyber recovery.
Speaker:In this episode, we're talking about something that makes me
Speaker:want to flip tables upside down.
Speaker:People using Windows shadow copies or VSS as their actual backup system, and
Speaker:then ransomware comes along and then deletes those shadow copies using your
Speaker:own built-in Windows tools against you.
Speaker:And suddenly you have nothing.
Speaker:I'm joined again by Dr. Mike Sailor, our intrepid cybersecurity
Speaker:expert and persona ndi of course.
Speaker:And we break down exactly why shadow copies are not a backup,
Speaker:how attackers are pulling off this move, and what you can do to protect
Speaker:yourself if you've been counting on VSS to save you when things go wrong.
Speaker:This is your episode.
Speaker:Uh, just a quick note about me.
Speaker:I'm w Curtis Preston, AKA Mr. Backup, and I've been obsessing about backup recovery
Speaker:and now cyber recovery for over 30 years.
Speaker:If that's your bag, then I'm your guy.
Speaker:Well, you're not gonna find anybody that cares about it more than me.
Speaker:Ever since 1993 when I had to tell my boss there were no backups of
Speaker:the database that we had just lost.
Speaker:Now I've written five books, a blog, and a podcast.
Speaker:And here we turn, unappreciated Admins and the Cyber Recovery Heroes.
Speaker:This is the backup wrap up.
Speaker:Welcome to the backup wrap up.
Speaker:I'm your host, W. Curtis Preston, AKA, and Mr. Backup.
Speaker:And I have with me a guy who couldn't bother to on time.
Speaker:Prasanna Malaiyandi, how's it going?
Speaker:Prasanna.
Speaker:You've heard about Indian standard time, right?
Speaker:I am not making any comments about anybody's stuff based on
Speaker:where they may or may not from
Speaker:so I do have an excuse because I had to change into my shirt,
Speaker:which Curtis, I see that.
Speaker:shirt.
Speaker:I should get one of those.
Speaker:Yeah.
Speaker:I think you have to just find yours is more what the.
Speaker:to God, I have turned this house upside down looking for that shirt that I
Speaker:paid good money for, I cannot find it.
Speaker:anyway, including yesterday's event where I literally took
Speaker:everything out of my closet.
Speaker:have no excuse, but maybe it's being used as a cleaning rug somewhere.
Speaker:Anyway.
Speaker:Of course have our intrepid cybersecurity expert, Dr. Mike Saylor.
Speaker:How's it going?
Speaker:Mike,
Speaker:It is going well guys.
Speaker:Thanks for having me.
Speaker:have you still have you yet?
Speaker:I have not.
Speaker:Have you seen our book yet?
Speaker:I have not.
Speaker:It's killing me.
Speaker:So we're of course referring to this book up here on my video, which if you're
Speaker:not watching us on YouTube, feel free.
Speaker:the same, this channel by the same name, the backup wrap up, and,
Speaker:learning ransomware response and recovery, available now, at, all the
Speaker:places that you buy books, although
Speaker:is.
Speaker:the.
Speaker:All the places, if you prefer Rainforest in South America, or
Speaker:if you prefer buying them directly from O'Reilly, or Barnes and Noble.
Speaker:I was in a Barnes and Noble yesterday.
Speaker:apparently it's a building in which you can buy books.
Speaker:It's a fascinating.
Speaker:Fascinating idea.
Speaker:maybe that'll take off.
Speaker:Anyway, so today, we're talking about, something a little bit up my alley.
Speaker:And I'm gonna just start out, imagine, if you will, you have developed a backup.
Speaker:I'm gonna make quotes in here, a backup system, because I don't
Speaker:think this is a backup system.
Speaker:You have developed a backup system around something called shadow copies,
Speaker:which is a ver a feature in Windows that we're gonna talk about in a minute.
Speaker:And then you just come in and you've heard that something bad has happened
Speaker:from some sort of cyber attack perspective, and then you find out
Speaker:that, you know that those previous shadow copies are not available to you.
Speaker:I like the, this reminds me of, for those of you that have been around a minute,
Speaker:this is the ultimate sweep, the leg move, that is of course a karate kid.
Speaker:Reference.
Speaker:and this also refers back to living off the Land, when we talked about living
Speaker:off the land a few episodes ago where they use your own tools against you.
Speaker:In this case, the tool would be VSS, admin, and they just delete this thing
Speaker:upon which you are counting for recovery.
Speaker:this sounds like a bad day.
Speaker:What do you think, Mike?
Speaker:I agree.
Speaker:Yeah.
Speaker:I think one of the problems, that people have is they become complacent.
Speaker:And I think that's, it's not just with your backups.
Speaker:I think in general with security.
Speaker:but for sure, whoever set up your backup strategy, whether that was you
Speaker:or your grandkid, or a volunteer or even someone been in it for a while,
Speaker:shadow copies have been around.
Speaker:and they come with, they come with the batteries, but they're, to, to
Speaker:your point, they're not, they're not a legit backup solution.
Speaker:Yeah,
Speaker:A VSS volume, shadow services, AKA, shadow copies, and the, is a service
Speaker:within Windows where you're able to create a stable, consistent, image
Speaker:through which you can view the platform.
Speaker:Let me go back in time.
Speaker:There was a time when, VSS did not exist, and when you went to go back up.
Speaker:Windows.
Speaker:the problem is you're backing up.
Speaker:When you have a large system, you're gonna, part of the backups
Speaker:are gonna be from this point in time, and part of your backups are
Speaker:gonna be from that point in time.
Speaker:And if the thing that you're backing up needs what we call referential
Speaker:integrity, meaning integrity between the different points of the system.
Speaker:You're not gonna have ref referential integrity and
Speaker:you're gonna have a bad backup.
Speaker:And there used to be a product, I don't know.
Speaker:person, do you remember St.
Speaker:Bernard?
Speaker:Is that before your time?
Speaker:That's.
Speaker:Or my time.
Speaker:Okay.
Speaker:Mike, do you remember St. Bernard.
Speaker:I don't.
Speaker:Man, this is okay.
Speaker:By the way, St. Bernard was what you bought something before VSS existed.
Speaker:And for the record, St. Bernard was a San Diego company.
Speaker:and, it came from the idea of the, that St. Bernard's, the image of St.
Speaker:Bernard's with a thing of whiskey around the, they came to save you.
Speaker:So that's where the Saint Bernard name came from.
Speaker:The, so VSS was this method through which you could create a stable
Speaker:image, and do various things with it.
Speaker:So Prasanna, why don't you talk about like the first, what,
Speaker:VSS was typically used for.
Speaker:Yeah, Because one of the things that you need to do with a Windows system,
Speaker:or if you're running say Microsoft SQL or Exchange is do your backup.
Speaker:And especially on physical machines, before we had
Speaker:virtualization and everything else.
Speaker:The only way to actually get a stable point from which to
Speaker:do your backup was to use VSS.
Speaker:And so you'd have the backup apps call into and VSS.
Speaker:It's a framework that Microsoft provides.
Speaker:So you'd call in, you'd say, Hey, I wanna do a backup of this particular app.
Speaker:Microsoft would then.
Speaker:quiesce
Speaker:the application.
Speaker:Take a volume snapshot, right?
Speaker:Shadow snapshot.
Speaker:And then it would unes, so then you can continue doing your stuff, right?
Speaker:And then what the backup app would do is it would then read from
Speaker:that shadow snapshot in order to be able to do the actual backups.
Speaker:And so this is what it was useful for, and it was more of
Speaker:like a software snap snapshot implementation, pre VMware snapshots.
Speaker:but yes, it was their mechanism to get a consistent point for which
Speaker:you could do your backups from.
Speaker:And there, like you said, it was a framework, right?
Speaker:So each application that wanted to be supported by VSS, they wrote
Speaker:what was called a VSS writer, and then VSS would, and then they just
Speaker:needed to plug into that world.
Speaker:And like you said, a backup app would just say, Hey, I'm here to do a backup.
Speaker:It would talk to V-S-S-V-S-S would talk to the writers.
Speaker:Everybody would do the right thing that worked for their application.
Speaker:Then we would take the snapshot.
Speaker:it's great for that, right?
Speaker:It is the easiest way to get a backup of SQL Server or Exchange or any, or
Speaker:even Oracle running on top of Windows.
Speaker:Just use the VSS infrastructure.
Speaker:but some people, Mike, have you seen this in the while where people use
Speaker:VSS, like as a backup mechanism?
Speaker:Have you ever actually seen that you have.
Speaker:I have.
Speaker:Yep.
Speaker:You're killing me.
Speaker:What?
Speaker:any you?
Speaker:Why?
Speaker:Why?
Speaker:Not that he's killing me, but,
Speaker:No?
Speaker:oh, you're saying why?
Speaker:Why would they
Speaker:Yeah.
Speaker:Yeah.
Speaker:And what was the purpose?
Speaker:Bike.
Speaker:Could you elaborate a little?
Speaker:I think it's just in the absence of an educated solution.
Speaker:so they, they tried to understand the manual or did a Google search or, saw the
Speaker:word, Or the idea that it was a backup of some sort, and they just went with that,
Speaker:Because you could create multiple shadow copies over time, right?
Speaker:and you could hold onto them.
Speaker:These are,
Speaker:a problem.
Speaker:which is also a problem.
Speaker:It's a problem for multiple reasons, right?
Speaker:Prasanna, I'm pretty sure these are copy on write.
Speaker:Snapshots, right?
Speaker:yes.
Speaker:Yeah, For those that don't know that there are multiple reasons why using
Speaker:VSS as a backup app are a bad idea.
Speaker:We're gonna primarily talk about one of them in the episode, but one of the
Speaker:other ones is if you've got 30 days of VSS snapshots, your performance is in the
Speaker:crapper, because of the way copy on write.
Speaker:Snapshots work.
Speaker:I'll see if I can find them.
Speaker:We have a, we have an episode where we just.
Speaker:Delve into, copy on write, snapshots and what the differences between
Speaker:them and redirect on write snapshots.
Speaker:but because of the way VSS works underneath you are going to significantly
Speaker:hurt the performance of your system if you're using it as a backup app.
Speaker:go ahead, Prasanna.
Speaker:one thing I do wanna.
Speaker:Clarify is my knowledge of VSS is slightly dated.
Speaker:They may have improved things over the years, but
Speaker:I doubt that they changed fundamental architecture from copy
Speaker:on write to redirect on write.
Speaker:You agree?
Speaker:I would agree.
Speaker:I, think it's highly unlikely, but today.
Speaker:We're talking about the cyber aspects of this.
Speaker:so Mike, why don't you talk, let me just look at the outline we have here.
Speaker:Mike, you wanna, you want to, just give a, again, a brief overview of
Speaker:the concept of living off the land and what, how that A applies here.
Speaker:Sure, yeah.
Speaker:Living off the land is a bad guy's strategy of utilizing data,
Speaker:applications and trust, already associated with a compromised host.
Speaker:So if I. I'm able to get access to any degree, to a laptop or a server,
Speaker:what things are already there that I can utilize to further my attack,
Speaker:whether it's escalating my privileges or collecting data, or moving from
Speaker:this computer to another computer.
Speaker:those are the types of living off the land.
Speaker:techniques or tactics a bad guy would use.
Speaker:So one of the first things that they will do, and this is the first phase in
Speaker:just about any attack, is reconnaissance.
Speaker:So I've got access to something.
Speaker:What is it and what do I have access to?
Speaker:what's here?
Speaker:What tools do I have?
Speaker:one of the things that they will look for are shadow
Speaker:copies, for a couple of reasons.
Speaker:One is that potentially leverage?
Speaker:for extortion or something else.
Speaker:and two, can I collect that data and potentially use that as information,
Speaker:to further understand the target?
Speaker:maybe it's, I've found out that this shadow copy is of, QuickBooks.
Speaker:so now there might be some financial information there, or, maybe there's
Speaker:a company name or a project name.
Speaker:something that's gonna make, the light bulb go off and help me determine
Speaker:if this is a valuable target.
Speaker:and then whatever my next steps might be.
Speaker:Go ahead.
Speaker:When you're talking about that, I'm actually not sure what's done today, but
Speaker:when an application is reading from a shadow copy, does that get logged in all
Speaker:the other normal processes or is this sort of something that's hidden away and so
Speaker:the normal mechanisms don't necessarily apply and so it might be harder to figure
Speaker:out if someone is exfiltrating data.
Speaker:No, you can actually monitor for, not only the store, access to where the
Speaker:files, the data is, but you can also, monitor for the processes that would.
Speaker:interact with the shadow copies, the, vs admin as an example.
Speaker:so yeah, those are all things that you can write detections for.
Speaker:and whether that's like anti-malware on the device or, much broader
Speaker:like network monitoring, cybersecurity monitoring platform.
Speaker:Yeah.
Speaker:the challenge though, and again, this goes back to the, to the, the nature of living
Speaker:off the land is that the tools in question are commonly used by other tools, right?
Speaker:VSS admin and w and WMIC, which is a, the Windows management,
Speaker:interface, on the command line.
Speaker:that's one of the challenges of the, you can monitor for it, but you might
Speaker:end up getting a lot of false positives.
Speaker:the, sorry.
Speaker:just looking at here.
Speaker:Ask another question while you're looking.
Speaker:Mike, sorry, I'm, I've never actually dug into, is it vs.
Speaker:Admin or VSS Admin, by the way.
Speaker:Two S's.
Speaker:Okay.
Speaker:For VSS admin, I've never dug into it.
Speaker:Can you access control VSS admin?
Speaker:I know you talked about you can monitor it for what files are being accessed, what
Speaker:processes can you also lock down who has access to it from a process perspective,
Speaker:or is that not really possible?
Speaker:That's a good question.
Speaker:I know you have to have a, you have to be a local admin.
Speaker:but I don't know if.
Speaker:a scripted like PowerShell, could do it, remotely or, that's a good question,
Speaker:Okay,
Speaker:but
Speaker:but Okay.
Speaker:But yeah, once you have local admin, then all bets are off right at that point.
Speaker:Yep.
Speaker:So there are ways to use, again, just so we're not anti snapshot, I
Speaker:am definitely anti any kind of backup system that is easily deletable.
Speaker:And this is one of them.
Speaker:There are ways to use snapshots and there are ways to use snapshots,
Speaker:perhaps storage array, snapshots, for example, that can be done in such
Speaker:a way that they're not deletable.
Speaker:basically if you get it to a place, if you get it to a version of the
Speaker:system that is, that is where you have worm turned on, that would be one.
Speaker:Write once read many, right?
Speaker:There are, again, not anti snapshot.
Speaker:Just this particular snapshot being used as a backup method.
Speaker:go ahead.
Speaker:I wanna challenge you is based on your definition, Curtis, of what we've been
Speaker:talking about for the last six years, is that shadow copy, even a backup,
Speaker:It's not,
Speaker:okay?
Speaker:that,
Speaker:Yeah.
Speaker:So what and why do you think it's, why do you think, I think it's not a backup.
Speaker:it doesn't follow your favorite rule in the world.
Speaker:it does not follow my favorite rule in the world.
Speaker:which of course would be the 3, 2, 1, which is now the 3, 2, 1, 1 0, but
Speaker:basically three copies of the data on two different media, one of which is offsite.
Speaker:Arguably the way I understand Peter Krogh, the person who originally coined
Speaker:this term, is the idea, he would not think of three versions as three copies.
Speaker:He would think of three actual copies, right?
Speaker:And then he would want, and those need to be on separate pieces of media.
Speaker:The two, the idea there is that you have it on different types of media.
Speaker:So that you have a disc drive and an SSD or an SSD and a tape or a disc, whatever.
Speaker:Cloud.
Speaker:that's what the two was meant to.
Speaker:Basically different risk profiles.
Speaker:And then the one definitely needs to be somewhere else offsite would be the term
Speaker:that we would typically be used, right?
Speaker:It's a difficult term in the cloud, but, but.
Speaker:The, yeah, you're right.
Speaker:this isn't any of the, this doesn't meet any of the three, parts of
Speaker:the definition of the 3, 2, 1 rule.
Speaker:So yeah, I would not consider this a backup, but based
Speaker:system can absolutely do that.
Speaker:yeah, so again, not anti snapshot, just anti using them like this.
Speaker:All right, Mike, you go ahead.
Speaker:What.
Speaker:But even if you are using a snapshot system, so I know as some of my
Speaker:past employers, they would integrate with the VSS framework to then
Speaker:trigger a hardware based snapshot because that was better performant.
Speaker:And then they would do their backup off of the hardware based snapshot.
Speaker:Yes,
Speaker:question though, is.
Speaker:I guess it depends how that snapshot is created.
Speaker:And is it possible to use the same VSS framework to manage those snapshots?
Speaker:Because I'm sure that those backup software has a retention
Speaker:life cycle that it manages.
Speaker:And so is there a risk that those snapshots could be also deleted if
Speaker:someone gains access to that Windows box?
Speaker:I don't think the way VSS admin work is it's designed to
Speaker:control third party snapshots.
Speaker:I think it's only the designed to work the other way.
Speaker:I could be wrong.
Speaker:but regardless of how off host snapshots are created, they need to
Speaker:eventually be somewhere that is worm.
Speaker:And if they're not, then I'm not a fan.
Speaker:and again, I'm not, it's not, I don't have a problem with VSS.
Speaker:I think VSS is a great tool.
Speaker:Use it for what it was designed for, which is to give you a stable,
Speaker:consistent image to create a backup from.
Speaker:so let's talk about.
Speaker:What people can do to, protect against this.
Speaker:And I, Mike, you've already alluded to this one, is this idea of,
Speaker:complaining, have your EDR tools look at.
Speaker:Things like vss, admin so let's talk about the first thing here,
Speaker:this idea of, give us a little bit more information on the configuring
Speaker:your EDR to trigger on this stuff.
Speaker:It's a process.
Speaker:And what I mean by that is, you've gotta your tools and your people to
Speaker:understand what's normal, so that.
Speaker:A normal process like this running in an abnormal fashion would
Speaker:be what you're detecting on.
Speaker:so with any good security tool or security service, security
Speaker:person, there's gonna be a lot of getting to know you the beginning.
Speaker:so you know, the VSS admin, kicking off.
Speaker:Twice a week, once a day, whatever the case is, those are gonna show
Speaker:up, as a privileged, activity.
Speaker:and so when we're getting to know our clients, we want to be very strict at
Speaker:first so that we can understand all of those things and then we start to
Speaker:flag those, or, build an understanding.
Speaker:we build a wiki, that says these things happen on this.
Speaker:on this frequency or in this manner.
Speaker:so that going forward as the, as we detect those things, we can determine
Speaker:if those are normal or not, or normal with some context like this will
Speaker:happen, but so will these other things.
Speaker:but if any of those normal things happen on their own, that in
Speaker:itself is an anomaly or abnormal.
Speaker:so creating a baseline and getting to know your, your environment is important.
Speaker:because again, living, living off the land would indicate that it's stuff that's
Speaker:already there and that should be okay.
Speaker:Mike, then, I know you've brought this up, in previous episodes as well, about
Speaker:okay, you will need to have someone come in, understand the environment, Kind of
Speaker:fine tune it for your particular use case.
Speaker:It's not like a. outta the box, everything is good to go.
Speaker:For you, how long does that process normally take?
Speaker:And I'm sure it's gonna vary significantly depending on the size of
Speaker:the organization, the number of devices, the applications, all the rest of that.
Speaker:But is this like something that'll take three years?
Speaker:It depends.
Speaker:Yeah.
Speaker:what's interesting about that is, identifying the deficiencies or
Speaker:opportunities for improvement, actually happens fairly quickly.
Speaker:It's the remediation and the follow up that tends to take time.
Speaker:as an example, and I'll preface all of this with the adoption
Speaker:of security in most companies.
Speaker:Has to be aligned with the culture or the compliance requirements.
Speaker:other words, if I'm a, if I'm a company that doesn't like security, but I
Speaker:want to do business in a regulated environment, I have to adopt security.
Speaker:how we adopt it, how we implement it is a little different.
Speaker:and may, it may take some time, but there are some companies out there,
Speaker:organizations that don't like security or their culture is security averse.
Speaker:We're too friendly to have security type of thing.
Speaker:or, it's keeping me from doing my job, so we're not gonna do it.
Speaker:there are a lot of, if not every organization has one or two or
Speaker:many, Very blatant, observable, deficiencies, even with just best
Speaker:practice, basic cyber hygiene.
Speaker:One of those is allowing users to be local admins on their computer.
Speaker:back in the day was very prevalent because it needed to be in order
Speaker:to, keep the overhead on already constrained IT support staff you have.
Speaker:let users do some of the basic stuff.
Speaker:I remember when one of the new versions of Windows came out and you had to be
Speaker:a local admin to change your wallpaper.
Speaker:that was ridiculous.
Speaker:And, but it was culturally people were causing a big stink because
Speaker:they couldn't customize their computers and they were unhappy.
Speaker:And so it got overburdened with these local ad, these,
Speaker:basic local admin requests.
Speaker:So everybody got local admin.
Speaker:today, we still see that to a large degree, people, users have local admin,
Speaker:just because, because they, there's an exception because they're an engineer or
Speaker:a designer and their tools require it.
Speaker:And there are better ways of going about that.
Speaker:Sure.
Speaker:Give them access to the local account, but don't let them run
Speaker:day to day as a local admin.
Speaker:run as administrator, not, Click on the thing and do run as administrator.
Speaker:don't log in as an administrator.
Speaker:So there's things, the more things you implement, the more burden
Speaker:you're introducing into people's workflows and how they do their job.
Speaker:And so the easy thing to do is just.
Speaker:Give them local admin.
Speaker:So there's that.
Speaker:but even in the, in those cases as a security, cybersecurity practitioner,
Speaker:so whether I'm teaching a tool to tell me what's weird or I'm monitoring an
Speaker:environment for what's weird, if I knew that about an organization, if I knew
Speaker:that somebody was a local admin, I could still, with that understanding detect
Speaker:anomalies, occurring as local admin.
Speaker:There are things that computers do, does bad guys do, that just
Speaker:are not normal user behavior.
Speaker:Yeah, the challenge I did a quick, quick look and it doesn't appear
Speaker:that with VSS admin that basically anyone with administrator rights can
Speaker:run it, and that I can't say, oh.
Speaker:Only Steve.
Speaker:Only Steve.
Speaker:Or only administrator.
Speaker:Only you know the base administrator of account.
Speaker:It doesn't appear that I'm able to restrict it by administrator, and so
Speaker:let's go back to your comment, Mike.
Speaker:I think we can agree that.
Speaker:The proper thing is to have an administrator account and not have
Speaker:everybody have admin privileges.
Speaker:But to go back to your earlier, thing, and by the way, this is the
Speaker:same in, in, in Unix world, right?
Speaker:Is that you have root and then you have, we, and you use SDU
Speaker:to, to run things as root.
Speaker:The same thing here, you have administrator.
Speaker:Then you have Curtis and Curtis maybe an administrator, but doesn't
Speaker:have administrator rights, and so I need to run it as administrator.
Speaker:but to go back to your earlier comment, I couldn't agree more.
Speaker:And it's the same in the backup world.
Speaker:If your culture is not one of cybersecurity.
Speaker:they're just gonna fight you the whole time, and I'm sure you've
Speaker:been in situations where the culture is the problem, right?
Speaker:I can think of a company, I can think of a, oh, this is a great story.
Speaker:There's this company.
Speaker:on the East Coast and I used to work for a big consulting company and they had this
Speaker:particular company, and this is a company that literally everybody would know this.
Speaker:they actually make things.
Speaker:they're the rarest of all companies is an American company
Speaker:who actually makes things.
Speaker:you have these things in your house, guaranteed.
Speaker:And we went to, they were having all these problems with their backups and I, I was
Speaker:asked to come in as the big guns and to come in and basically tell them, 'cause
Speaker:they were trying to get things done.
Speaker:And then the organization was fighting them and I came in to give
Speaker:a, presentation, which basically the title of the presentation was,
Speaker:are the 10 things that you can do to screw up your backup system.
Speaker:And congratulations.
Speaker:You guys have managed to do every single one of them, But there was a person,
Speaker:there was a person that, that had power.
Speaker:Who was 100% against everything we were trying to do.
Speaker:She just, and when you have that, when you have a person, whether they're high
Speaker:up in the organization or down low in the organization, if they've got influence
Speaker:and they're actively fighting against you there, there's just not much you could do.
Speaker:And I'm sure you've.
Speaker:been in that situation, Mike, where you're trying to do something like this
Speaker:where it's like you shouldn't be using your VSS snapshots as backups and you
Speaker:really shouldn't, and you should also be monitoring who's running VSS admin.
Speaker:What.
Speaker:What, by the way, lemme just stop for a second.
Speaker:If you ag, if you follow the suggestion from the first part of the call, which
Speaker:is don't use VSS admin as a method of doing snapshots, this concern becomes
Speaker:much, much less important, right?
Speaker:Because the worst you could do with VSS admin, if you're not using it for backup,
Speaker:if you're not using them, as a backup method, the worst thing you could do with
Speaker:VSS admin is create a snapshot, right?
Speaker:Um, so.
Speaker:Um, no, but there's no snapshots to delete if you're not holding onto your
Speaker:snapshots right then who's running VSS admin becomes much less important.
Speaker:okay.
Speaker:Let's move on to
Speaker:so Mike, I know we talked about okay, running VSS admin
Speaker:within the Windows box is bad.
Speaker:Keeping it long term is bad.
Speaker:we talked earlier about hardware snapshots are good.
Speaker:Could you talk about from a cybersecurity perspective, how this kind of
Speaker:might change with virtualization?
Speaker:And if virtualization and VSS lead to less cybersecurity concerns.
Speaker:I know you talked earlier about like auditing and looking for
Speaker:anomalies and that sort of thing.
Speaker:Huh?
Speaker:Is that a loaded question?
Speaker:Is that a hard.
Speaker:no.
Speaker:so on a virtual machine or a virtual host with multiple virtual
Speaker:machines, Again, it depends.
Speaker:It depends on what the bad guys were able to compromise.
Speaker:If I compromise the host, then I've got access to all the virtual machines.
Speaker:If all those virtual machines are doing their own snapshots, I now have
Speaker:access to all of those snapshots, because if I've compromised the
Speaker:host, I'm also privileged on this box across all those virtual machines.
Speaker:So there's that.
Speaker:one of the things I do want to add though, even though the majority of, VSS.
Speaker:Process and, data require admin privileges as a normal user on the machine, I
Speaker:can still read those shadow copies.
Speaker:So even if I'm a bad guy and have, I'm not admin yet, I can still do recon into
Speaker:those shadow copies to see what it is I have access to and potentially exfiltrate
Speaker:that leverage in an extortion attempt.
Speaker:Because it's already packaged and compressed, right?
Speaker:I don't have to redo that.
Speaker:it's a nice little nugget, that I found.
Speaker:But in a virtual environment, just as virtualization makes things more
Speaker:efficient for us, it also makes things more efficient for bad guys.
Speaker:And back to a comment.
Speaker:or a thought, thread that Curtis was on?
Speaker:As far as culture goes, I've seen it both ways.
Speaker:I've seen where majority of a user base drives culture.
Speaker:so even the leadership team may say, we need security, but then
Speaker:you've got this workforce that says, we will not use security.
Speaker:especially in a service or organization, where they're out, your user base is out
Speaker:delivering something and they're rebelling against you, you will often cave to them
Speaker:unless it's truly regulatory required.
Speaker:and then I've definitely seen the opposite where leadership always wants the,
Speaker:they're gonna push it down to everybody else, but they've got the exception.
Speaker:And then everybody sees that, or maybe they become aware of it and
Speaker:they're like, Leadership's not doing it, I'm not gonna do it.
Speaker:setting the bad example type of thing.
Speaker:Yeah.
Speaker:And this is one of those, I, I've been in environments where, I've
Speaker:got people in power that are telling me that raid is backup, right?
Speaker:We don't need backups.
Speaker:We're we have raid, and then they're actively lobbying against you.
Speaker:That's the situation where like you need to just walk away, and, let them leave.
Speaker:Go ahead.
Speaker:there, there is an example of raid that is acceptable, but you have
Speaker:to unplug it and put it on a shelf.
Speaker:Okay, that's not just saying, not what we're talking about here.
Speaker:Mike.
Speaker:You're killing me.
Speaker:am just saying it.
Speaker:I've seen it.
Speaker:They've plugged in a storage array.
Speaker:They've done a backup.
Speaker:They unplugged it and put it back on the shelf.
Speaker:That's not braid, that's, you know what I'm saying?
Speaker:Dang it.
Speaker:don't, you're trying to get my, trying to get my blood pressure up.
Speaker:It's
Speaker:so this is a little bit depressing of a podcast episode, but is the takeaway that
Speaker:VSS is evil and no one should use VSS.
Speaker:just don't use VSS as a backup.
Speaker:That's literally, we could, this could this episode could have
Speaker:been two minutes long if you're using VSS as a method of backup.
Speaker:Not to be confused with using it to make your backup.
Speaker:I, don't you, you understand what, I don't know you're using it to facilitate
Speaker:whatever actual backups you're doing.
Speaker:In other words, the way we just, where we talked about where we said, Where
Speaker:you're creating a stable, consistent image that you're then creating an actual
Speaker:snapshot of or an actual backup of.
Speaker:I have no problem with VSS if you're using VSS to create multiple shadow copies
Speaker:you're storing those on your system, it's a bad idea for multiple reasons.
Speaker:We've talked about performance.
Speaker:Mike talked about using them as a method to get historical information, and it's
Speaker:also bad because all it takes is one.
Speaker:Bad, one black hat with VSS admin rights, and then they just delete 'em all.
Speaker:So it's just bad.
Speaker:So forget all the cyber stuff.
Speaker:We said, just stop.
Speaker:Just don't do this.
Speaker:get a real, go get a real, there, remember there, there was a, I used
Speaker:to have this, I don't know, it's probably bigger in my life than either
Speaker:of yours, but there used to be this Dilbert comic strip that we had, and
Speaker:it was, it was the, where the guy says.
Speaker:There's a guy with suspenders in the beard he is oh, I recognize you.
Speaker:And he says, beard and whatever, and then smug expression on your face.
Speaker:You're a Unix admin.
Speaker:And then the guy flips him a quarter, and he says, here's a quarter kid.
Speaker:Go buy yourself a real computer.
Speaker:That's the way I feel about people that are using their VSS, as backups.
Speaker:okay.
Speaker:I don't know.
Speaker:Do we have any final thoughts on this?
Speaker:So
Speaker:I think you just said it.
Speaker:go ahead, Mike.
Speaker:VSS is often necessary as the guys talked about earlier.
Speaker:it, it helps you, quote unquote lock files so that they can be prepared for.
Speaker:a backup.
Speaker:but that backup is some other solution.
Speaker:a lot of backup solutions today do that for you.
Speaker:they put an agent on your machine.
Speaker:It does the, it does this function, in the absence of that capability, you're use,
Speaker:you're using some other backup solution.
Speaker:VSS is still some, is still often necessary, to prepare
Speaker:your data to be backed up.
Speaker:and then the last thing I'll add from a detection perspective is again, if I
Speaker:knew you were doing backups every night or certain days of the week or once a
Speaker:week, and that's often scheduled, right?
Speaker:It's either a task or, a task on the local machine or a scheduled,
Speaker:activity across the network.
Speaker:If I knew that from a cybersecurity perspective, and then I saw.
Speaker:VSS or even some other privileged, backup related account, doing things
Speaker:outside that schedule, that anomaly, would be what I would respond to.
Speaker:Absolutely.
Speaker:Yeah.
Speaker:I like by the way, there was a, it was an earlier topic.
Speaker:when you're in a new environment and you are trying to configure
Speaker:their EDR tools or XDR tools, can't you just put it in and let it watch
Speaker:for a while to see what normal is?
Speaker:You can.
Speaker:and during that period, you're gonna get a ton of alerts.
Speaker:so there's your false positives.
Speaker:Okay.
Speaker:as you're monitoring, I found a new file that's not common across other
Speaker:user bases or, Bob logs in at 2:00 AM is, I don't know if that's normal yet.
Speaker:during that baselining period, you're teaching those tools,
Speaker:tweaking them or whitelisting them.
Speaker:there's a ton of things that, that happens during that onboarding period.
Speaker:so that you're essentially creating the what's normal baseline?
Speaker:You can't just say, just watch for a month and don't tell me anything.
Speaker:you could, it's not gonna
Speaker:you don't advise it.
Speaker:Okay.
Speaker:I see what you're saying.
Speaker:no, I don't advise it.
Speaker:Okay.
Speaker:this is why you're the cybersecurity professional, and I'm Mr. Backup.
Speaker:All right.
Speaker:All right.
Speaker:this has been fun.
Speaker:somewhat depressing, but yeah, just don't do that, doctor.
Speaker:It hurts when I, it hurts when I do this.
Speaker:Don't do that.
Speaker:Yeah, I'd like to, that's a new t-shirt idea.
Speaker:Just don't do that.
Speaker:do that.
Speaker:the antithesis of, Nike.
Speaker:Yeah, don't do it.
Speaker:Thanks guys for being on the show.
Speaker:Thanks, Prasanna,
Speaker:It was fun and I'm glad we gotta talk about something that was near
Speaker:and dear to my heart many years ago.
Speaker:So.
Speaker:and thanks again.
Speaker:Mike, you're a little fuzzy all of a sudden.
Speaker:What happened with your camera there?
Speaker:it's like an auto focus thing and it's got its own mind.
Speaker:So I'm not real sure, but,
Speaker:Anointing
Speaker:a few more of these with you guys, I would expect some logo.
Speaker:I'll wear a logo at some point.
Speaker:you'll get one when I get one.
Speaker:That's what I'm saying.
Speaker:on.
Speaker:All right.
Speaker:Okay.
Speaker:thanks to our listeners, we're nothing without you.
Speaker:That is a wrap.







