How to Protect Backups from Ransomware (Encore)

In case you haven't heard, ransomware is after your backups.

This encore episode is one of our most popular episodes from the last few years,

and it covers exactly how to protect your backups from ransomware, why attackers

are exfiltrating the data before they even try to encrypt it, uh, what real

immutability looks like, uh, who should have root access, and the questions that

you need to ask your backup vendor before you give them, uh, your data and trust

them with your last line of defense.

If this is your first time watching me or listening to me, I'm W.

Curtis Preston, AKA Mr. Backup.

I've been obsessing about backup and recovery for over 30 years now.

If that's your bag, then I'm your guy.

You're not gonna find anyone that cares about it more than me.

Ever since 1993, when I had to tell my boss that there were no backups of

the database that we had just lost.

Now, I've written five O'Reilly books, a blog, and a podcast.

Here, we turn unappreciated admins into cyber recovery heroes.

This is the Backup Wrap-Up

Hi, and welcome to Backup Central's podcast.

I'm your host, W. Curtis Preston,

Oh.

a, Mr. Backup, and I with me, a guy whose only major flaw is that he

won't go see scary movies with me.

Prasanna Malaiyandi.

How's it going?

Prasanna.

I am good, Curtis.

I, yeah, I'm not a big fan of scary movies, especially on a large screen.

Um, I don't like, I know some people like you love it,

ah,

but it's just I don't find joy in watching a scary movie.

I do remember in college watching the Ring.

On a big screen.

That movie messed with my head.

I remember coming home and asking someone to take my picture.

And saying, is there a thing in the picture?

You know, I was scared.

I was scared to take the picture.

Yeah.

Yeah.

was, that was a fun movie.

Um, it really kind of messed

Yeah.

And you just saw a scary movie just recently.

Yeah.

It, it,

it's not a

know if I'd called

Scarer.

Yeah.

per se, it's more like a thriller.

Um, It's called the Menu and it stars.

Uh, Ralph finds, um, a k a Voldemort, and uh, Anya, the

Hello.

Queens gambit.

Yeah.

Um, and some other, other character actors that you may be familiar with.

Um, uh, it, it's, it's a movie that I, I enjoyed the movie.

I will just say this, um, trigger alert for those who are bothered by such things.

It does contain suicide.

Um, having said that, within the context of the show, it, it was a

fits in.

part of the storyline.

Mm-hmm.

Um, and um, having said that, it was a really unique storyline.

That's a commentary on Modern Day Society.

You know, there's a class thing, there's a, there's

Yeah.

yeah.

It's not often that you come out of a movie and be like,

oh my God, that was so good.

yeah,

Like, I don't think,

enjoyed

yeah.

like, oh, that was, I thought the ending.

I don't know.

think I would've done something slightly differently within the ending, but the um,

That is why you are not the director, Curtis.

We're the writer.

Yeah, that's why I'm not the director or the writer.

I, I was not the only one, by the way.

I read some reviews and stuff, and a lot of people felt the ending

was somewhat anti cli climactic.

But it was one of those movies where you're like, what?

What is over?

Like, it's one of those kind of movies, but everything all the

way up to that point I thought, I thought was really, really good.

All the actors are really good.

Um,

never seen a Bollywood movie?

yeah, what's

Where it's like anti-climactic at the very end.

It's like they literally, a lot of the movies boiled the ending down

to like the last like five minutes.

interesting.

Um,

Yeah.

Shrek, oh, you would be proud.

Shrek.

And I watched while you were watching this movie, Shrek and

I watched a Bollywood movie.

Uhhuh,

On Netflix and it was also suspenseful

mm-hmm.

of a comedy thriller ish, but I would say that they basically

wrapped up all the entire movie in the last like five minutes.

And then at the end, Shraddha and I were like, what did we just watch?

Interesting.

because we, because it left so many things open at the end and we're

like, did we just watch all of that?

Just for that last closing part?

That's funny.

it's like two and a half hours also, so,

Interesting thing about this movie is, you know, how the movie's gonna

end about halfway into the movie.

Hmm.

Like they tell you how the movie's gonna end.

it's just how it gets there.

It, it, it's just was, was

Yeah.

Nice.

Um, so I recommend it.

Um, and, uh, you know, the, the

are you and

saga continues.

yeah.

Are you and Jeff gonna talk about it on your next, on the other podcast,

know, he keeps threatening to record another episode and then he doesn't do it.

But I'll , I'll take

I think Curtis is calling you out here.

Jeff and I actually recorded a new episode this week, so look for it at,

The Things That Entertain Us podcast, wherever you listen to podcasts.

Today we're going, we're continuing in our backup to basic series, and

we're covering, we're covering the book, modern Data Protection on Hold.

those of you watching this on backup central.com where you can watch the video

of Prasanna and I waxing philosophical.

Um, and, uh, you know, what we're doing is we're sort of going through the book

just discussing, you know, some important topics that I, that I found in the book.

and this next one is about protecting backup and archive data, which

think we can all agree is important.

If you would like to download a free e-book copy of Modern Data Protection,

you can do so at druva.com  slash ebook.

Uh, we have to protect it from a couple of different things, right?

We have to, mainly what I'm talking about in here is protecting

it against attacks, right?

both attacks like a, um, like a disaster type type attack.

But, but mainly what I'm talking about is protecting it from cyber attacks,

Um, I'm gonna have these same conversations over and over, so

sometimes I forget what we say here versus what I say other places, but

it, it is a well acknowledged fact that some of the largest of the ransomware

bad actors are specifically targeting backup systems in order to do the.

The extortion style of ransomware attack, and let me specify what I'm saying there.

So, you know, a, a bunch of companies started realizing that, well, if I just

get a decent backup in recovery and disaster recovery system, I can say, you

know, pound sand the ransomware folks.

Mm-hmm.

the ransomware folks then said, oh, well, we'll show you.

And so they developed this, this exfiltration style, right.

and the um, The idea there is that they find databases and file

systems and spreadsheets and whatnot that contains sensitive data.

That could be two different things.

That could be your company secrets.

It could be your company plans.

It could be your 11 herbs and spices or your, your, um, You know, your

corporate plans for the next year, you know, um, but it could also be

secrets that are like embarrassing.

Uh, I think the best example of that, and I know we've talked

about it, is the Sony attack

This is the one where someone broke in to the movie studios and got a

bunch of emails talking about scripts and actors and all the rest of that.

and very, some, some, and not very

Yeah.

flattering terms.

Yeah.

So it could be that, right?

the point is, there is no way to protect against that once the data's out.

Yep.

It's not like the other attack.

If you get the, the old school attack, if you had a good backup system, you're good.

It doesn't matter how good of a backup system you have, if the

data is exfiltrated, you're gonna be forced to make some major, uh,

Yep.

right?

So, um, what, what these, uh, ransomware groups have started realizing is,

they, uh, instead the, the people are starting to harden

their primary environments.

Mm-hmm.

And so, but one, you know, you, you, you've heard me talk, did

I say one thing that has never changed in backup and recovery in

the world of backup and recovery?

Do you, do you know what it is?

I often say like

It's

that has never changed,

that the junior person is always responsible for backups

and recovery.

that is technically a cause of the thing the thing that

No one who cares about

no one wants to do the backups, right?

one wants to raise their hand and say that they want to be the backup guy.

So why does that matter?

Well, it's because it means that, that it ends up being an an or and ignored system.

Yep.

and as a result, it that it gets ignored by everyone, including

the cybersecurity folks.

Right?

They're like, ah, that's just a systematic in the corner.

It's fine.

We don't care about it except it contains all of our data.

Yeah.

And depending on how, how you handle, things like encryption,

they may be able to unen encrypt.

Said backups and then, you know, get the keys to the kingdom.

Yeah, and like you were saying earlier, Curtis, the fact that a lot

of people focus on production systems harden them, but they may not always

be considering these backup systems.

Right, right, right.

Easy target.

Easy target.

And so you need to.

not be such an easy target, right?

Yeah.

the old joke about the guy, two guys crossing a field and they see,

they see a bull and the guy says, you know, can you outrun that bull?

He said, Nope, I don't have to.

What are you talking about?

All I have to do is outrun you,

You

So, so you just want to be less of an interesting target

than the person next to you.

So the, the first thing that we talk about in terms of protecting

backup data is encryption,

Yeah, which

uh,

everyone should be doing.

Like if no, if you aren't doing encrypting your backups, I don't

know what's wrong with you.

Like even if it's not encrypted, honestly,

like that should be your default.

Persona.

no, because encryption isn't some heavyweight thing that it used to be.

Right.

You don't have to go physically encrypt your data as it comes out.

You could do disc level encryption, right?

There's so.

Ways you could leverage encryption to fit into your environment.

It should be a no-brainer that all your data at rest is encrypted.

And in a lot of companies, right?

If you wanna fit, be sort of like, uh, what is it?

If you wanna look at HIPAA com, uh, readiness or other things, right?

Other regulatory obligations.

A lot of 'em say All your data at at rest should be encrypt.

Mm-hmm.

. Well, I, um, do think, I do think backup page backup data should be

encrypted for this very reason.

Right?

And I'll give you different reasons for the different things, right?

So if it's, if you got a backup server on-prem and you're storing

your backups on a. File system, you know, an NFS mounted file system.

Then they're unencrypted, you know

Everyone has access.

Yeah.

Everyone has access to those backups and a lot of these PR backup products they offer

to, to, to, so some of the backup products store the backups, not in a backup format.

They're in native format, so you could just mount a disc, right?

Yeah.

backup products, put it in a backup format.

but they offer an ability to read that backups like it,

it's, it's for DR purposes.

So, so if you're not encrypting them, then you're just, you're

just asking for trouble.

Yeah.

the disc side.

The second side will be the tape side.

So if you are, and believe it or not, many people are still making backup

tapes and they're making 'em primarily, I think for offsite, uh, storage.

Yep.

And so, And this is the, this is the one where I'm like, if you are ma, if you are

putting your company's like crown jewels onto a little, you know, plastic tape and

then you're handing it to a man in a van

Yep.

you are not at this point encrypting that data,

you know, professional malfeasance at this

point.

and I wonder, it's almost as if that should be like a default, you know,

like you can't write data out without encrypting it, but I guess you have to

deal with key management and all the rest of that, and so, , do you have

flexibility to shoot yourself in the foot?

yeah, exactly.

but I will also say this encryption is only good, is only as good as the

authentication and authorization and key management system that is connected to it.

Yep.

And if you can easily defeat.

That, uh, system then, you know, and, and, and I mentioned, so for example,

I know we mentioned it on the podcast, please do not make your backup server

part of your, uh, active directory.

What,

Domain.

it,

Domain.

domain.

Yeah.

Um, Just please don't do that.

so the next term is one that we can debate.

No

I, honestly, I don't even need you.

I can debate it myself, right?

It's one of those where I go back and forth, you know, and that's the term air.

What is like air gap?

Curtis?

well, well, there

Traditionally?

Yeah.

Yeah.

It's, and it is where it, it is from once we get the term.

the idea was to put a gap of error between the protection copy and the protected

Yeah.

And for

you're dealing with plumbing right now, since you're dealing

with plumbing right now, right?

You have an air gap normally, right?

When you think about your dishwasher

Now you made me think about the dishwasher that

has made my

or,

fine.

Fine.

Or a toilet.

How about that?

that's actually technically a water gap, but

Don't, you're killing me.

Don't be, don't be bringing plumbing into this.

anyway, yeah.

So you, you had to separate the, it meant, again, it meant the, it

meant the man in the van, right?

You, you, you, you put some tapes in a box and you handed that box to a, you

know, a man or a woman, and then they got into a van and they took it someplace.

And in order to get to that thing, needed to physically, Um, access it.

Right?

Yep.

that if you were a hacker and you wanted to get to it, it, it

would be virtually impossible.

Not completely impossible.

There's no,

It's

cuz

penetration rather than cyber.

yeah, remember for example, we would, uh, once a year at an

unannounced time, we would attempt to circumvent the security of.

Uh, storage fender

Mm.

we would, uh, use really good like liars and whatnot.

And more specifically, we would use someone like me that they

knew, they knew the person.

And so they would go over there and they would, and we would just have

this really, really like elaborate.

Scenario

yeah.

that basically would want, would want the, it would make the person

that works there want to leave us alone in the vault with the tapes.

And at least once we were able to do that,

Hmm.

Was a person go, or did the person have to go through training, I guess.

Yeah, I don't, I, you know, honestly, I was so long ago, I don't remember

what happened to that, to that person who left us alone in the vault.

But, so that's what I'm saying.

It's not, it, it was never 100%.

Right.

Um, the, um, you've seen the movies, you

Yep.

I've seen Oceans 11 and 12 and 13.

The um, but the problem is most of us don't use tape.

Mm-hmm.

And most of us, or many of us don't even use a data center anymore.

Yep.

So this idea of putting a, an actual air gap between A and B is quite difficult.

And

Yep.

we've sort of adopted or adapted

the, you know, the virtual air gap.

Right.

Um, the, um, Uh, so this is in as many ways as we can.

I don't know.

You want to talk about

Yeah, and I think we talked about this on the episode about backups,

or maybe it was restores, but.

It's even though they are available online, right, that second copy,

so it's not the same as having that tape in a vault somewhere.

You wanna make that, make it as close to that as possible.

And we totally get that.

You still have to have connectivity, right?

It's still gonna be up, but can you lock it down?

Can you basically make that target as inaccessible as possible?

Uh, to prevent it from being attacked by say, someone, um, gaining access

to your backup source, right?

And now they have access to your vault.

Uh, some of the examples are if you're backing up in the cloud, right, put

it in a separate account that no one has access to other than whatever

is transferring the data, right?

Make it in a different region.

So it's not easily accessible.

And there are many, many, many other things you could do as well.

But those are just some of the example.

Yeah, I like if we're talking cloud, I like the idea of putting it in a

different region and a different account.

Right.

Um, because we're not just talking about hacking.

We're also, we also need to think about like, Disaster recovery and things

Yep.

right?

So again, O V H Cloud, we don't want to have that happen to us.

So we want to make sure that a copy of the backup data is, is well far away

from the thing that it's protecting.

And yeah, separate account a much more limited, an account

with much more limited access,

Yep.

and you can.

Um, you can consider using immutable storage, but we're

gonna talk about that later.

But essentially, this is the, we, we separated as much as This

Yep.

um, you know, one of your previous employers data domain, right?

And, and, and, All of those things, they would always talk about

replicating to another data domain.

And I always, I always remember thinking, well, if I'm replicating to another data

domain, it's essentially like in order to do that, I gotta have it in the same

So,

like land.

Right?

Or, or at

or Van.

like it's in the land.

Yeah.

Well,

Um,

that's why they actually introduced a feature, specifically talking

about data domain, right?

Their cyber recovery solution.

Mm-hmm.

that allowed for writing to a lockdown data domain that didn't

have access from anywhere else.

And going back to the point you were talking about, Curtis, okay, is it

on the network and I can connect to it then, doesn't that make it open?

And so what it would actually do is shut down network ports right

outside of your backup windows, such that it's not available, or

at least your attack surface is.

Could you delete the backups?

Was it.

Could you delete the backups once you send it to it via the backup interface?

Typically no.

Okay.

Because that, cuz that would be, that would be an important

Yeah.

It would also replicate like your backup environment, right?

So you'd have your own backup server in that vault, if you will.

So there's no connectivity to anything outside, because if there

is, then you're just open to risk.

Right, right.

Um, so just like, so I've got a list here on virtual air gap here.

talk about disabling or impairing R D P. Why does that matter?

Ah, so many ransomware issues have happened.

Rdp, which

is Remote Desktop

the way?

Protocol,

you,

right?

Which is how most people connect remotely to a Windows client.

Um, most people forget to turn it off or they leave it on because it's just easier.

They don't need to physically connect to it.

Uh, The only downside is it's a big attack surface, and there are

a lot of exploits using R D P.

And so when you enable R D P, you're letting yourself or leaving yourself

open to hackers and other exploits coming in, which could then move

laterally across your network and take out other pieces of your environment.

Yeah, there is a way to automate sort of disabling and enabling R

D P, um, like on a large scale.

Um, I also talk about putting it in a different operating system, right?

Not

Yep.

operating system for your backups in your primary,

Yep.

that's possible.

I know that there's a lot of big window shops and they use windows.

Uh, backup servers.

I don't think that's a good idea.

Um, but you know, Um, and by the way, I, I forgot to throw out our disclaimer.

I'll throw out, um, Prasanna.

Prasanna.

And I work for different companies.

I work for Druva.

He works for Zoom.

this is not a podcast of either company.

This is an independent podcast and the opinions that you hear are ours.

And, uh, be sure to rate us@ratethispodcast.com slash restore.

um, if you'd like to join the conversation, you want to talk.

What's going on in your world?

Uh, you know, whether you're, uh, you know, an end user.

If you're an end user, we really want to have you, you know, real

people that actually do backups out there in the trenches.

Weed.

You're,

Do.

Do you miss it, Curtis

No

I do, you know, as soon as I think about, um, Back when I used to be the

person in the trenches, like when I was a consultant, and I, and I, I flashback

to this one time where we were completely redoing the backups of, I'll just say

a large television station, like, like a television station, and remember.

I remember that time when the, when the, the, the of it basically said,

going home until this is done.

He literally, I mean, there's a long story following up to this, but he basically,

um, held us hostage in the data center said, no one's going home until this,

this thing you said is gonna happen.

You said it's gonna happen.

I'm like, yeah, but it's not gonna finish until.

10 30 at night, like, yeah, so we're not going home until it's done.

And I'm like, okay, well this is kidnapping, but whatever.

Um, you know, uh, so I think about stuff like that, right?

Yeah.

moments of terror.

But yeah, I, you know, I wanna live vicariously through those who

have been through those moments of

Yeah.

Well, and it's also interesting to learn like what the more recent challenges

are in environments, because otherwise

how do you

Yeah,

otherwise, how do you build great products, you know, or build

products to solve

exactly.

So what, what I was thinking about saying, which is what reminded me to do the

disclaimer, is just that, um, you know, one way to have a different operating

system is to use a SaaS provider.

Druva is not the only one, but there are SaaS providers

that are cloud native or use.

Non windows, uh, tools um, you know, you do that.

Also, you, you gain the second thing that I list here, which

is about separating the storage.

Yeah,

don't, um, you know, we talked, we talked about that already,

the, these replication, right?

Uh, but um,

but it's also

you're just.

use nfs.

Yeah, exactly.

Yes.

Don't use NFS as a way to back up to the server use.

There are ways to back up to, for example, a data domain box in other

boxes exposing the backups via nfs.

Uh, I also talk about using object storage.

I'm a huge fan of using object storage.

Now, some of you go, well, object storage is too slow, to which I

say, then you're using it wrong.

Yep.

Right.

Yep.

Druva uses object storage.

All our backups are on object storage, and we are not too slow.

In fact, we get into competitions all the time with these big

on-prem companies and we win.

And here we are, we're a copy in the cloud and they're an on-prem

appliance, and we win that restore test.

Object storage is not too stor slow, but if you, if you treat it like file

system storage, I think then it is,

And it's a challenge.

Yeah.

all your backups in one object, Tonight's, tonight's backup of

server X is an image, and that image becomes an object on, uh, s3.

Yeah.

Then it's gonna be slow, right?

Um, we talk about using aut storage.

We're gonna come back to that in a minute.

Um, that, that's an option that we talked

And then,

S3

yeah, and then your.

And then your favorite topic is using tape.

I know, Curtis, you always like to throw out the disclaimer.

Actually, I should thank you because before this podcast,

I knew very little about tape.

Now I know just a bit more about tape,

yeah,

the number of experts who've come on in talking to you.

But yeah, it's

not

there's no, there's no beating the tape when it comes.

That's an actual air gap.

Yep.

right?

If, if you, if you want a real air gap, I'm, I know there's challenges with

it, you know, uh, but there is, you know, there is, there is an air gap

Yep.

Uh, and I already talked about using a backup service.

So let's talk about immutability.

Yep.

What is immutability?

Is it a made up word?

It sounds a little like a made up word,

you know, this is another one of those things where we have to go back in time.

Historically immutability wa it, it, it was like a legal term that you needed

to be able to prove that the thing you were presenting in court had not changed.

It was immutable.

That's all, you know, immutable just means cannot be changed.

Yep.

when you're, when you.

uh, you pres, you, you know, you have to preserve chain of custody and you

have to, you have to be able to say, or it's really helpful if you're able

to say, this email that I'm showing you is exactly the same email as the email

that we got five years ago from Steve.

Yep, and nothing changed and no one had the ability to change it yet.

That's where we used to talk about immutability.

Now we talk a lot about it in terms of cyber attacks.

And I would add to that, we also talk about it in terms of things like bit rot.

We've talked about bit rot on this podcast where this is

silent data corruption, right?

That bits just flip underneath magnetic storage devices.

It's just the way they work.

and the, you know, and if, if you've got, if you've got a bunch of petabytes

of data, you got flipped bits in there.

It's just a matter of, it's just

when you're gonna run across it, right?

yeah.

Yeah.

Um,

so, when we talk about immutability, we say, well, we, if somebody

says they have immutable.

Backups or immutable storage.

What you're saying is stuff that gets put here can't be changed.

Mm-hmm.

here's the thing.

Depending on how you look at it, it's complete nonsense.

Because everything's just software.

Anything can be cheap.

is changeable.

Yeah.

me a torch, I'll change that.

S o b, right?

Give me phy.

So All bets are off.

Everyone knows that, um, if it's something on a server that you can, that, that,

that is in your data center, that also, in my opinion, is not really immutable.

Um, because if someone has root, uh, on that server, they can wipe the server.

it's even built, like, you know, when we talk about immutable storage on

Lennox, it's not really that immutable.

It's it's immutable ish.

So, so why do I say, like, why do I talk about it if it's bs?

Well, here's the point.

Nothing was ever a hundred percent immutable,

Even that document right, that you were talking about.

Yeah, exactly right.

As long as like if, if, if you put something on a a, a, a, right once

cd, uh, optical platter, right?

A worm tape, it's immutable to a point.

If you have physical access, it's no longer immutable.

Yeah.

You could, you could say from a legal standpoint, you could later, if you have

the same disc and you've preserved chain of custody, and you could say, we can

show that this disc was not destroyed,

Yeah.

we can show via these check sums and whatnot, that the stuff that we're giving

you is the stuff that we have before.

Saying that you've got a storage system that can't ever, ever, ever be changed.

Um, I would argue it's just nonsense.

Right.

So it's, it's a, it's a, it's a, it's a degree.

It's a what, what did we call it?

Like a,

Spectrum.

like a spectrum.

Thank you.

I

Yeah.

was a word in there.

And even

prism is what was coming to mind.

and even for those storage systems that have immutability, right,

that are used for like financial records and everything else,

Mm-hmm.

of that is certified by the storage vendors saying they have all the checks

in place and all the rest, right?

In the end it's all just software, right?

So they've built in the, into the software, those checks to make

sure that data cannot be deleted,

right.

is present.

And the, and you know, we're doing all this, like we're gonna, I'm

slamming it and then I'm gonna talk about how important it's . But the

other is, um, Uh, I'd say the, one that I like the best right now is

like the, the object lock and s3.

And what I like about that is because it's, it is at least

physically separated from.

You.

Now, that doesn't mean that somebody can't go crazy in the AWS data

center and, and blow it up, but even that, it's built into that.

So it, it would have to be like, it would have to be a really concerted

attack to be able to attack multiple locations of S3 to be able to do damage.

it's, it's not only

the

multiple

is

location.

as what's

Yeah.

It's not just multiple locations of s3, but also they have to tack your primary

site as well, so everything needs to be coordinated across multiple vendors and

corporations, which will probably have their own security practices, et cetera.

this is, this is that, this is that, uh, spectrum, right?

Yeah.

I'd put, I'd put that and write, you know, worm tape, write once,

read, write once, read many tape.

Um, and then, and, and optical plat.

I put that on one end of the spectrum.

on the other put at the end of the spectrum is I would have unencrypted

backup stored on a NFS mounted storage system behind a Windows backup server.

Yep.

that's your spectrum.

So we wanna be closer to this end than that

Yep.

Right?

Um,

But there are some

ever

downsides.

Yeah.

And there are some downsides to immutability too, right?

Uh, you can't, like once the data gets written, you can't delete it

before that time period typically.

So there are some challenges as well.

So, um,

so what we need to do, Is we need to, I, I think

I.

multiple things there is that, um, we need to protect against access.

We need to protect against, you know, change deletion, corruption.

Um, uh, was I, what was I talking about?

Um, bit rot,

Yep.

So there are vendors out there.

That, they're like, well, we have a, append only file system, and, we,

and we have data lock the backups.

And so they say, we're immutable.

and I'm like, okay.

And, and they say because they're saying that like, you can't encrypt the backups

with a, with a ransomware attack.

Yep.

and I, and I go, that's good.

That's farther, that's, that's closer to the other one than the other ones.

But it's like, well, what about other things?

Right.

What about other types of attacks that attack the operating system itself?

Right.

You gain privileged access to that server,

yeah.

And I wanna clarify though, what you're talking about is someone

who's written their own file system or is leveraging a file

system on top of a standard server.

Yes.

Not a storage appliance, because I think that's a little bit more,

that's like further along in the spectrum, I would say like a

Yeah,

storage appliance entire too.

So,

yeah.

So, So, so, we got a couple different types of backup servers here, right?

So

Yeah.

the purpose-built backup appliance of various

shades, shades flavors.

Sure.

Um, and I would put them.

More immutable, right?

So immutable is supposed to be a binary

Yeah.

I don't see it as that, right?

So, I put it more immutable than I bought a box.

I installed Windows or Linux, and I install my favorite backups or

Yeah,

Why?

Why, why do you feel that way?

I agree with you.

Why do you feel that way?

I feel that way because when it comes to that appliance, typically

there's more things locked down.

There's less configuration options, right?

It's kind of purpose built for that reason.

Versus when you're rolling your own, you have to worry

about all those dependencies.

What flavor of the OS are you taking?

Are you running through all the security patches?

Is it, are there any nuances in the way it's being deployed

today that leads to security vulnerabilities and things like that?

Yeah, exactly.

And, and when you update those appliances, you update an

image which updates everything.

Yep.

if I have a a box, a Windows box, I gotta update Windows or Linux.

And then the backup

software.

new security, then you gotta update the backup

Yep.

So, yeah.

And these are, and I would say that's, so I would say that the

appliances are more immutable than.

Than the build drill in box.

Yep.

and the, uh, but I, but I'm just gonna say that if you, if we're

talking physical access, I still, I'm gonna put a service like S3 or a

service like Druva that's up in the cloud, farther down the immutability

spectrum than a survey, than a server that is sitting in your data center.

Yep.

And I think it's important because everyone is now doing virtualized, right?

It doesn't matter if it's a physical server or a virtual server.

Anything that's running on your infrastructure or in your

infrastructure is less secure in that immutability spectrum.

Yeah.

the, uh, yeah, I put in the book a lot of things are mislabeled, immutable,

uh, I just, again, it's a spectrum, but.

If, if all you're doing is writing backups to a Linux file

system with the immutability flag turned on, that is not immutable.

I mean, it's, it is more immutable than not But if I, but if someone

with root can and, and you gotta have someone with root, right?

Yep.

if a person with root can go in and unset all those immutability flag,

It don't matter.

That's, I don't know.

That's just not, it's be, it's better than nothing.

Yep.

Right?

So I, I don't want to be the, the, you know, is the enemy of good or

whatever, but I don't need perfect.

And so I don't wanna say that that's crap, but I, I think it's, it's not as immutable

as those appliances that we talked about.

And I still think that a service nobody gets access to your servers

is more immutable than that.

Yeah.

and I still want everybody to back up everything to Tape

Oh, Curtis, you and

your tapes.

at, I work at a tapeless backup company.

Um, so

here's

here's

Go

one thing I wanna ask, and I know we've covered this on prior podcasts.

What are things that an admin can do to understand where on that spectrum,

when someone says, when a vendor says immutability, what are questions

that they should be asking to be able

to determine?

Who has root or admin, right?

Determined.

Who has that and what controls are placed over that?

So, um, if it's an appliance, so like I, I know of like one company.

They do have root, they have a they have the root password on your

system, or they have the password to an account that has a u i D of zero

Yep.

of you.

Um, is essentially the same thing, but in order to log into that account, they

can't log into that account remotely.

You have to.

You have to do an SSH

yeah.

stuff, right?

You have to open up a door for them to log in.

Um, you know, what kind of protections are put against that.

Uh, if the answer is it's just a local box and you're the one that has route,

or there's unprotected route access from someone, um, I just, I worry well.

you know, so if, if, if you or anyone in your ministry in your place has root, not

very immutable, it's better than nothing,

Yep.

not very immutable, right?

you know root, like if, if, if you normally never log into

the system as root, right?

No one in your, you only at you, you go to a ui, you log it as you, right?

Um, and.

There isn't direct route access by anyone.

And the only way you can get route access is you can, um, you do the SSH

tunnel thing that's more immutable.

I like that better.

Um, you still, but again, if you don't know this already, you

still have to do physical security

Yeah.

Yep.

Exactly.

do every other box.

Right.

Um, and then, You know, you ask a lot of que just ask a lot of questions.

What happens if so, for example, I'm currently wondering because I haven't

found a good answer online, I'm currently wondering what happens with Amazon S3

object lock if I delete my account.

I don't think they allow you to.

It depends on what type of object lock you're using, because there are

two.

what, I know what you're saying, right.

Um, but if I'm u is it the compliance mode?

It's the more,

Yeah.

yeah.

So assuming I'm using compliance

Oh wait.

Compliance is R L C R, lg.

I think compliance is less in governance is more, or is it the

other way around?

it

Yeah.

Whichever way

it's.

think it's the other way around.

But anyway, so I'm using the more strict one.

and my credit card stops working.

Right.

It's not like they're gonna keep my storage forever.

You, when you say they won't let you delete it.

Well, I don't have to delete it.

I'm just gonna have my credit card not work anymore.

Yeah.

Right.

What

like you said, it's not protected in all scenarios, right?

Right.

I don't know what the answer is.

I am curious.

I actually bought, I, I keep forgetting to do this, but I want to

go create an Amazon account separate.

Mm-hmm.

Put some object lock stuff in there.

Not a whole lot, just like days or something,

Yeah.

like a couple of gigabytes or something, and then go delete my account and see what

And See what happens.

let me delete the account.

Yeah.

I did read something somewhere that, that there is this like 30, 60 day timeframe.

Hmm.

So again, that's still better because I'm assuming that if you're using Amazon S3

You don't want it to go by.

or any of the other folks you're gonna, you're gonna know that

somebody deleted your account

Yeah.

Right.

Um, and I know what I would want you to ask is what happens if, what happens

if someone gets through someone?

So I was just seeing a thing that said that it's like more than half of hacks

are through compromised credentials.

So what happens if someone, you know, you're a backup service?

What happens if someone ahold of my admin credentials and is able to circumvent mfa?

Yep.

What

damage can they

do

you have

Yeah.

Yeah, how much damage they can do.

Can you undo any of that damage, et cetera, et cetera, et cetera.

I know that go back in time, let's say two years, I don't think Druva

had good answers to that question.

Yeah.

Right.

I remember being in those meetings and saying, Hey, we need to address this.

We have addressed that.

I think we now have really good answers to those questions.

but does your, does you know, if you're not using Druva,

does your vendor have good

Yeah.

questions?

I don't know.

Right.

So, yeah, so come up with worst case scenarios and that one.

compromised admin credentials, whether you're talking a service or

so, like for example, I know that like Druva has data lock, you know,

rubric and cohesive have data locks.

I know that with them, if you, if you compromise, uh, credentials

and you log into them and try to delete, if you, if you enable

datalock, you will not be able to do

Mm-hmm.

So, , what is your vendor's answer to that question?

Um, and um, and then also ask 'em about bit rot.

What are they doing about bit rot?

If they're using object storage, I feel better object storage will automatically

detect bit rot happening underneath.

Yeah.

cuz it will change the hash and somebody will be like, Hey, well not

somebody, a program will figure it out.

Um, I think Biro is rare.

Just for the record.

I think it's rare.

Um,

It's rare.

bit error.

Yeah.

But,

But it's still bad

when it happens.

Yeah.

And the problem is when it's silent, that's the worst.

Exactly.

That's the worst.

All right.

Well enough about protecting backup data.

You got any final thoughts?

Well, I think we covered it all.

I really like virtual air gaps, but that's just me

I like the old school air gaps, but it's really hard to do those these days.

All right.

Well thanks for listening to us.

The Backup Wrap-Up is written, recorded, and produced by me, W. Curtis Preston.

If you need backup or DR consulting, content generation, or expert witness

work, check out backupcentral.com.

You can also find links for my O'Reilly books on the same website.

Remember, this is an independent podcast, and any opinions that

you hear are those of the speaker and not necessarily an employer.

Thanks for listening