Building an immutable backup system

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

We got some feedback from somebody from our last episode, the the 10

things that every Backup System needs.

He pointed out that we missed a few critical, uh, security features.

So persona and I are taking a deeper dive into what makes

an immutable backup system.

If you're concerned about ransomware and um, I hope you are, then these

security measures are now mandatory.

Let's talk about how to actually protect your backup infrastructure from the

biggest threat you're probably facing.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup.

I've been passionate about backup and cyber recovery.

For over 30 years, ever since I had to tell my boss that there were no backups

of that database that we had just lost.

I don't want that to happen to you, and that's why I do this.

On this podcast, we turn unappreciated admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the show.

Hi, I am w Curtis Preston, AKA mischief backup.

And I have with me a guy who suddenly seems to be asking, when do I get my

name on the cover of a book persona.

Molly Andi.

How's it going?

Persona?

I am good, Curtis, I noticed in the background are a whole bunch

of more books up there than normal.

Yeah.

Well, it's just, uh, well, I think they were just covered up before.

Um,

another

is,

that was

oh, well I had the cover, I had the, uh, you know what I was,

it's funny that you say that.

I was, uh, prep, I'm doing a video actually for O'Reilly skills,

Hmm.

is, uh, kind of very exciting.

My first time doing a video for them, for their, it's their

learning platform and, um.

Oddly enough, we'll be talking about backup, but um, I was trying

to make it like, you know, look more, um, aesthetically pleasing.

And so I have the lovely picture of my wife and, uh, that is our baby.

If you can imagine this, this is, here's a picture for you.

That is me from

so for

years ago.

for folks who may not be aware, we have YouTube videos of this podcast.

So in

So.

listening to us on your favorite pod catcher, you can

also watch our smiling faces.

You can see Curtis's background.

You could see the picture of what he looked like 30 plus years, 30 years ago.

Oh,

Yeah.

Yeah.

Uh, that was our baby and interesting, that was our first baby.

Um, um.

Photo session with the baby.

And uh, she was screaming as we were heading into the thing and,

uh, we were like, she's hungry.

So we fed her and then she fell asleep and she is out cold

in that, in that photo shoot.

Anyway, that baby's now 31.

Um,

So check us out on YouTube.

yeah.

up is the name of the channel.

Subscribe, leave comments.

all the rest.

Absolutely.

And by the way, this episode and the next episode are based on feedback

that we got from our last, um, podcast, which was 10 things that

every backup system needs to have.

And, um, you know, they, they felt that I had, I had left off a few and I disagree.

These were like table stakes, right?

So, just a, a real quick.

Thing.

You know, these are like, if you don't have these, you don't

even, don't even talk to me.

Don't even tell me you have a backup system.

Right.

So we.

in case, so then they can learn what

What?

doing.

Oh, yeah, no, I'm saying if you're not already doing this.

Yeah.

Okay.

I, I, I see, I see your point there.

Okay, so here's a quick list of a recap of last, uh, the last episode.

So we talked about 3, 2, 1 rule, which is now the 3, 2 1, 1 0 rule.

Three copies of your data, two media types, one of them off the site, one

of them immutable with zero errors.

We talked about automated backups, right?

If you're having to do your backups manually, that is.

That is not really a backup system.

We talk about recovery testing.

If you haven't tested your backups by trying to recover, then you

don't really have backups defined.

RTO and RPO.

Recovery time objective, recovery point, objective.

Um, and then also, and we're gonna dive a little deeper on this one.

Isolating your backup system from a security perspective.

Uh, backing up your SaaS data.

Look at documentation and runbooks of your system.

Uh, defined retention policies.

Monitoring and alerting.

Make sure that somebody is constantly looking at your backup system.

And then finally, a listed endpoint, device protection.

And

And the thing is,

has

go ahead.

and he is going to go become an auction person who does like the biddings.

So you, so funny.

Um, so you know, the, uh, when you think about the things that.

A modern backup system, and again, for those that haven't listened to us before.

When I say backup, I mean that in the broadest possible sense, right?

So some people see DR and backup as two different things.

I do not, I just see them as two different, um, like ways

in which the backup is used.

Anything that is storing the data for the purposes of recovery

is a backup, in my opinion.

Um, as long as it meets the definitions of the, the things

that we talked about, right?

If it meets the 3, 2, 1, 1 0.

Um, for example, snapshots that are not copied somewhere, don't meet the

definition of backup, but snapshots that have been copied somewhere, uh, and

are being monitored and are, you know, scheduled and all this stuff, I think

they can meet a, the, you know, they can meet a definition of backup, right?

But if you look at, and we talked about this the last episode, the

number one reason that we are restoring these days is not hardware failure.

And it's not even like.

Fat fingering.

It's not I deleted a file or I corrupted a file.

You know, by, by hand.

I would say that the number one reason people are restoring

these days is ransomware.

Um, what do you think about that claim?

agree.

And there's so much monetary

right?

Just from an organization like we've talked about, companies that have

I.

like bust because they were not able to recover from a ransomware attack.

In fact, I think, was it?

Jaguar Land Rover is currently still trying to recover from an attack.

I

Yeah,

Ashi Brewing Company is trying to recover from an attack.

the, and

it's crazy.

Yeah.

and months, trying to recover

Yeah.

Um, and, and that's pretty much the, the number one reason we're re we're

reaching for a backup tape or, or, or disc or whatever it is that you're,

you know, I, I, I dated myself there.

Uh, most people are not using tape as their primary protection mechanism.

So some of them may have tape as a secondary protection mechanism, especially

I'd say if they're a smaller organization.

The reason why I say that is when you start talking about petabytes

and exabytes, for example.

Tape is feasible.

It's just that getting the, that amount of data regularly copied to tape,

um, can be well is, uh, difficult.

Right.

Okay.

It's not, not that it can't be done, uh, it's that it's

definitely, uh, more challenging.

But if you actually, just to go back to my comment about the smaller company.

I just saw an article that, a new company whose name I wish, and I'll,

I'll insert it here, A new company named Simply . they're actually selling LTO 10

tape drives they're USBC made directly to plug into, let's say a MacBook right now.

The tape drives are $12,000.

but losing valuable data,

Yeah, yeah, yeah.

So if you're, if you have, if you have the, the money to, to do that,

I'm just saying that the, the tape technology does continue to be, um,

um.

So let's talk about, uh, four more things that I think that you need to look into

if you want to basically modernize your backup system from a security standpoint.

Okay?

And the first one, and, and we, we touched on it already in the last

episode, and you know, I talked about segregating your backups.

And, but I don't think we really dove deep on modernizing the authentication

mechanism Historically, how did you log into a backup system?

Username, password.

Yeah.

What's, what's wrong with that?

hopefully you at least had a username and password.

Yes.

Hopefully you did have that.

Uh, well actually if we go back in the day, back in the day.

Um, the, the software ran like in user space.

You didn't have a separate authentication login, so you were only as good as

whatever thing you were logging into.

Uh, and generally it ran its root, um, good times.

So, yeah.

So at least in the last, like, say 15, 20 years, username and password was the

way, what, what's the problem with that?

Uh, you have users trying to manage passwords independently.

Passwords get stolen.

Passwords don't get rotated.

People will pick the easiest password to remember

Yes, like backup password with a, but we put dollar signs for the s's.

They'll never guess that.

Um, yeah, you know what it is?

It's, it's pa, it's a backup password.

So we just, we put password.

backwards.

backwards.

Yeah.

Yeah.

They'll never guess that either.

Um, and so, yeah, so that's the problem really.

Is it, is it, is it your one stolen or, uh, you know, guessed

password away from, um, having your entire backup system thwarted.

You know, when you, when you listen to the stories that I tell back

in the thing, the backup system is like the one system that is, that

is connected to everything, right?

And which is why it needs to be.

More secure than anything else.

Right?

And so the idea that you could just log into a backup system with a username

and password with no additional authentication at this point, uh,

is absolutely insanity, right?

So, um, there are two things that we can do, uh, to, to, to thwart that.

The first thing that you should absolutely be doing is some sort of

password management system, right?

So that you have, so that your password is on all of the different

pieces of your backup infrastructure.

Your password is unique, not just one password logged in everywhere.

Yep.

Right?

Uh, but then there are two things that we can do to, or do you, are you doing

a, but before you even get to the two?

before you get to the two things,

Yeah.

about local passwords, you talked about password managers, A lot

of enterprises, companies, it's organizations, they use active

directory or other things like that.

And so

Yeah,

to you is yes, we said

typing in a username and password.

As the only way of authenticating is bad.

Uh, you said using a password manager is better.

Yeah.

Mm-hmm.

talk about the two things is do you see things like active directory

or Entra ID or whatever they now call all these things, right.

I can never keep up with

Yeah, it's, it's, it's intra id, but, which is a weird name and

I preferred active directory.

But anyway, yeah.

So,

as part of

yeah, great question.

Um, I think we covered this a little bit in the last episode,

but the, um, I. I see it as another way to hack the backup system.

So if you're gonna use active directory, this should be in a separate domain.

There shouldn't be a way to hack the active directory or interest system in the

primary domain, and then somehow use that as a way to get into the backup domain.

Uh, so at a minimum, there should be a separate.

Username and password, or perhaps if you, you know, if you really want a, a

separate inter ID system for the backup system, there's nothing wrong with that.

It's just most people aren't gonna go to that cost and management to do that.

Um, and you, and it's difficult to do that without also connecting

it to your primary environment.

But I do think it should be separated as much as possible.

So then we're, we're logging in with, with a local username and password.

Um.

And, uh, but if that's all we're doing, then this is a problem, right?

And there are two things that we can do.

Uh, one is sort of now starting to be the old school way, but it's

still probably the primary way.

Uh, and what would that be?

Multi-factor authentication.

Yes, A-K-A-M-F-A.

Um, some people call it two FA, but it's really multifactor authentication.

'cause there could be, there could be different ways.

And why does, why does MFA, uh, help here?

Because if someone knows your password, unless they have

your other factor, they can't

Right,

into your backup system.

right.

Um,

So

and that is,

something you have, right?

yeah.

Yeah.

And most systems, most MFA systems, at least.

You know, good, better, best.

Good.

Better?

Yeah.

Good.

Better.

So, yeah.

So good is modern.

SMS authentication.

We come back to that in a minute.

Better Is MFA, uh, based on like an app or something?

And then best would be a token based, physical token based, right?

Um, why, why do I say?

'cause you and I have had this conversation.

Why do I say modern SMS authentication is at least good.

because it's,

Yeah, it's difficult to say it right?

well,

Yeah,

modern SMS is.

It's better than nothing.

it's better than nothing.

if you look at all of sort of the sim swap attacks and other things that

have happened, it's not bulletproof.

Yeah.

It's not bulletproof at all.

Right.

But the modern, modern SMS, like.

It's, it's a lot harder with, with current hardware, it's a lot harder to

like steal someone's identity that way unless they physically stole your phone

and then you don't have a password and, you know, so the sim swap I think used

to be, I, I, by the way, if, if somebody knows better than me, and again, I'm

not saying this is the way to do it, I'm just saying it's better than nothing.

And I am not putting email on this list at all.

Yep.

Right.

I'm saying, please don't use email.

SMS is still better than nothing, but you should aim for better than that.

Can I make a comment though, even about SMS,

free.

Yeah.

so you're, we're just benching unless they steal your phone.

Yeah.

systems these days allow, like, on my iPhone and I have a Mac,

Mm-hmm.

me to receive my SMSs on my Mac

Yeah.

and then in Safari

authenticated into the Mac.

yes.

But then in Safari it auto-fills that SMS based on receiving it in iMessage.

So I'm saying if someone has access to your laptop, right, that it is

possible because it's linked, or if they for some reason have access

to your, uh, apple account, even

Yeah.

IMEs or having an SMS may, it's still better than nothing, but

Yeah.

Which is, which is where I,

Ty bitty.

I still think it's, I think it's way better than nothing.

I don't, I think there are edge cases, but, and you know, and, and if someone's

determined to hack you, this is definitely not the, not the way to do it.

Right.

So better I would say is, is some sort of OTP one-time password generator.

Um, I like Authy.

Google Authenticator is the most popular.

The thing I like about Authy is that you can back up the configuration and easily

move it to another device if you, um.

Uh, by the way, I almost, when I, and, but by the, the problem with

Google Authenticator is actually what happened to me not that long ago.

I dropped my phone in the water and I. Didn't do the right things.

And so the waterproof phone, uh, died.

And I needed to, I needed to move all my OTPs over to another device.

And with Google Authenticator, you need to do that live.

You, you can't, you, you have to move it.

Right.

And so that's when that happened to me the last time.

That's when I started looking at aie.

And so I liked that, but I, I don't do it often, and so I couldn't

remember what Auie was asking for.

Right.

It was like, it, it just, it just.

It used a, it used a weird phrase that I didn't understand what it

was asking for, and then I remember eventually I remembered, oh, it's just

the password for Authe that I stored my password management system for the

backup, because if I didn't have that.

they

That password, I would've been starting from scratch again.

So, so that's better, uh, because you need both, you need the phone, um, and you need

to be able to authenticate into the phone.

Right.

By the way, au no longer runs on the laptop.

Hmm.

Um, uh, I don't think Google Authenticator does either.

Um, I could be wrong there, but, what was the third option?

I was, so we talked about SMS good, but not really that good.

Uh, OTP.

Oh, by the way, that's funny.

Look at that.

Did you see that pop up?

the thumbs

That's really funny.

I, I didn't even know that was a feature.

Um, anyway, so what was, what's the, what's the third option that we could do?

tokens.

Yeah.

Physical tokens and companies that are truly, uh, concerned,

uh, of network of security.

This is what they do.

Yep.

Right.

Um, and this is something like, uh, UB key, um, the, what are the, what's

the big, the, uh, like RSA, right?

Uh, these are, these are.

One-time password generators, but they're, they're physical.

Also, by the way, there are, there are commercial solutions like, uh,

Symantec VIP for example, my, my bank requires me to use Symantec

VIP to be able to log into.

So MFA is a better way to at least, uh, make it harder for people to get in.

But what is the challenge with MFA?

The problem with MFA is, I don't know about you, Curtis, but every

time you log into something, you get an MFA notification,

Yeah.

and over time someone could mistakenly authorize.

A request, which

Yep.

I think it was the Okta hack, where it was sort of MFA fatigue, which someone was

Yep.

keep getting these requests.

Okay, I'm just gonna accept it.

Yeah, it, yeah.

Which is why yeah, which is why it's not foolproof, right?

Because it's, it still depends on people,

who is the weakest link,

so.

You are the weakest link.

Um, we, we are, humans are always the weakest link in

any, in any security system.

So the next thing and um, which is really the, um, I think going

to become the standard as we move forward is, um, uh, pass keys and

pass keys are, um, you know, they're, they're based on this concept called

Fido, which is fast identity online.

Um, and so.

It's an alliance.

Yeah.

And so basically, you, you, a a a system, a passkey system is

referred to as being Fido compliant.

And the thing is that the, there is no known way to thwart a Fido compliant,

um, authentication system yet.

Um, and so what, what, how is that, don't why you get, I'm just saying

that is the claim that's being made.

So I think even for PAs keys, I think there's good and,

or sorry, better and best.

All right.

Oh, so you're saying there are better ways to have PAs keys?

Yes.

As an

Okay.

passkey,

Yeah.

people aren't aware of pass keys, right?

Passkey store a certificate on your device that then can be used

from just that one website, right?

So it's linked to that.

And to verify your identity,

Right.

And.

Ideally, it should only be linked to that one particular device.

However, you can imagine, that leads to awful user experience If in every

single device I needed to log in.

And so companies like Apple, what they do is they use iCloud key

chain to actually synchronize pass keys amongst your devices,

Yeah.

right?

Which sort of defeats some of the protection in term in order to

provide a better user experience.

So that's why I call

Y

versus the

Yeah,

approach for pass keys, which is it's always restricted to a particular device.

So if someone, for instance, got your pass key off of your device,

somehow exfiltrated it, they're not able to replay it, and other

things like that, that would be my

yeah.

say.

Yeah.

I, I, um.

I think we should, I think we could very, we could, we could get

a passkey expert on here and we could talk about these concerns.

Um, I think that there, I I, I'm, I'm not concerned as much about

that, uh, at the, at the moment.

Uh, I am, uh, the, the, my main concern, like you said, like u

usability as the thing, so I've been, I've been working forward

with, uh, using my password manager.

To store pass keys, right, and it's my user experience interesting is different.

My password manager requires me to put in my one password, my like

global password for the password inventory every time I use a passkey,

which is way less convenient.

Than when I use it to put in a password and it's, it's, it was more secure.

It's also more annoying.

Um, but yeah, so, but the, but the point is that you that look into

a passkey based system, that is definitely the way moving forward.

I think you, you had, you state some, some valid concerns about that and,

um, and perhaps we, well, there's no, perhaps I'm sure that we could do an

episode just on passkeys and on bad passkey implementations, um, that we

could do what we could do in the future.

Um, okay, so speaking of, um, TLAs, right?

Um.

Uh, the next thing is we want to make sure, and I think you should do

this everywhere, but you absolutely need to do it on your backup

system, and that is to disable the ransomware deployment protocol.

Are you gonna patent that?

Sorry, copyright that phrase.

I, I don't, I, I don't, I, I don't think I coined it.

I think I borrowed it from somebody else, so I don't think I could do that.

But yeah, so we're talking about RDP, which is actually the remote desktop

protocol, which is the way a lot of people, and then it is built into

so many operating systems, right?

Um, and it's enabled by default and, uh, it does allow for remote

management, but it also absolutely allows for deploying, uh, ransomware.

because someone can easily access, there's a bunch of flaws with RDP

that may not always be patched, and once someone's in, they basically

have full access to your system.

Yeah.

And um, you know, and of course --some might say, well, what, how,

how am I gonna remotely access thing?

There are other ways to remotely access the system, which are much more secure.

Right.

I would look at some sort of SaaS based.

A system that has a completely different user, you know, authentication so

that you can authenticate again, go you, you, you use pass keys to

authenticate to that system, which then has remote access to the system.

I think that's the way to do it.

I also think like using a Bastion host or VPN and only allow people inside

the network to connect via RDP also

Yeah.

way you could potentially do that.

No, I don't like that.

Sorry, if required.

I understand the,

the systems

by the way, what's a bastion host?

this is kind of like a jumping point that you can use to connect from external

systems, very secure, and then from there you can get access to other things.

Yeah.

I, I, I, uh, I get it.

Um, I just, I, I, my preference, my preference would be to put

this very important, very.

Scary backdoor in the hands of a company that this is what they do for a living

rather than a bastion host that I create and manage, uh, and possibly misconfigure.

like restricting it to VPN and never allowing RDP outside.

I, yeah, I just think you should turn off RDP period.

I just think it's so dangerous that you should turn it off and then use

something other than RDP as your way of, of managing remote hosts.

Um, so.

ways to set up RDP, by the way that most people don't use.

That is true.

Yeah.

So again, good, better, best.

Uh, good is, um, I'd say the, the basically, uh, VPN, you

know, and a bastion host, right?

The better, I think would be a sa, a SaaS based platform best, perhaps

maybe an actual hardware based, remote management system that is based on,

you know, hardware that you plug in.

It's probably also more expensive and more difficult to manage, et cetera.

Right.

Um, just like everything else in our good, better, best, uh, thing.

So the next is, uh, a little thing that we like to call RBAC.

What is that?

RBAC.

Ah, good old RBAC.

This is role-based access control, right?

So making sure.

That you are not giving it.

It kind of goes hand in hand with your least privileged access,

Yes.

Yes.

It 100% does that.

Yeah.

not assigning individual users right?

You're doing things more methodically based on groups, and you are giving

them access only to what they need rather than everything at the company.

Yeah.

The thing that you really need to understand is that the person that has all

power in the backup system is literally the most powerful person in the company

from a data destruction standpoint, right?

Because they can overwrite everything in the data center.

They can overwrite all the configuration in the backup system,

and they can delete all the backups.

Yep.

It is an incredibly powerful position that we often give to the most junior people,

With great power comes great responsibility.

Exactly.

Exactly.

So, um, so yeah, so it is just, this is, um, so role based auth, role based

administration, uh, basically says we're gonna give this role to this person.

And so, for example, you could give somebody the ability to run the backups.

But not configure the backups.

Or they can configure the backups but not do restores.

Right?

Or they can do backups, but not restores, restores, but not backups, right?

So, um, each of those is, um, a different level of trust.

And if you can divide, if you can do division of powers and you can, uh,

you know, this obviously is only gonna work in a, in a large company, but

what you could do, you could also do.

You could create a separate role just for restorers and, uh, you could require,

uh, four eyes authentication for that.

What is that?

Yeah, four eyes is two.

Two eyes, so it's two people in order to be able to do some

action that may be destructive or

Yeah.

It's not like Curtis.

It's not like Curtis with glasses.

It's not, it's not four eyes like that.

It's two.

It's two people authenticating for certain actions, like destructive

restores, changing the backup configuration for example.

Why would, why would we wanna protect changing the backup configuration?

Because a malicious actor or even a rogue admin can go and sort of be like,

Hey, all those servers or whatever else, let's just check change the backup.

So it's only running one day a week and the

Yeah,

is one copy.

yeah, yeah, exactly.

zero

yeah, exactly.

The final thing I'd like to suggest to potentially harden your backup system

is to consider some type of service provider that this is their only role.

Um, so this could, this could be a number of things, right?

So one is you, this could be a an M-S-M-S-S-P, right?

Um, a managed security service provider, like, um.

Our good friend, Dr. Mike Saylor, uh, at, uh, black Swan Security and somebody

who's going to actively make sure that your backup system is configured in the

most secure way, all the way up to using a SaaS based data protection, so that,

um, basically you put the security of the backups in the hands of, of a company

who this is, this is what they do,

But

right.

that they are a reliable, trusted

company, not some random company off the side of the road.

Yeah.

Yeah.

Trust but verify.

Absolutely.

Right.

Um, and talk to them about, uh, what sort of penetration testing that they

do and what, you know, how they conform to, um, you know, ask 'em if they

have a passkey authentication and, and if they stare at you blankly, then

maybe you should look somewhere else.

Right.

Um, talk to them about a lot of things like immutability

and, you know, how, you know.

Could, are you able to delete the backups?

Right.

You know, the, that whole thing we talked about in the last one,

it, I'm, I stand by strongly.

If you are able to delete the backups, then so can the bad guy, right?

So, um, that's not something you want to do.

All right, well, that's a quick list of some additional things in addition to the

10 things that we talked about before, some additional things that you could do

to take the, the specifically the security of the backup system up to the next level.

Um, so thanks Prasanna.

Thanks for having a chat.

no, and

Yeah, no, and thank you to the listeners who left their,

This is

this is why we do things, so please, if you want us to,

you think

if you think that we should be talking more about a specific topic.

things,

things, leave us a comment.

We love to hear and we do respond.

Absolutely.

And we, and thank you very much.

And thank you.

Uh, you know, we have a new person who seems to have discovered our

channel and is commenting a lot.

Uh, and, uh, I, I app who, whoever you are, you don't have your real name.

You have like CX something, some acronym, and so I don't know who you actually are.

Uh, I do appreciate your comments, even if you disagree with stuff that,

that, uh, we say, um, what I will say to everybody is don't be pushing your

company's particular, um, product.

Right.

Uh, because I feel that I need to, um.

I feel that I need to discuss that.

If you're gonna make a claim about your company's product or the way

your company does things, uh, we try, as you can see, hopefully.

See, we try very much not to be pushing a particular product, uh, and more on the

here's how to do things the right way.

Um, and, uh, hopefully your product is able to do things the right way, right?

Uh, and very rarely my opinion.

The problem with your backup system is almost never software or hardware.

It is almost always wetware, right?

Um, the human is the weakest link, as we mentioned earlier,

in every security chain.

And, um, so while, yes, I'm sure some products have more interesting security

features than others, um, you know.

There's no, there's no perfect backup system.

Um, and with a few exceptions, there's no really awful backup systems.

So anyway, uh, thanks for listening folks.

That is a wrap.