Disk Backup Security - Disk Make Things Worse?

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we're talking about something that might make

you a little uncomfortable.

The idea that dis backups for all their benefits actually created

a massive security problem that we're still dealing with today.

I remember when we moved from tape to disk and it was amazing,

but disk backup security wasn't actually part of the original design.

Those backups sitting in ecolon slash backups.

A threat actor can access them and delete them with one command persona.

And I explained why this happened, and most importantly,

what you can do about it today.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup,

and I've been passionate about backup and recovery ever since I had to tell my boss.

That there were no backups of that production database that we just lost.

I don't want that to happen to you, and that's why I do this.

Uh uh, on this podcast, we turn unappreciated backup admins

into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the backup wrap up.

I'm your host, w Curtis Preston, AKA, Mr. Backup, and I have with me persona.

I don't need no tape.

Maldi, how's it going?

Persona.

I'm good.

You know.

don't think it's, I don't need no tape.

I think it is.

What's tape Malaiyandi,

Yeah, you've,

a VHS tape, right?

Or a cassette

yeah.

Yeah.

Linear tape.

Open my friend.

Digital data storage.

Uh, exabyte.

8,200.

By the way, exabyte best named company in the history of naming

companies, and it's not anymore.

I wonder what happens,

to Xite?

not that company, exabyte.

Someone must have bought that name.

Well, surely, but

surely.

on.

sir. I know what you mean.

So Exabyte.

It's interesting.

Yeah.

So some company, oh, a premium domain exclusively for sale

on the brand bucket network.

Guess how much it is?

$150,000.

$212,000.

And I just made it 300 based on my comment.

Yeah, so Exabyte was a, a tape drive manufacturer back in the day.

Um, and it was the first tape drives, well, technically the second

tape drive that I cut my teeth on.

I also worked on quick drives, which were QIC, which were not quick.

Um, the, they were actually quite slow, but

grandpa's talking about tape again.

Um, but you know what?

The, the reason why we're having this conversation, like, and, and

you know, and this isn't like a, we should all go back to tape episode,

but I think it's one of those I think we should at least acknowledge.

We, we can acknowledge the good that disk has done and it has done more

good than harm, but it definitely, at least in one area has done harm.

Um, so let, lemme just, um.

Hang on.

But, but, but, but,

what,

but,

I'm gonna go back.

I'm gonna, but we, we'll get to the

no, no.

button in a minute.

What?

but no, continue.

Oh, okay.

Okay.

So I think that for the modern audience, grandpa does need

to explain tape a little bit.

Okay.

Um, because tape, tape was not perfect.

There's a reason that, that the world went to disk as a primary

backup and recovery target, right?

But it also had a lot of good about it, right?

And, and also it was not as bad as people thought it was, et cetera,

et cetera, et cetera, right?

And, and by the way, more tape is sold today than ever before.

So that's, uh, that is a fact.

But let's just go back to the.

Back before disk based backups were a thing.

By the way, at one point everybody did backups with tape.

Right?

And then at some point there was a company who tried to address the

challenges that we had with tape by putting disk in front of the tape system.

that,

What,

the challenges are

well, hang on, just, I'm just gonna say this.

I'm gonna finish this.

Who, who was that company?

And they were, we said they were crazy.

IBM.

Yeah.

IBM Yeah.

Okay.

Anyway.

Alright, so, yeah, so basically a tape drive, you know, you, you

have a, you have a tape, right?

And you have a tape drive that lays the data down magnetically

on the, on the tape, right?

strip, right?

And it's like a physical strip that

Yeah.

Yeah.

So you have two different kinds of tapes.

You have cassettes and you have, uh, cartridges.

So a cassette for, for those of us that remember cassette tapes, right?

So a cassette tape, a lit. Technically a lot of people are like, oh,

well, you mean like cassette tapes?

Like, you know, like I had of my cassette tape player.

Like

So a cassette tape literally means a tape with two spools.

Mm-hmm.

Right.

But most modern tape drives are what we call cartridge tapes, which means

that they just have one spool, okay?

And the way a cassette tape works is the tape stays entirely inside the the box.

I'm making a really, that's the biggest cassette tape, actually.

V-C-R-V-C-R would be a cassette tape, right?

The tape stays in entirely inside the box, and it just goes

from one spool to the other.

Spool.

A cartridge tape, like LTO is a single spool, and the tape is

pulled entirely out of the, the, the cartridge and spooled onto another

device for use, and then it, and then it's pulled back into the cartridge.

Right.

Um, there are also two ways of writing the data on the tape because, um.

One thing that is important to understand about tape and this'll,

this'll come to resurface, and that is that in order to get a good signal

to noise ratio, which you'd need, you need a good signal to noise ratio in

order to reliably write the data to.

To tape, right, to a magnetic media.

Uh, the tape head has to be going very quickly across the medium, right?

Uh, the, the tape, right?

And there are two ways that we make that happen.

With a cassette tape system like the Exabyte 8,200 a IT, which is the

most, probably the most modern, um, system that was a cassette system.

It's a helical recording.

The, there's a drum that's slanted and it spins, and the tape is

pulled slowly across that slanted head and it writes slanted.

Um.

Stripes across the, the tape?

an angle.

Yeah.

At an angle.

Yeah.

And, um, and the tape is actually going pretty slow and it's the head that's

spinning, that's going very fast.

The industry pretty much gave up on that design for whatever reason.

And they went with the, linear type tape where you have a, a stationary head.

And, um, and then it goes, the, the tape is pulled very quickly

across that head in order to get that high signal to noise ratio.

Right.

Um, but unfortunately that came with a side effect that the tape

was not great at going slow.

Right.

Um, that you,

do, how do you define fast and slow?

Because I think people probably don't have a notion

yeah.

relative speeds of these,

Great question.

So in terms of megabytes per second,

Mm-hmm.

um, that, uh, like a modern LTO 10, which just started shipping,

wants a gigabyte a second, right?

Yeah.

Um, and, and back when I was dealing with things like, it was like we

were talking like 15 megabytes a second, 30 megabytes per second.

And the numbers just, the problem is, in order to get the tapes bigger, you

put the bits closer together on tape,

Mm-hmm.

as the bits got closer together on tape, the tape got faster.

Mm.

And so, and, but the problem is as the tapes got faster and faster,

the ability to give data to the tape was what didn't get faster.

And so you got this.

fire hose to feed it?

Yeah.

And you couldn't, you couldn't do it.

You, you, you know, you needed a fire hose and what you got was

a, you know, a bathroom faucet.

Right.

Exactly.

Right.

Um, and, um, so.

You had this fundamental mismatch between the ability of the tape

drive to go needing to go fast.

It couldn't go slow again.

It had to go fast because of the signal to noise ratio.

And so it couldn't slow down.

Uh, if it slowed down, you get a low signal to noise ratio.

well,

Um,

tape drives do have a low speed, right?

so they have a low war speed,

Yeah.

but that.

Yeah.

not that low,

It's

right?

There's just a

Yeah.

that?

It's not like a megabyte a

No, like in the case of LTO 10, it's probably 500 megabytes per second.

Right.

Which is still really fast.

Right.

And also when you, when you, um, when you match that with the.

Type of backups we were doing, most backups are incremental backups.

Right.

Which supply like a megabyte every minute, right?

You're, you're scrolling through the file system trying to find files or

blocks that need to be backed up.

You're not concerned with how many of them, uh, you know, right.

So you, you had this fundamental mismatch between what was happening

on the supply side and what was needed on, on the drive side.

Right.

And when you have that, you end up doing the shoe shining thing where you're,

the tape is going back and forth to try to keep up with this slow, uh, speed.

It, it cannot, literally cannot write slow.

So what it's doing is it's, it's like imagine a car.

Imagine trying to put people into a car that only knows

how to go 60 miles an hour.

Right.

And what it's doing it, it's going up to 60 and then, you know, you're

throwing people in the car and then it's backing up and you know, it, it is

just crazy what it, what it was doing.

Right.

It was wearing out the tape.

It was wearing out the drive.

It was making the drive unreliable.

And so,

Everyone complained.

Everyone complained and we started, uh, looking at a way to use disk

as a way to ameliorate that issue.

Right.

Which goes back to your thing about IBM at the start

Yeah, exactly right.

And IBM was the, uh, the first company with what was originally called A DSM.

Uh, and then it became called TSM, and now it's called Spectrum Protect.

Right?

Um, that this idea of disk staging.

So we're gonna put the, we're gonna do all those incremental backups and

put them up to disk and then we're just gonna spool them over onto tape.

When they first started doing it, we all thought they were crazy

'cause this was so expensive.

This then became less expensive.

Um.

And And then what happened right around 1999, we started partying like it's 1999.

Because someone invented what?

Deduplicated

Deed duplication.

Yeah.

And.

This idea where we're going to find the duplicate blocks of data between

different backup sets and we're gonna, we're just gonna put pointers.

And the pointer thing doesn't really work on tape.

I mean, it can technically work, but think about the idea of you having to

load a hundred tapes to restore one file.

And that's why DDU doesn't really work on tape.

So.

Uh, we went from using disk staging and then we more and more

people started using d, you know, deduplicated disk storage, right?

Avamar was the first company I remember working with right

originally called Undo with two O's.

Really funny that a company that, that Ddu had two o's in their company name, I

think they got too many, too many Razrs, and, and so they changed it to Avamar.

but I think one of the keys, right, that DDU became so popular, like you had

mentioned, it's the cost of disk, right?

disk was

Yeah, yeah,

the difference between disk and tape was significant.

That

yeah,

were like, there's no way

yeah.

backup purposes, I can spend millions of dollars on this.

It, it was literally like two orders of magnitude cheaper.

Right.

And people were like, well, we'll, we'll just deal with it.

And so what this did was it brought disk down.

It's still nowhere near as cheap as tape, but it made it.

Doable.

Right.

And, you know, Avamar was the first one I remember working with back in 1999.

And then, um, uh, data Domain was another big one and they did really well.

And I worked with a number of companies, uh, along the way that did either target

site Dedupe the way data domain did, or source I ddu the way Avamar did.

And, and, and it, it basically made disk feasible.

It made it, it made it not be crazy expensive.

There was another thing that happened, um, that another technological change

that happened right around the same time.

Do you remember what that might have been

that helped make disk backup media more affordable?

Oh, this is like the nearline disks.

Yes.

So what, what do you mean.

Oh, this is because previously enterprise disks were all fiber channel,

Right.

And then they started looking at serial a TA disks.

Yeah.

Yeah.

to lower the cost because these did not need all the performance

of your fiber channel disk.

This is just backup media.

It's your secondary copy.

Right?

You don't

Yeah.

on very expensive storage.

Yeah, so when you coupled the fact that they were using Sada disks with, you know,

less expensive sort of almost consumer grade disks with, um, deduplication,

you put those two things together and backup disks suddenly became way

more affordable than it used to be.

And it came with some really great features.

I'd say one of the best feature a co a couple of them.

Right.

So, 'cause I, you know, I'm, I say good things before I say like how bad it was.

Right.

So one great thing is that it's super easy to do backup verification.

Right.

And, you know, and, and I remember when Veeam came out with, uh, their Sure.

Backup, I think that's the name of it, their Sure backup feature

where you could create a, a recovery group and you could, um.

Um, you could automatically test your backups without

actually having to do a restore.

You, you could basically run your, you could run your VM

from your backups, right?

was an, and also they had the ability to do, to test your backups.

I remember

Yeah.

feature where it was like, Hey, we will spin up everything in

an isolated environment for

Yep.

to bring it up, to test everything, to make sure your backups are actually,

uh, restorable and good to go.

And then we'll spin everything down and you can continue on your way.

Yeah.

And that is only possible with disk.

Right.

Um, and then another thing that, that, that sort of came as a, and

these are all things that modern day users, I think just sort of assume.

That they're there, but they, they're new to those of us that have been

around a few years, and that is the idea that I could replicate backups,

Yep.

right?

So we could have onsite backups and offsite backups without.

Handing tapes to a man in a van.

'cause that's the only way we got data off site.

We made a bunch of tapes, we copied 'em to a bunch of other tapes,

and then we put 'em in a box and we handed it to a man in a van.

Now, because we've really reduced, not just the total storage that we need to

store backups, but the daily amount, like it was less than like a half a

percent of the size of the environment each day, then we could replicate

those backups and so we could have an onsite backup and an offsite backup.

A hundred percent automated.

Right?

Which you may recall in the, the episode that just aired, uh, today as we're

recording this, uh, automation, right?

You can have a hundred percent automated backup.

So backups are so much more reliable than they were back in the day.

What?

Yeah, sure.

Also, the notion of virtual synthetics.

Yes.

Why?

Why don't you talk about what, what's a virtual synthetic?

so like you had alluded to earlier, Curtis, right, with tape, you sort of

had fulls and incrementals, and in order to restore your data, you had to always

go back to the full and then replay all your incrementals till you got to

the point that you needed to get to.

Yeah,

with storage deduplication, you could actually create each

copy being a virtual full copy

right.

your data, such that you only need to go to one copy in order to restore the data.

You don't have to go do all the replaying.

It significantly cuts down on your recovery.

Yeah, agreed.

Uh, and, and there, there were two ways to do that, right?

You could do it through the software, the backup software, where you basically

just sort of create a new full by.

Copying, you've got all the stuff all in one place and you can just

create a new fold by copying it.

But then there were, there were products like data domain, um, that, uh, that they

would just do it with pointers, right?

And, uh, and so you don't, you could create a new fold that, that, that

behaved like a full and look like a full to the backup software, but

you didn't actually do any data movement and that's awesome, right?

Um, now it will be.

Just as I take a stab at this, it will be the most fragmented, full backup

you've you've ever seen in your life.

'cause the bits are all over the place, right?

Uh, but it will behave like a full, and we don't, you know, again, backups are so

much better than they were when, you know, back in the day because back in the day.

The best I, the best design I had back before we went to disk was a

monthly full, a weekly cumulative incremental or differential, depending

on which product we're talking about.

And then a daily incremental.

So a typical restore you would restore the weekly full you would restore,

I'm sorry, you would restore the monthly full, you'd restore the latest

weekly differential, and then you'd restore six, um, incrementals, right?

And.

If any files changed multiple times, you were actually restoring

the same data multiple times.

Now we know exactly what the latest version is and we

can just go straight to it.

Yep.

So

But it all sounds amazing.

why are you, uh, knocking on disk then?

Yeah.

RM minus R star, that's why

DEL star, star.

I I Is there, there, there's a, a recursive option to delete,

isn't there slash r or something?

Or is it back slash r?

but, but, but, but, but, but, but, but,

Yeah.

okay, so I agree.

There's no agreeing or disagreeing.

It's a fact.

me, let me, it

Okay.

but are techniques with disk storage to help prevent

Yes,

of situations from happening,

there are tech techniques.

Yeah.

Such as immutable storage or setting like object lock or whatever the

mechanism is that the system supports

Yeah,

order to be able to prevent action

there are many technologies that we have since invented in order to address,

to solve the problem we created.

Uh, I don't dunno if you've ever heard, uh, you know, NIT

we never solve any problems.

We just move them right?

Um, we definitely created new problems, and the worst, I think the worst

sufferers of this, and again, I, I don't want to pick on my friends at

Veeam, okay, but Veeam customers and products like Veeam, it's not just Veeam,

Mm-hmm.

basically the default setup, you put the backups in, like e slash backups,

Yeah.

and then a a, a threat actor comes there and says.

Look at that.

E slash backups, R minus R, you know, well, I guess delete, delete

startup star slash RI don't know.

I should really look that up anyway.

I know there's a recursive option, right?

Or they, or if they have console access, they just right

click on it, delete it, right.

And then empty, empty, uh, recycle bin.

Yeah.

Yeah.

And so that's the real prop.

And, and.

I remember, um, you know, I remember this, um, when I was at Veeam on many years ago.

Um, you know, Veeam really acknowledged this, right?

It, it was a difficult session, I think, for them to, to sort of tell people,

Hey, this is a, this is a threat and we have this 'cause you remember,

you remember the, what was the Veeam?

The, no, what was the, no, no, no.

What was the Veeam uh, motto?

We make it easy.

It just works.

Right?

It just works.

Right.

And so they were like, it just works.

Having said that we need to, we need to do this thing.

Right?

And, and they've done a good job at responding to this threat.

Right?

But, uh,

It takes

there, there was a, there was a time there where, you know,

a lot of people were attacking.

Windows based backups, the biggest of which, uh, is Veeam, I still think

they're probably the biggest Windows backup software, uh, in terms of

number of installations for sure.

Um, and so we just need to acknowledge, I I, I, that's my goal of this episode

is I need you to understand the risk that your backups are under, right?

We talk about this in other episodes, that backups are the number one target.

Your, uh, of your threat actor, if they get, uh, an initial access, the first

thing they're gonna do is try to figure out what your backup software is, and

they're gonna try to take it out, right?

And you can, you can address this, but again, the first thing we

have to admit that we're powerless over, you know, step one, right?

Admit that we're powerless over threat actors, uh, and, you know, appeal to

a higher power of, uh, immutability.

Of calling it Veeam, call it like networker or Avamar

or something like that,

Oh, like re rename, renaming the folders.

and your process names.

Rename it like, definitely not backups.

E like, don't look over here.

Um, e slash um, pork recipes.

Um, you know, this is what, what do we call, what do we call

that, that, that there are people that do that kind of stuff.

Well, I was gonna say security by obscurity, but you are, you are correct.

Obfuscation is definitely the i, the, the formal term, and you should do that.

Right.

These are, this is on the list of things that you should do.

One of them that you've heard me talk about is, uh, and we're gonna talk this

more about this in other episodes, but.

Is to get the backups out of user space.

It should not be in eco and back slash backups or slash backups,

whatever, whatever os you're running.

Is the other thing, also, don't run your backup software as root.

Well, but you kind of need to, right?

It needs to be by the way, your, your Linux is showing your, your Unix.

Uh.

Hmm.

Like, you know, the fact that you're, well, we're both, we're both, you

know, Lennox recovery, uh, recovering Lennox people, um, of course, but

do, do you remember the world before?

Lennox?

No.

Okay.

Lennox was the thing by the time you were okay.

Because I remember the world before Lennox.

Yeah.

I remember the world before Windows too.

I do too.

Yeah.

And I remember like.

all the fancy, nonstop kernel systems my dad used to work on.

Oh yeah.

Yeah.

Your dad's old.

Um, tell him I said hey.

Anyway.

Yeah, so there are, there are a number of things that we can do and we'll

talk about them in other episodes, but just a, just a quick idea is one is

to get the backups out of user space.

So if you can see your backups as econ back slash backups, this is a problem

Anyone

the.

see that too.

Yeah, exactly right.

Um, but if you're able to put it on, um, basically immutable, truly immutable

storage, meaning again, the standard is if you can't delete the backups,

then they can't delete the backups.

If you could delete it, then maybe they can.

Right.

Um, but, um.

You know, immutability, immutability, immutability.

Right.

You know, it keeps coming up in, you know, every episode, but

it's like, if you don't have your backups on a truly immutable storage

device, um, then this is a problem.

Right?

And you're just, uh, leaving the, it's sort of like, if we're being chased by

a bear, I don't have to outrun the bear.

You just have torun me.

I just have to cover from you.

Okay.

And, and what, what, what does that have to do with this?

You don't necessarily have to beat every, you know, threat actor.

You just have to be less, uh, appealing than the other person.

Right.

Um, and um, so there, there are techniques that we cover.

Uh, by the way, I haven't mentioned, uh, my upcoming book, uh, how to, um.

that's.

Oh, the, the one that, yeah, the one that's right there.

Um, the, uh, learning ransomware responsive recovery that I wrote with

Dr. Mike Sailor, who is a blue team expert, uh, fighting a good fight

out there in the trenches every day.

And, um, uh, coming to a shelf near you.

Uh, you can read, you can, you can actually see the early version if you're

a o' rally learning platform, uh, person.

You can see that right now.

Uh, and then the, the.

Regular version will be coming out in March of 2026.

So, all right, so that's the thing.

disk is awesome, except when it's not.

Um, just like tape is awesome, except when it's not.

yeah, you need to understand the limitations of different technologies

and use it in the appropriate way, and then make sure you're

able to cover the weaknesses.

Man, man's got to know his limitations.

Oh, Curtis.

Dirty Harry.

Love that movie.

Um, I can't remember if I shot five or six.

Do you feel lucky?

Anyway, sorry if, if you don't know what I'm talking about, that would be early.

Clint Eastwood, dirty Harry, uh, set.

Where?

Come on.

All the dirty Harry movies are set.

I don't know.

San Francisco, dude.

Oh.

Yeah.

Um, what the 44 Magnum, this is the 44 Magnum most powerful handgun in the world.

Blow your head clean off.

So question you gotta ask yourself is, do I feel lucky?

I actually re-watched that just not that long ago.

what I was gonna say.

You know those lines too well.

Well, I wa, I mean, I watched that movie many times and that's one of the most

quotable movie lines, you know, ever.

Um,

is amazing that you have not seen

what

Kung Fu Panta,

I have seen Kung fu Panta.

but not enough.

You can't

Not enough.

Not as many as you, how many times have you think you've seen Kung fu Panda?

The 200 something.

I, I don't, I don't.

I don't get it anyway.

All right.

People, disk based backups are great, but, uh, they do have one

major limitation and we made things better, but then we made things worse.

And the, the threat actors are coming for you, backups.

You've got to make sure that they can't get to them.

All right.

Uh, thanks persona for the chat.

No.

Then make sure you pick up Curtis's latest book.

Persona's name's in it.

Um,

that is a wrap.

The backup wrap up is written, recorded and produced by me w Curtis Preston.

If you need backup or Dr. Consulting content generation or expert witness

work, check out backup central.com.

You can also find links from my O'Reilly Books on the same website.

Remember, this is an independent podcast and any opinions that you

hear are those of the speaker.

And not necessarily an employer.

Thanks for listening.

I.