Understanding Backup Levels (Backup to Basics Series)

Hi, and welcome to Backup Central's Restored All podcast.

I'm your host, Debbie Curtis Preston, a k a, Mr.

Backup, and I with me, a guy that I'm hoping knows a little

something about plumbing.

Prasanna.

Malaiyandi, how's it going?

I am good, Curtis.

how's that?

YouTube?

How's that?

YouTube, the, the YouTube knowledge that you have, does it cover plumbing

kind of, sort of.

Unfortunately it's not something I'm as familiar with, but depends

on what you're looking to do.

Yeah.

I gotta replace tonight.

I gotta replace a couple of valves.

Um, Shut off valves.

Um, one of them, I, one of them is on the, the, it's the one that goes

behind my fridge and in order to like, it's like I, I can't easily,

it's a, it's a, what do you call it?

It's the, the threaded, no, it's not a compression fit.

It's the, it's a threaded pipe.

Like, uh, I really gotta get in there and get a hold of the, the upper part

of the pipe, uh, in order to hold that pipe still while I'm undoing it.

Cuz it, my first attempts at it did not, I don't want to be

cracking off a pipe up in the wall.

Yeah, I was just gonna say that's, yeah,

the last thing you want to do.

I was thinking for the shutoff though, I'm assuming you took the handle off

The handle

Prasanna Malaiyandi:

to give you more space

it?

No, there's no handle.

It's just, I mean, the handle's like integrated.

It's really small.

It's not a space problem.

It's just, uh, it's um, you know,

Prasanna Malaiyandi:

Or they also say heat.

Yeah, give it

So when I saw my, oh, so when we had our water heater

replaced, I was watching cuz I'm always curious to see what people do.

And our pipes were really, really bad at old.

because the last time our water heater was replaced was like 15 years ago.

And so the guy's like, you just heat it up really, really hot and then

you tighten it first to break the stuff off and then you loosen it.

Oh, interesting.

Yeah.

Maybe I'll see.

I'll take a look at that.

Yeah.

Uh,

took like in a settling torch or a butane torch

and then heated up nice and hot, and then he tightened it just a little.

what, I'm not gonna be able to do that.

You know why?

There's this

Oh, the drywall.

Oh,

W. Curtis Preston:

Plastic, worse than dry.

yeah.

Um, so hey, we're gonna talk about, uh, we're, you

know, in our continuing, um, backup to basic series, you know, we talked

about, like recently we talked about a difference between backup and archive.

, in a previous episode we talked about protecting your backup data.

This one, um, you know, this one is, um, it's about backup levels, right?

And, you know, it's, honestly, I'll just say this isn't the most

exciting topic, Um, and maybe, maybe, we'll, maybe we'll make it a short

episode if it's not interesting.

You know, we're not gonna make, we're not gonna make a long episode cuz

this isn't gonna be one that's gonna be, uh, littered with stuff about

cyber attacks and things like that.

Backup levels

Prasanna Malaiyandi:

really straightforward.

really straightforward.

Yeah.

Uh, I do want to help for people that sometimes struggle with

the different types of backups.

Um, I want them to help them understand.

I, I'd say that, you know, if you just go and I'm, uh, throughout

our usual disclaimer Prasanna, and I work for different companies.

I work for Druva, he works for Zoom, and the, this is an independent podcast.

It's not theirs.

And it's, uh, the opinions that you hear are ours.

Uh, be sure to rate us, uh, go to your favorite, um, pod catcher and

uh, scroll down to wherever they got the stars and give us all the stars.

And the best thing is if you give us a comment, those are always nice,

especially on Apple Podcasts, which is.

the primary way that people listen to us over here.

And, um, also if you wanna join the conversation, I am w Curtis Preston gmail

and also WC Preston for now on Twitter.

I dunno what's

do you have a master on?

Twitter man, I don't have a Macedon account.

Uh, I am interested in that.

We'll see if, I don't know if the average person can, can

I don't think so.

From what everything I've heard.

Yeah.

It took me a minute, right?

Twitter, you know, you sign up for account, you're done right here.

It's like, well, what community do you want it attach to?

I'm like, oh, it's so complicated.

I don't wanna know what community, you know?

Um, we'll see.

But, uh, anyway, at WC Preston on Twitter, maybe I should sign it

for my, at WC Preston on Mastodon.

Make sure I get that.

I think you were throwing out the disclaimer though,

because for this chapter, if people want to actually read the chapter and

understand all these terms, right.

Druva is offering a free ebook version.

I, you know, that wasn't why I was saying

it, but, we'll, we'll do that.

So we are discussing my book, modern Data Protection.

There's a picture of it with the little armadillo for those of you watching on

the video version on backup central.com.

And uh, you can get a free ebook copy of it by going to druva.com/ebook.

That's d r uva.com/ebook.

And.

while supplies last, or while the contract lasts.

So, you know, this, this, this episode will live for a long time.

If, if you go there and it's not there, it's not my fault.

So, uh, first off, you know, again, this is pretty basic, but.

You know, you should know what a full backup is.

Obviously.

That's it.

It backs up everything, right?

Typically.

Well, not typically.

You always have to do a full backup before you can do any,

any kind of incremental backup.

That's the, that's the next type.

The, there are amazingly, a handful of different types of

the traditional incremental

but before you get to incremental, so.

Yeah.

Even for fulls, are there different types of fulls?

Well, there's not different levels for fulls, but there

are different ways that fulls get done.

Right.

Like you, like we could be doing a file system backup or we could be doing a block

level backup, but either way you're still backing up all the things, all the bites.

Yeah.

Um, the, uh, and it's, it's the full is the thing that puts the most amount of,

of stress on the thing being backed up.

Right.

The, this is why we try to limit them whenever.

, and this is why, this is why backups broke when VMware took off because everybody was

just doing full and incremental backups and they weren't coordinating them.

And you know, it was just you,

lot of resources being used.

Yeah, exactly right.

And um, if you do, if you do a nightly full backup, or let's say the weekend

full backup, and you do 'em all on Friday, And you've got 20 different

physical servers, no big deal.

But if you do 20 full backups on 20 VMs that are all on one

physical server, this is a problem.

Right.

Um, so I've got a couple different types here.

There's the, the typical incremental backup, which is, it basically,

it backs up everything since the last backup of whatever kind.

And typically what what you do, and this is with backups that have

levels, you do a full backup, then you do a series of incremental

backups, and then you do another full.

right?

That's the, that's the sort of the typical way.

There is something called a cumulative incremental backup.

Um, I prefer that to the term differential backup, and I can

talk about that in a minute.

But a cumulative incremental backup is essentially just that it backs up

everything that has changed since the last full different, um, different product.

Uh, do different things.

Uh, the differential backup is a term that you see a lot in windows and, um,

I, I, and I, I don't use it because I find the term, um, the, the, because

depending on where you ask the term differential means different things.

And so I, I don't like it.

Right?

So I like the term cumulative incremental.

Even if your backup product doesn't use it, just learn

what, whatever they call it.

What?

wait.

So as far as I understand it, , you have the thing that's like full then.

So say I do a full on the weekend, right on Monday, I do, uh, incremental,

Uhhuh.

right?

Which will be the differences for whatever happened between Sunday and Monday,

Yeah.

right?

Tuesday I do another incremental

Yeah,

right?

That'll be the differences between Monday and Tuesday.

yeah,

And then Wednesday I do another incremental.

That's the difference between Tuesday and Wednesday.

Now, for different, uh, for cumulative incrementals, it's, I

did my still, I'll do my Sunday, full Monday I'll do my cumulative.

Incremental, or yeah, incremental, which will just be

Sunday to Monday, differences.

Tuesday, when I do the cumulative incremental, it'll

be from Sunday to Tuesday

Correct.

and Wednesday when I do a cumulative incremental, it'll

be the differences between Sunday and.

Correct.

And so what, what we used to do back in the day was I would do a monthly

full weekly cumulative incrementals and daily incrementals, right?

So that way, even if I'm 28 days into the month, I would only need

three backups to get the job done.

. Right.

Um, I, I don't use the term differential.

Differential is what a lot of places call accumulative a criminal backup.

But again, different products use it to mean different things, so I don't use it.

Um, but interestingly enough, what we haven't talked about is backup

levels, which is the term that, again, you don't see too much these days.

Have you, have you, are you seeing it?

Very, very rarely do Does anyone talk.

The only place I've seen levels really is when you're talking

like database backups, but most.

what?

That is true.

Right?

Yeah.

But most backups software itself no longer talks about the levels.

Right.

It's either fulls or incrementals.

I think it was too complicated.

Yeah.

The basically if to, to do.

To do a full on a weekend and then daily regular incrementals, you would

do a 0, 1, 2, 3, 4, 5, 6, 7, right?

Well, seven.

And then maybe, um, so then, and again, this is, this is impossible to do without

a whiteboard, but if you did, um, a zero followed by all the way up to the five, if

you did a one again following that five.

you would get all the changes since to zero.

Hmm.

So it was accumulative.

Incremental, right.

But what some people would do is they would do this, um, this,

there, there's, there was this thing called towers of Hanoi, which,

um, it, it comes from the game.

like

Yes.

Yes.

It's like the thing with the game.

Yes,

three.

Yeah.

Comes from that game.

Um, and so the, and again, this is really, don't try to understand this, I'm just

putting it out there just for, this is for the true backup anoracks, so who's gonna

Daniel Rosen Hill.

We're looking.

Daniel Rose Hill.

We're looking at you.

Exactly.

So, um, let's see.

So what I've got here is, , um, a towers of Illinois Schedule 0 3 2 5 4 7 6.

And what it does is it, is it like every file that's changed

ends up on two different, um,

Mm,

two different, um,

days.

backups.

Yeah,

Um, and um,

you don't run the risk of if a backup is bad,

you've lost the file, which could happen if you just had a full and you

never did another incremental again.

Yeah.

Or sorry you never did another full again, I, I

think one thing we forgot to mention Curtis though, is the benefits of doing

a a, we might have talked about it implicitly, but the benefits of doing a

cumulative incremental versus a normal incremental when it comes to restores.

Well, you know, here's the thing.

The benefits were much bigger when we were using tape.

Hmm.

I, I think that as, and, and what we're gonna discover is

that this is why a lot of this has, I think, gone by the wayside is because,

because when we were using tape, you'd gr you'd grab the full tape, you would grab

the cumulative incremental tape, and you would grab any incremental since then.

So you've got maximum like eight tapes if you did a full,

followed by 30 incremental.

, you would, um, you would need 28 tapes, you know, at the end of the month.

So this way you only need like a handful of tapes.

And what it allowed you to do was it allowed you to do

less frequent full backups.

Because it was a, it was a, an extra cost.

You're like, well, by doing a weekly cumulative incremental,

I could do a monthly full backup instead of a weekly full backup.

Thus really reducing how big the backup system needed to be,

how many tapes I needed to use.

Um, And, and by doing that cumulative incremental, it made

the restore much, much faster.

I would argue if you have a decent backup system today and you've got a monthly

full and 30 days of incremental, or even 60 days of incremental, the difference

between restore times, um, depending on how you store the backup data, shouldn't

be as significant as they were back in the day when we were loading tapes, cuz

every tape loaded was like two minutes

Yeah.

But,

to the first.

but I do wonder though if when you are doing

those restores, depending on what you're restoring, like I could

see a file system maybe not being a big deal for doing the 60 days.

I'm just wondering like database backups, right.

To do those incremental restores.

Yeah.

Um, I think that's a val, I think that's a valid.

Uh, concern, right?

This is why to this day we still do more frequent

Fulls that need it.

on, on database backups.

Right.

The, that's, that's a good point.

Um, and since a significant portion of the world is on, is,

is structured data then, uh, yeah, that's a really good point, right?

So, um, the.

So now we've got a just some different types of, so we're done with the levels.

Now we're talking about different types of backups.

What about a block level incremental backup?

What is that?

Prasanna Malaiyandi:

Block level, incremental.

Prasanna Malaiyandi:

Are you referring to like database slash v.

Prasanna Malaiyandi:

Where you're just copying the differences in whatever the application

Prasanna Malaiyandi:

block size is, rather than block backing up the entire image or

Prasanna Malaiyandi:

the entire file potentially again.

Yeah, so basically this is the difference between, and

again, this really only applies if you're looking at things at the block level,

which you tend to do at the database, but there are some file systems that

are so dense that backing it up at the block level would be faster than

backing it up at the, at the file level.

Yeah.

And so you can do a block level incremental, basically.

The point is, when I'm doing an incremental.

A file system in increment, a file system incremental.

I'm backing up the files that have changed since the full, if I'm doing

a block level incremental, I'm backing up the blocks that have changed since

the full or since the previous bite.

Right.

And then here's something that hopefully you'll know something about, and it's

called Source side deduplication.

This is another way to do incremental.

What can you tell us about.

Yeah.

Source side de-duplication.

So when de-duplication appliances came out, um, what you ended up happening

is whenever you would do a full right, you were talking right Curtis, about

doing weekly fulls, daily incrementals.

Every time you did that full, probably 98% of the data was the same.

So on the source you would read it, you would send it over the network, it would

go to your de-duplicated appliance and it would throw away 98% of the data.

And so source side de-duplication is, Instead of sending all of

that data to be thrown away.

On the appliance, you basically split the deduplication algorithm

such that you're running some of the processing on the source.

So you figure out what is duplicate, usually at a smaller block level

or some other, uh, granular size and only send the unique blocks

over to the target appliance.

And the rest of the data is sort of reconstructed, if you will.

There are different methods you could look at.

It's sort of rebuilt because you still need to account for those pieces

of data, even though you haven't technically sent the data over the wire.

Yeah, so if we think about it, like the only reason

ever to do an incremental backup is to save something, save bandwidth,

save CPU time, save storage.

And what, you know, what you're talking about is even when you had

target site dation with like data domain and the like, you, you.

, you save storage.

You, you can do full backup every day and you save storage.

But you were still using up all that CPU in that network by doing

Source I duplication and making that decision before you send it.

You save CPU time, you save bandwidth, you save storage, you save all the things.

Right?

Um, source ID is not as prevalent as targets.

I dedup.

There's, there's only a handful of companies that,

that do it, but uh, they are,

Yep.

Um,

it does require more integration into the backup product on the

source in order to take advantage of it.

exactly right.

Uh, for the record, Druva is a source site duplication product.

So the next thing we talk about here is called synthetic full backups,

uh, which is and interesting.

I remember when this concept first came out, again, we're trying to

minimize the impact of the backup.

on the system, we're backing up.

So the idea was we already have all of the bites necessary to create a full backup.

Why do we need to go get, you know, we're doing a, we're doing an increment.

We're doing the next backup and 99% of what's on the system we

already have on disc somewhere.

This is really a disc thing.

Uh, we did it on tape too, but it was a lot more work.

Yeah.

So we'll go get the 1% that has changed.

and then we'll create a full backup by copying the stuff we already

have and the stuff that we just got.

So we create a synthetic full.

Um, have you, have you run into those out in the wild?

no, it's very common, uh, especially when

you think about VMware images.

Mm-hmm.

depending on how it's being backed up, a lot of folks

would back up a VM as incrementals and then, because like you said, most of

the data is the same, they would sort of synthesize a full image on the target

side in order to have that full copy.

Therefore, when you need to restore, you have that.

What image you could pull from or would a lot of people start doing is you take

that one image and you can now mount.

, right.

And access it directly.

Instant access is what simple some people call it.

So you can spin up the VM from your target system and then

vMotion it over or whatever else.

So you have shorter RTOs than trying to first restore the

data before you spin it up.

and there, and there's a couple different ways that

you can create a synthetic full.

You can either do it by copying, which is what I was talking about before.

There's also this concept of virtual synthetic full,

where you can just create one.

, you know, magically, right?

Like the, I know that again, data domain I know supported

that on the back end, right?

That if the backup product could say, you have all the bites you need to make a

full, why don't you just put 'em together and, you know, everything's beautiful,

You know what else does it too,

what's that?

Oracle?

When you do incremental merch,

That's right.

Incremental merge creates.

Exactly, exactly.

Um, but here's the thing.

Wherever we can get rid of full backups, I think we should get rid

of full backups, Right, because they're just the dumbest thing.

Right?

They're just, they're just, they're, we did 'em because that's the way we did 'em.

And we still do 'em because that's the way we used to do 'em.

Uh, so I'm a fan of, go ahead.

I, so I, I wanna be careful, right?

I think we should.

Get rid of the concept of backing up full copies of data every

single time or on some increment.

I a hundred percent agree with that statement.

Isn't that what I just said?

No, no, no, no, no.

But, but I don't agree with, but there's still a necessity to have the

equivalent of periodic fulls, right?

Something that represents a full backup.

Now it could be stored on de-duplicated systems to save storage space, cuz

most of it's the same, but I'm just saying you don't wanna go the route of,

it's just a full once a year and just incrementals physically stored on disk.

So using virtual synthetics or some other mechanism to end up with full copies

periodically, I think is beneficial.

So I would argue that that's what Incremental Forever

is a true incremental forever system.

Right.

That basically you, which is what, what I was about to talk

Okay.

Okay.

Um, and that, you know, the idea is store each backup

in such a way so that a, any restore from any backup, you just, the, it's

stored in such a way that it looks like a full right and a restore from it

behaves like a full, I, I don't think we need to keep making fulls or make or

synthetically creating an occasional full.

I think that if you design it from the beginning, so that you do

incremental forever, and then you store that data in such a way that

every backup is essentially a full

Yep.

from, from a way it behaved the way it behaves

during a restore, then I think that's as good as it's gonna get.

Right.

I, I think that works in the cases of file

systems and virtual machines.

I do think in the case of databases, it is a little database is always that oddball.

I had the caveat back earlier where I was

like, wherever we can, right.

Um,

Databases have so many nuances with fulls

and incrementals and log archived, redo logs, and all the rest that I

think, yeah, for everything else, a hundred percent agree with you, Curtis

Yeah.

Um, and there are, there are a handful of products that back up this way

that basically, said full backups is so from the tape land, right?

Except for database backups.

Right.

Um, generally speaking, there're gonna be products that have come

out in the last 10 ish years.

Right.

Um, products that have been around for 20 years.

They're gonna have spent time in the tape land and they're gonna have

parts of their architecture that are left over from the tape plan.

I also do wonder if some of that is just the

people who are banishing the systems.

It's something that they are used to, and so change is hard, you know?

is hard.

Inertia is something.

Yeah, exactly.

And I would say like if you, if you're doing source side dup with incremental

forever, I think from an efficiency perspective, assuming that the way you

store it, everything behaves like a full, an efficiency perspective, I don't see it.

I don't know how it could get any more efficient in that.

Right.

You could maybe argue the block level incremental forever where you're.

Right.

So there, there are like c d P and near CDP systems that's continuous data

protection where they're doing block level changes and they're replicating that.

That's also quite efficient.

It's not source I, it's still, it's still, so I would say,

so I'll restate my statement.

Either source iddu or block level of replication and then incremental

forever stored in such a way so that everything behaves like a full.

Yeah.

you know, that includes a handful of

products , one of which is Druva.

But, um, you know, there, there are other products that behave like that.

Like, you know, your old employer, NetApp, right?

That was a block level incremental replication.

Um, yeah.

Um, so I'm gonna throw out one, um, this is gonna be one of our shorter episodes.

I think.

We'll see.

We're almost done.

I'm gonna throw out one, like, uh, what's the cranky old man?

Comment

get off my lawn.

this is the archive bit and Windows.

I hate the archive bit and windows.

Um, so what is the archive bit?

It's a flag on every file that when your backup software backs up

that file, it can unset that flag.

If the file is new or it changes, the archive bit is set.

And then when the flag, uh, when the back, the, uh, backup software backs

it up, the archive flag is unset.

Why.

This is the old way and it's still the current way.

Uh, not looking.

This is if you're doing a file system level backup in Windows.

Um, I think it's so dumb for multiple reasons.

The first of which it's name,

Archive.

it should be the backup bit.

Yeah.

They called it the archive bit anyway, but that's not

really my problem is you can't, if anyone comes along with like, any, cuz there

are a number of like third party tools that can back up your Windows system.

If you run a third party tool to create your own backup, like let's

say you, you know, a lot, some people don't trust the backup system, right?

So if you, if you being an admin go and decide to make your own backup

and you use something that uses the archive bit, it will clear the

archive bit on all the files that have changed since the last full.

and then when you go, when the real backup software comes in, it actually

not gonna find,

until the next full backup.

Prasanna Malaiyandi:

and I bet people don't

problem with it.

Yeah.

Yeah.

And of course people don't notice it.

Um, and so I'm gonna go back to the story, um, that I talked about.

I don't know what the, the one about me being kidnapped, was

that in this episode or was

That

was a previous.

Okay, well I hinted at this story from um, this other,

from the other episode, but here's the full story cuz we got time And you know

what, if you don't want another old backup story then you know, thanks for

joining and uh, see you next episode.

But this is a good story and if figures in a good buddy of mine, Rob Worman.

I know Rob listens to the podcast.

, he figures into this story.

So I was at this large, you know, national entertainment company and I was helping

them to redesign their backup system.

We were doing a massive redesign.

And one of the, and one of the things, you know, like we pushed them from, you

know, weekly folds to monthly folds.

We, we changed all their multiplexing setting.

We changed what, uh, we changed how.

, the schedules were, I mean, we changed everything and we were doing this

to, to do massive efficiency change.

They were running like, it was like 18 tape drives simultaneously,

and I told 'em that that was their pro, they were gonna buy, uh, two

Oh, this is a sh when you're, you were, you're talking about the

shoe shining problem and they were trying to tape goes too fast for what they need.

Yeah.

And so they were gonna buy two new tape drives and that was gonna fix the problem.

And I'm like, no, it's not.

It's actually gonna make it worse.

And I explained to 'em and I'm like, listen, give me a few weeks and I'll

redesign your backups to submit.

It'll cost you less than what those tape drives were gonna cost.

Cuz they were expensive.

These were the storage tech tape drives, like the 90, what?

I can't remember the tape name.

Tape names.

But anyway, um, so we did this and.

, um, and things went really well, but so I knew about the archive bit.

This is all about the archive bit.

I knew about the archive bit and I knew that if I had, I basically

left their parallel backup system.

I left their parallel policy.

Up and

I left their existing policies running in parallel.

Well, I was gonna do the regular backups, uh, during the day,

um, and something like that.

It's been a while.

And, but I knew that I would screw them up if I had the archive bid on both systems.

So I used, this was net backup.

I used a feature of net backup that said, okay, don't use the archive bit on.

On the sink and what I didn't notice or what happened over time, over about

two weeks is the arc, the incremental backups got bigger and bigger and

bigger and bigger, and they got huge

on your new

and they on the new system and they got so

big that they weren't finishing.

By the way, when I got there, they were backing up 28 hours a day.

So my, my opinion was no matter what I was doing, it still wasn't gonna be that bad.

, but backup, but the backups weren't finishing and I didn't understand, right.

So, so we shut off the old backups.

We're like, okay, we we're, we're beyond the point of no return.

We shut off the old backups, backups still weren't finishing.

And uh, you know, I was working through diff, you know,

multiple levels of support.

And I actually at, at one point ended up with my buddy Rob,

Hmm.

I'm like, dude, I don't understand.

And we, you know, we figured out that when that.

Doesn't use the archive bit.

What it does is a, again, this is, this is a hundred years ago.

I don't know what it does today.

If it doesn't use the archive bit, it would, it would traverse the directory

tree and then it would, as soon as it encountered a directory that whose

modification time had changed, it would back up everything in that directory tree,

Okay.

Including data that does, yeah,

including data that hadn't changed, right?

It, it was really inefficient.

And so it tried to add that level of efficiency by working up the,

you know, so long story short, the back, the incremental backups were,

were getting close to full backups.

And so once we figured that out, we're like, okay, okay,

okay, okay, we figured it out.

We're gonna tell it to use the archive bit now.

And, um, and we turned on the, we turned on the archive bit.

and it was the same the next day.

Right?

Okay,

Now I look, now I look, and meanwhile this, this

manager, he is looking at me, you know, looking at me over the, you know, he's

beginning to think I'm a complete moron.

And, um, he, um, I said, look, I, you know, we figured it out.

This is what, you know, he's like, but it, but it didn't fix the problem.

And that's when I was really talking to Rob.

I'm like, I don't understand.

And again, I was on the edge here.

I would, no one had done what I was doing, you know, and so, so what we

figured out was when you tell net backup not to use the archive bit,

it doesn't use the archive bit.

Now, what that means is that when it backs up a file, it doesn't

clear the archive bit, which was the whole point of me using the thing.

And so that meant that

The

we,

when you turn it on

to run the, we had to run the one backup.

that would use the archive bit to clear everything.

But because we had run one in a long time, it was also gonna be really long.

And so then it's like, okay, tomorrow night is gonna be all better.

And that's when the boss said, okay, smarty pants, no one's going anywhere.

He was, he was really upset at the level of instability.

And I can understand him being.

But he was really upset at the level of instability in his mind that

I had introduced into his system.

Nevermind that the new backup system was backing up 50% more

data than the old system because I changed the, the inclusion factor.

And they had, they had, they had left out a lot of really important data.

So it went from 20 terabytes to 30 terabytes and.

Nevermind the fact that we were now creating two copies, whereas before

he wasn't even getting one copy done.

So I was feeling pretty good.

But all he sees is his

Stability.

Yep.

Yeah.

And so he's like, well, nobody's going home until, and literally

he just, he stayed with us.

I mean, to his credit,

Okay.

stayed with us and like he ordered pizza and stuff, uh,

but he literally wouldn't let us leave.

I couldn't even say like, well, I kicked off the back house.

We're gonna go get dinner.

I ain't nobody going anywhere.

He's like bathroom.

You already had your bathroom break.

but we're gonna sit here and Yeah.

And so we were there until pretty late.

Um, I don't really remember how late, but I remember like, it was like a whole

second day, you know, like it was, it was a long, long, long, long night.

And that my friends, . That's what happens when you don't understand backup levels.

Anyway, All right.

Well I think this is enough torturing people with backup level information.

Prasanna Malaiyandi:

I, I a hundred percent

talk about, go ahead.

Yeah.

Next week we're gonna talk about metrics.

Ooh,

yeah.

I think it's a little more exciting.

yep.

It's backups.

Like how exciting could it be

Come on, Curtis.

What are you talking?

Telling our audience, our listeners,

Woo.

I dunno what to tell you.

You know what, maybe you're excited as you're as excited

about backups as I am, maybe, to which I say, welcome to the party.

All right, well, uh, thanks for listening folks, and remember to