Backup TCO: The Costs Nobody Talks About

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we're talking about something that hits every backup

admin right in the budget backup, TCO, the total cost of ownership

for your backup infrastructure.

When most people think about backup costs, they tend to think about, at

least I think, the software license.

And the hardware they have to buy and probably cloud storage

that they have to pay for.

Those things are expensive, and I don't want to minimize that.

We're talking about six and even seven figures sometimes.

But here's the thing that most people don't wanna say.

Those are actually the small part of the cost.

The big cost is us.

The people we're talking about in this episode, what drives those costs up?

What you can do to bring them down and why soft costs are probably the most

expensive line item that you're not.

Tracking.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Backup, and I've been passionate about backup and recovery for over 30 years.

Ever since.

I had to tell my boss that we had no backups of the

production database we just lost.

I don't want that to happen to you, and that's why I do this.

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the backup wrap up.

I'm your host, w Curtis Preston, AKA, Mr. Backup.

I'd have with me a guy that knows about the topic that we're talking about today

from a spreadsheet Prasanna Malaiyandi.

How's it going?

I am doing okay, but I don't know if it's fair to say just spreadsheets

because I have talked to customers.

never actually fired this in anger, if

Yeah, you've never, you've never been the guy on the keyboard in the customer.

Yeah, yeah.

which tends to be most of my life actually.

yeah.

That, that is true.

This pretty much sums you up, right?

Like, you know, you advise me on all these different things that

you've never actually done on.

It's, it's, it's somehow you're like secret power.

Um, so much so I'm, I'm gonna, I'm gonna bring up a memory for you, my friend.

Oh, no.

TCO.

Uh,

the, the three words that brings shivers to my,

So he, you got put in charge of the TCO analysis at a previous employer, and it,

you, you loved it so much, didn't you?

it nearly broke me

in fairness.

A lot of it was,

it's hard to provide apples to apples comparisons different vendors

using different terminology and different techno technologies.

Even things like what's your DUP ratio?

How different companies that is very

Yeah.

Yes, very much.

It's very different, right?

Um, because yeah, it, I, I remember that being a constant source of

frustration to me as just trying to evaluate technologies because people

would just say, um, you know, um.

300 x.

Yeah.

Three, 300 x, which you can get 300 X if you do fulls every day.

Right.

And I remember one previous company, you know, I'll let them go nameless,

but they're like, well, every backup that we do is like a full.

It, it's not a full, it's an incremental forever, but every backup

that we do behaves like a full in that you can just do one restore.

Yep.

and so therefore they would say like 400 x, right?

Um, yeah, which is just nonsense.

But the, but the cost of managing a backup system.

Comes from a number of different places.

So let's talk about, 'cause that's what we're talking about today is,

is backup management costs, and let's talk about the small ones first.

And when, and when I say this, I, I think the average user is gonna be

like, how is that the small part?

Right?

So, the three small parts are the software, the hardware,

and some cloud infrastructure,

Yeah, but that's what the biggest part of when I go for budget and, uh, go to get

money to procure a system, like I spend all my time fighting for the budget just

Yeah.

three.

It, it's the thing we, we do, we do end up talking and focusing a lot on these, uh,

things and I don't wanna minimize them.

Backup is really expensive, right?

Um, you, you of all people would know because you did the spreadsheet.

Should we pull up, should we pull up the spreadsheet Prasanna?

Do you want to go look at the TCO analysis?

no, no, no.

Thank you.

So, um, the, so it is very expensive and even when you look

at like our former employer where you, you're paying as you go.

Even there where you're essentially paying by the month.

You don't pay by the month.

You, you typically buy a year or two years in adv advance or credits for a

year or two years in advance, and then you, you debit against those credits.

Right?

Which, which is which, and I liked that model.

But my point is, even in that situation where you're paying only for what you

use, it's still a very big purchase.

Right.

Uh, these were, these were.

Six and sometimes seven figure deals, right?

Um, so it, I I don't wanna minimize the cost of the software, the hardware,

uh, and the cloud infrastructure, especially that third part.

Why, why is it the cloud infrastructure, especially if you're rolling your own?

Right.

Uh, and you're, you're gonna put a, let's say you like the idea that we've talked

about in other episodes, like in the past episode about having an immutable

copy and say, you know what, we're gonna put a copy of our stuff up in, uh, you

know, uh, object lock type cloud storage.

Why is that part so expensive?

Because you pay for that per month for however long your retention

Yeah, forever, right?

Um, yeah, you do pay for it for a really long time and depending on how

well you, you need to manage your, so let's talk about that for a minute.

Because we are gonna focus a lot on the, the final thing.

But I do, I don't want to minimize this.

I think a lot of people start copying some backups to S3.

Right.

And, and the competitors.

But so much, so many times it's S3, right?

Um.

They do that because the idea of having an object lock backup is good

having an, it's both object locked and, or it can be object lock.

I hope it's object lock.

Please have it be object lock.

And then, uh, and it's offsite, right?

So they, so they get the 3, 2, 1, they get the 3, 2, 1, 1 0.

You get all of that stuff, right?

Which, which we talked about in, in a previous, previous episode.

Um, the.

But it's very easy to just keep copying stuff and not deal with,

um, the, the retention part.

And if you don't do that, you'll suddenly get one of those bills that

you're, that you're not gonna like.

and this is exactly what I was talking about on the previous episode where it's

like, Hey, immutability sounds great.

I. But a cost associated with it.

And I think when you also start to add in incremental forevers, you can never really

clean up those old copies the same way you would have if it was a self-contained

tape, which you could then recycle.

Right, right.

Oh, for the day when we could just recycle the tape.

Um,

with the

uh,

you have so, and forever incrementals, you have so many dependencies.

yeah.

don't know if your object lock allows you to actually eventually delete that

data or if it'll just keep growing and growing and growing, which Amazon loves

of course, because you're spending

Yeah.

with them.

Yeah.

And which means that you need to take that into account in your design.

Right.

Um, let me, um.

Let me, let me bring a, let me bring a story back.

I would you like to hear the story, Prasanna of the whole reason I I did.

I am what I am.

Um, and that is the story, by the way, I referenced this story

in every episode because in the.

Opening episode in my, you know, uh, little story of why I do this.

Because we had no production, no backups of the production

database that we just lost.

Um, the name of the server was Paris.

It was an Oracle database.

This is 1993.

And, uh, so many of our listeners weren't even alive.

Um.

And in 1993, I, we, it was tape based and it was an Oracle database, and

we lost the server and I went, I did what I was trained to do, which is go

look at the tapes and, and I, I, I, I looked at the tapes and I was told,

look for errors, and I was looking for the errors and I saw errors.

And so, okay, well, no problem.

I'll just go there The day before.

Nope, those that has errors, uh, go to the day before, oh nope, that has

errors and so on, and so on and so on.

And I finally found a tape, um, that was called, uh, I dunno,

like let's say Friday, whatever, whatever the tape was labeled, right?

And the, the backup was made six weeks and a day ago.

Huh?

Right?

Oh, no.

That's not the worst part.

Okay.

Ask me what our retention period was.

That's why when you said six day, six weeks and a day, I was like, oh, no.

So Curtis, what was your retention?

It was six.

Oh.

So we had no backups of Paris whatsoever.

And um, I remember, um, I remember that a guy by the name of Joe Fitzpatrick,

he pulled out some UNIX magic.

He was doing stuff with like the magic numbers.

I dunno if you've ever had to like do FSCK with, with magic numbers.

And he was doing that and he managed to somehow like resurrect this disc.

Um, and we only lost, it was something like three or four days

worth of data in the purchasing database for a $35 billion company.

Um, and so.

Retention matters.

Um, it, it would be nice, you know what you're talking about it

that really needs to be taken into account in your backup design.

This forever incremental.

If you're going to be doing a deduplication based, forever incremental

thing, you have to have the ability to, um, to take the, the blocks out in the

middle, the objects out in the middle.

The problem is that's not really compatible with object lock.

Yep.

So what, what a lot of people do is they sort of translate those into images.

Um, they put those images in the object lock system, and, and this is

sort of the last ditch copy, right?

They create like.

Um, a synthetic full, if you will.

They created a synthetic full and then they lock that.

Um, but um, I will say that deduplication, uh, you know, object level deduplication

is kind of incompatible with.

Immutability.

Yeah.

Um, but, and if you don't do that, right, if you do, if you do the, if

you do the first part of what we said, and then the second part of what we

said, your storage costs could be astronomical and growing forever and,

and, and never going away, right.

So that, yeah, that's the cloud, that's the cost of the cloud storage.

But I think that the, the part that everyone forgets is the part.

That we're about to talk about, and that is you, you do pay for the hardware,

you pay for the software, and you will continually pay for the cloud storage

piece if you're using the cloud.

Or perhaps what you're paying for is, um, you know, an Iron Mountain, you know, a a

a shelf or two at, or, or, or a thousand,

else.

or yeah, or whatever, whatever company that you're using for that.

But the thing that you're going to do forever, regardless of any of this, is

you're gonna have, at least for now,

Yeah, I was gonna say for now,

for now, you're going to have a human being or several in charge of

looking out after this backup system.

yep.

This may be in, um.

Many companies, it will be a collateral duty.

Um, um, in some companies it will be a full-time job.

And the company where that I was at, which I, you know, I, I think about this

all the time, like it's amazing that we had this amount of infrastructure, labor,

infrastructure for what, now I look back on, it was 300 gigabytes of data, right?

Um, we had, we had a staff of like.

I dunno, like nine people that were working, um, basically it was 24 by seven

Yeah,

and uh, they worked, the way they worked was, uh, they did three

13 and a half hour shifts a week.

And, um, so that, you know, that was their, that was their 40 hours and, um,

the, um, and then they would overlap.

So each.

Every day, you'd have an hour and a half overlap between the

ongoing and off ongoing shifts.

Um, but you know how much labor you're spending is, is going to be very much

the, you know, it's gonna be dependent on the size of your infrastructure, right?

The size of the, the data.

And then also it's going to be based on the degree to which

you take our previous episode.

Seriously, what were we talking about there?

What were we talking about?

Um, the,

Oh,

we just recorded a half hour ago Prasanna.

It was

backup infrastructure and ransomware.

Yeah.

Yeah.

So if you take, if you take that.

Episodes seriously about the fact that 96% of ransomware attacks right now are

targeting the backup infrastructure.

Uh, that was from one study that we, that we used in the last uh, episode.

Uh, if you do that, then you're gonna spend more time, you're

going to spend more time monitoring the backup infrastructure.

You're gonna spend more time configuring the backup infrastructure with a, with a

proper, uh, leAST privilege type setup.

What is that?

Basically what it needs access to is all it has access to.

Right.

that everyone has access to entire kingdom

Yeah.

Back in the day, everybody had to do with backups had root, right?

They had root on the password.

even the hospital in Germany.

Yeah.

There was, so they gave, yeah, that was a good one where they gave everybody,

no, it wasn't Germany, it was Spain.

Um, but uh, it there, yeah, there was a hospital that.

Um, where they gave everybody doctor level privileges instead of, because that

was the easiest way to do that, right?

That's the opposite of least privilege, right?

Give everybody the least amount of privilege that they

need to do their job, right?

Um, so if you're, if you're doing all of that, that takes more

time and it takes more effort to properly configure and manage.

but that's like the configuration aspect.

But there's also sort of the day-to-day operations pieces that also depends

on the size of the environment and the software vendor you decide to use.

Because some vendors are optimizing for sort of user experience and

making things a lot easier to manage at scale versus others where it

feels like the 1993 interface still exists, probably what you're used to.

Yeah.

Yeah.

Um, the, yeah, I can certainly think of, I can think of backup software

where it seems like the, it, it literally is the same interface

that I was using 30 years ago.

Right.

Yeah.

So here's a question.

What can, let's say I'm, I'm a backup admin.

What can I do to lower costs?

Yeah.

So to lower your costs, you as a single person, big environment,

you can't handle everything.

You have a thousand other things to worry about.

You need to automate.

Isn't automation kind of like the key word these days?

Yeah.

automate, automate ai, do everything for you.

yeah, yeah.

And, and, and the, the more I, I think this is true in, in cyber infrastructure

is true in backup infrastructure.

The more that you can take the human out of the equation, um,

the, the better off things will be.

Let me give you an example of.

The opposite of this, right?

The opposite of automation.

And that is, uh, time machine, right?

I have a Mac, they have time machine.

And, and while I like Time machine, I don't use Time Machine.

The only time I use Time Machine is like going from one laptop

to another laptop, right?

Um, and one of the, one of the issues that I have with Time Machine is that if

you're using Time Machine on a regular basis, if you're doing it correctly, you

are, um, hooking up and then unhooking.

The backup every time you go to do it.

That is the opposite of automation, right?

Because you're, you want to disconnect it from the thing that you're protecting

while you're protecting it, right?

Um, and then, but then you have to hook it back up to it, uh, to get a backup.

But then you have to unhook it so literally every day, which

means, and no one does that.

The people that do use time machine, uh, I remember I had a, a Mac

mini, actually, I still have it.

It's, I think it's sitting over on a shelf somewhere, but.

The Mack Mini comes with two internal hard drives, and I backed up the one

hard drive to the other hard drive you, you know, with time machine.

But that's not, that's not good design,

Yeah.

Um, and, and so we need to do that as much as possible.

And I will say that I think that.

Backups today are significantly more automated than they were when I was the

guy, you know, swapping tapes, right?

Because when I, when I think about, when I think about back then, the first thing

we had to do, we didn't even have robots.

Yeah.

First thing we had to do was go around all the servers and inject all the

tape drives from last night and then create this, this box of tapes, right?

That that's all gone.

You know, for the most part, even people that are using tapes,

and there's still quite a bit.

There's still a number of people that are still using tape, but if they are,

they definitely are using automation.

They're using lar large tape libraries, or even small tape libraries so that you know

that that's not part of the equation, but.

I think today's backup infrastructure is a lot more automated than

it was even 10 years ago.

And that is due to the advent of, uh, ddu to the ad, to the increased

use of disc and backup and the, the increased use of cloud and backup,

Yep.

Because we can automate all of that.

Uh, when I go back to, you know, the.

The, the, um, the object storage part of the discussion we previously

had, if you do it right, object storage is self-managing, right?

It's not like, um, a a volume.

It's not like you go and buy a data domain box and then you fill up

the data domain box and you have to buy another data domain box.

It's not like that at all.

It's, it's object oriented and which means that the, the capacity is just.

Hundred percent managed for you.

Um, so that is the idea of, you know, using, using technology that

reduces the, the human component.

Yeah, which is why actually Rubrik, Cohesity data domain all support

a cloud tiering model as well,

Yeah.

they realize that's what customers wanted.

The fact that they don't have to manage storage capacity.

'cause my gosh, is storage capacity management.

Painful because you have to understand when you're gonna run out.

You have to go through your budgeting cycles, procurement, uh, racking

and stacking, configuring, hoping you don't run outta space again

before the next budget cycle.

When you add D dupe into that picture, trying to pic, trying to calculate

capacity for D dupe, uh, is, is, you know, it's just not a thing, right?

It's just, it's just literally not possible.

You, you can, you can, you can take a, you know, a ballpark number, you could do a

swag, a scientific, wild-ass guess a swag.

Um, but you're not, you're not gonna get it right.

You know?

And so what you typically end up doing is you tend up.

You end up over purchasing, you end up over provisioning.

Um, that is another thing.

So if we talk about one of the things that we can do to lower our

infrastructure, perhaps it's time to look at your backup software, right?

Look at what that system does to lower your overall cost.

Do you remember when, um, you know, when we used to work at,

uh, that other company over there?

Um.

A lot of times when you come in there as a salesperson, the first thing the customer

will do when you're doing a TCO thing is, I don't wanna hear about soft costs.

Yep.

Right?

But the point of this conversation is that soft costs are the

biggest part of the costs,

Yep.

right?

So maybe, I guess what we're saying is don't, don't do that.

Right.

Yeah,

Don't, don't, don't minimize.

because thinking that, oh, it's already a, it's already a sunk cost, right?

Right.

the person's there, but it's like you've hired a person.

They could be doing other things to add value to the organization

rather than managing on a daily basis your backup infrastructure.

They could be learning ai.

I was thinking more about helping plan for these future projects,

protecting against ransomware, other things like that, you know?

Yeah.

Um.

And so, so that's, I'm I, I guess the overall concept here is do what you

can to minimize those upfront costs.

The biggest thing, uh, two big things from the ongoing cost is the cost

of the cloud storage component.

That's gonna bite you in the butt if, if you don't pay attention to

it in the design phase upfront.

And then the third is, do not dismiss the cost of labor, right?

Here's a study from Unitrends that said that more than half of the

environments that they studied were, uh, they were spending more than 10

hours a week, uh, managing backups,

Mm-hmm.

Uh, obviously this is, you know, there's a bell curve here, right?

Uh, there's some people that are spending way more than that.

There's people that have, you know, an entire dedicated team to backups.

But even if you are, even if it's a collateral duty.

You want to see what you could do to minimize the, you know, the,

um, the effect of that, right?

Do you think so I know we talked previously about like budgeting and

how to go and procure additional funds for these systems As you're

evaluating that first category of software plus hardware infrastructure.

Yeah.

Do you think people should also take into consideration these

soft costs that evaluation?

Because if you are spending, 10 hours a week with vendor A and it drops

to five hours a week with vendor B,

Right.

B is slightly more expensive, right?

Right.

Yeah, I, I think that's something you should take into account.

I think the difficulty, and the reason why most people say I don't

wanna hear about soft costs is that it's difficult to know upfront.

Hmm.

difficult.

Every, every backup software says that they're easier to use than the other guy.

Right.

Um, I will say I do, and, and again, this will come across as

being serving my former employer.

I am a fan of SaaS based backup, right?

Because you don't have to worry about the hardware infrastructure, the

software infrastructure, all of the upgrading of all the software, and

really importantly, you don't have to worry about the cybersecurity

of the backend infrastructure.

Yeah.

And I think that that is really a big deal.

If what you want to do is lower your costs of day-to-day backup

infrastructure, uh, you know, of running the backup infrastructure,

the way to do that is to have someone else run the backup infrastructure.

Now you're going to pay for that, right?

But I think this is one of those things of like.

Perhaps you could save money in the long run and also

increase your security posture,

Yeah,

By having, uh, somebody do this that does it all the time.

Uh, I think that that's possible.

You could both reduce your costs, even by a little, but

increase your security posture.

but maybe the backup wants to learn these new skills.

Curtis, why are you taking it away from 'em?

Uh, go learn on your own time, buddy.

No, I, I think cyber, you know, one of the things that came up a lot in the book

was that there are certain elements of cyber, uh, security that really should

not be attempted by an amateur, right?

I'm a big fan of cybersecurity, um, you know, service providers and,

uh, of which Mike is one, right?

Mike with Black Swan Cybersecurity, uh, you know, he's one of

those I'm a big fan of.

When you have an area backup and cyber, very similar in this regard.

Backup is, there's not a lot of experts in it because no one

wants to be an expert in it.

Right.

In cybersecurity, there's not a lot of experts in it, because to be an

expert you have to like go to battle.

You have to have been day to day battling the, you know, the bad guys.

So it's really difficult to become a true expert in cyber.

Unless you're doing it, you know, and you're, you're actually actively

battling bad guys every day or on a, on the very regular basis.

And so this is one of those where, and and this is outside of the

realm of this topic, but um.

Perhaps a way to lower costs is to outsource that part of the infrastructure

because your person isn't gonna be as good as that person, and perhaps

they'll, they're actually gonna spend a significant more amount of time,

uh, doing, uh, not as good a job.

and maybe money as they're learning and making mistakes or

not choosing the best solutions

Yeah.

your problems.

Yeah.

And perhaps that person would be much better served doing something else.

Right.

Um, and, and bringing more value to the team, um, you know, in, in

an area where they can excel, so.

All right.

Well thanks for chatting Prasanna.

I'm really upset that you brought up TCO.

Thank you, Curtis.

Thank you

PT, PTSD.

You are welcome, sir. And, uh, hopefully you guys, uh, hopefully the

listeners got a, uh, some schaudenfreude some en enjoy enjoyment at the,

at the misfortunes of Prasanna.

Uh, that is a wrap.