Recovery Time Objective vs Reality: Closing the Gap

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we're tackling one of the biggest lies in it,

your recovery time objective.

I don't care what your RTO documentation says or what you

believe you've promised your bosses.

If you haven't tested it, you can't meet it.

Period persona and I break down why most organizations are living in fantasy

land when it comes to recovery time, objective, and more importantly, what

you can actually do to address that gap.

If you've ever felt that pit in your stomach when someone

asks you about recovery times.

This is your episode.

Let's get real about RTO.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Backup, and I've been passionate about backup and recovery ever since.

I had to tell my boss that there were no backups of that production

database that we had just lost.

I don't want that to happen to you, and that's why I do this.

On this podcast, we turn unappreciated backup admins into cyber recovery heroes.

This is the backup wrap up.

hi, and welcome to the backup wrap up.

I'm your host, w Curtis Preston, AKA, Mr. Backup, and I have with

me the rarest of All Beasts lately.

Anyway, Prasanna Malaiyandi how's it going?

Prasanna, I.

I am good, Curtis.

I know it's been a

It is been, it's been a minute.

we've

why, that's why the listeners have been listening to like repeats.

Uh, yeah.

'cause you of course you're gonna blame it on me with I know, I know.

I was working the election, I was working the, uh, I, for those who don't

know, I'm gonna, I'm a poll worker, you know, and I'm not doing other things.

Site

Yeah.

I, I am a site manager of the Yeah, the Bonsall Vote Center.

In San Diego.

And so we did have our special election and so I worked for 11 days, not

including the setup and tear down day.

So I've been a little busy.

Okay.

What, how many voters?

Yeah, you've been a little busy and how many voters Yeah.

Did you have

Uh, we had like, like 75 over the first 10 days.

And then on the last day we had about 400.

Um.

Oh

Which was, which is, which is a lot.

Um, and I, you know, I love, I love, I love democracy.

I love people.

I want everybody to, uh, to, to, to, to vote.

Um, you know, if you don't vote, you don't get to bitch.

That's my,

But please don't wait

but yeah, for the love of God, look into, look into your, your sight

most, or your state most likely has.

Early voting, look into early voting and early vote or vote by mail.

Right?

Um, those 400 people could have come any time in the previous

10 days, and we would've, they could have voted just the same.

Um, yeah.

Anyway, so please vote.

Um,

Well, welcome back.

Yeah.

So, um.

I wanted to, we're, we're gonna kind of, you know, we're kind of

redoing things after a, you know, a couple different phases here.

And, uh, we're gonna just try to do some, some hot topics

that, um, I think are important.

And one of them that we're gonna talk about this week is RTO.

And specifically which recovery time objective.

Of course, we're, we're gonna what?

What?

Yeah.

Return to office.

So we're gonna talk about return to office and then, um, and you know, what it is

and why it is fantasy for most people.

And then what, what they could do, um, you know, to, to address that.

So first off, you want to define recovery time objective.

Yeah, it's basically your objective, right?

Your goal for how long it should take you to recover from some disaster, right?

And get back to a good known spot.

This is including things like recovering your data, reconfiguring

your network, right, and different.

disasters might have different recovery time objectives, so it's also important

to remember, like recovering a file may be a lot less in terms of RTO

say, recovering an entire data center.

If it, uh, something

Yeah, it's interesting you brought up that that's actually a hotly debated topic as

to whether or not RTO should ever change.

I agree with you that, um, the RTO is situational, situationally dependent,

um, and that, you know, if, if you've been attacked by ransomware.

For example, there's no way you're gonna meet sort of what

I would call a normal RTO.

Uh, and the same, you know, and the same with like, if it's a complete

disaster that wipes out your entire data center and you have to physically

build a building into which to put your servers or something like that,

that RTO should be, um, larger than, you know, we lost a single server.

Or like, you know, you said a lost a single file.

Or if you go tell like your application admin who needs to recover data, oh, by

the way, it's gonna take one week or two weeks to recover your data because that's

the RTO you set for like site disasters.

They're also probably gonna be unhappy,

Yeah, absolutely.

wait, wait, wait.

That makes

So that's another really important thing that you brought up right there,

which is the, and, and this is a really important concept that goes

through almost everything that we teach, and that is that you, meaning

the backup admin, the sysadmin in charge of backups, whoever you happen

to be, you do not determine the RTO.

Right.

The business unit determines the RTO or whatever, whatever term is appropriate at

your governmental entity or NGO, right?

Um, that is the, the entity that determines, uh, the the

recovery time objective because it's based on finances, right?

It's based on, uh, you know, like if it's a business, it's based on

how much money are we going to lose.

While we are down, right?

Um, if it's a governmental organization, it's based, it,

it's very different, right?

It, it, it's more along the lines of how much damage to our organization

like reputationally will happen based on how long we're down.

And also how much more difficult will it be to redo the things that

we, you know, to, to do the things we had to do while we were down.

Uh, you might have to switch to, you know, to paper in the meantime.

And, uh, so you, but, but the point is, all of these calculations are

things that the, the business or management should be doing, not

those, uh, in charge of backups.

Uh, what role do you think the, the, the backup people play in determining,

uh, recovery time objective?

I think it is basically to figure out, okay, based on what the business has asked

for, say they come back and say, okay, my Reto recovery time objective is one

day.

Based on that, here are some options that we can do technology wise,

and I think their goal is to come back and say, okay, here's how much

it will cost you if you want to support that recovery time objective

Yeah.

And, and you know, in the very beginning this is gonna be ballpark numbers, right?

Um, well the first thing I would say is you come back and you go, okay,

you've asked for A, we do B, right?

We do a times four.

Um, right.

So, well, let let me ask you this.

Why do you think, uh, I have, I have my opinions, uh, I'm

curious, why do you think.

Most organizations, if they have an RTO or even if it's poorly documented,

et cetera, et cetera, et cetera.

If we haven't agreed upon RTO, why are most organizations

completely unable to meet that RTO?

Well, the biggest thing is they probably haven't tested.

To understand like is it actually like, and that's why I said like when you

asked the definition of RTO, right?

It's your desire, it's your objective.

It doesn't mean what you will

actually hit because there are so many other things involved.

Like we talked about.

Maybe part of your RTO is just bringing back the data or the

application, but then what about.

Like making sure I'm able to procure the servers to recover or get those

up and running.

Uh,

maybe I,

need to bring up active directory or Intra or whatever it's called

now, whatever Microsoft calls

it's, it was rebranded while we were on this recording.

Yeah.

right?

But all of these other things, which maybe you don't necessarily have control

of, and maybe you're only thinking as a backup admin of, Hey, I need to recover

the application or just the data, restore

Well, and also in addition, and again, let, let's make sure that

we, we, we say that the recovery time objective has been met or not

met when you, when the application.

Is fully up and running and available for use by the user, right?

It's not, oh, well I did my restore.

We got a four hour R-T-O-I-I did my restore and it only took four hours.

No, the question is, is the application back up and running?

And that includes it, like I said, any hardware, uh, procurement, which, which

hopefully you're doing in the cloud.

But any hardware procurement, any.

Stuff you gotta do.

And if we're talking about ransomware, all of the, the, the stuff you gotta

do to make sure that the, the server is ready to, uh, be restored and the

restore, depending on what type of thing you're recovering from, the

actual restore may be the smallest part of the, uh, recovery time.

Actual is the term that we use.

Based on your sort of consulting,

Yeah.

Mm-hmm.

this right, what would you estimate is that ratio between time to

actually restore the data or the application versus the end-to-end?

RTO

and I?

I'm

Yeah.

If, if we're not talking about ransomware, it's like 80 20.

Right.

Uh, meaning 80% of the time spent doing the ba the Restore, the other 20%

in a modern day scenario where we're probably, uh, gonna do this in the

cloud so that we can, you know, snap our fingers and we have the hardware that

we need, uh, we're doing the Restore.

It's well, you know, well tested, uh, although often it is not right.

And then there's some amount of time to do some initial functionality

testing to make sure that all the dependencies have been met.

And then, um, you know, and then we're, we're ready to roll.

Right?

So I'd say it's like 80 20 in a, in a ransomware scenario,

it's like, you know, 10 90,

Yeah,

right?

yeah,

gonna spend most of your time making sure that you're recovering

to a, uh, pristine environment.

yeah.

And I think that's one thing you just touched upon in your previous statement,

which was like, sometimes things change.

And we are talking and sort of like understanding why do most people's

RTOs not meet what is expected?

Do you wanna touch on some

Well,

I know

yeah, so I'm gonna say the number one reason is that they simply don't have a

backup or disaster recovery system that is capable of meeting that RTO just period.

Hmm.

They didn't do, uh, and, and this is.

This is quite possibly they, I, I remember when, when I, you know,

back, go back 30 years, right?

That, that we knew we abs everyone knew that our backup system wasn't

anywhere near capable of meeting the RTOs that we had discussed.

Even though we didn't use that term back then, I, I'm sure the term was

available, but I didn't use it and the.

We just knew that it, it just wasn't, wasn't possible.

Right.

Um, I mean, in some cases it was laughably impossible, right?

Um, and that we had servers that it took us a week.

It took, took us a week to get a full backup.

Okay.

Like, how are we gonna meet a four hour R-P-R-T-O if it takes

a week to do a full backup?

Right?

Uh, and by the way, our next episode we're gonna talk about

RPO recovery point objective.

So it's a, it's very much a sister episode to this, but that, I'd say

that's the number one reason is that people's backup systems, and, and again,

I used the term backup very broadly.

Anything that.

That brings the server back to the way it looked, you know, before the

disaster is a backup system to me.

There.

Well, you just use 'em for different purposes, disaster recovery, et cetera.

Um, but that's the number one reason.

The other reason that's very closely related to that is

that they have no idea, right?

They, they haven't tested right.

They, they've got a system, they've got a clue.

Right.

And they're like, oh, well it takes us, you know, three hours

to, or four hours to back up.

Therefore, we should be able to do a four hour restore.

There's a lot of ifs in that, right?

Yeah.

the, the other thing is a, as you know, restores often a

lot slower than backup, right?

For a number of reasons that, you know, are all over the

place that, that they, they

the

ahead.

is like incremental

Yeah.

Yeah.

Forever incremental.

Right.

That I would probably say is the biggest

Yeah.

That you're, that you're piecing together a restore from many, many, many stuff.

Uh, I, I think if you're, if you, if you have a proper design that alone

shouldn't, um, you know, impact you.

If you are doing a, a, you know, sort of the old school full restore, followed

by each incremental restore, and that means you're actually restoring some

files multiple times, then Absolutely.

Right.

If you're doing, if you, if you have a, if you have a. A system that is

properly de uh, developed, right?

That fixes that issue where if we know a file has changed, then we're not

gonna restore that file multiple times.

We're just gonna restore the latest version of the file.

If you have that, that's not really the problem.

But you do have the issue of ddu, right?

You have the, the DDU tax that quite often really rears its ugly

head when we go to do a restore.

Now why would that be?

Why would that be the case Prasanna?

Because when you're dup Deduplicating data, throwing away a whole

Mm-hmm.

But the problem is when you need to read it, you're basically doing random reads

across the entire system in order to be able to recreate that single file.

Because you might have old blocks from one part of the disc and a

different part of the file from a different part of the disc.

And so you end up with all these random reads, which as we know, our disc

drives are not very good at doing random

Yeah, it, it's the, it's the ultimate fragmented file system, right?

Uh, you are just, you're absolutely guaranteeing that everything

you need is everywhere, right?

Um, and, um, the, that, that is absolutely one of the cases.

And, and if we're coming from tape.

Right.

Uh, which is probably less likely for most people.

But if we're coming from tape, then we really do start talking about that,

the, the, the forever incremental stuff and, uh, you know, because you're having

to load all these tapes, uh, but also there's a network can get in the way.

There's also, depending on what.

Raid, uh, we're using, right?

If we're using RAID and, and we're using raid, right?

Everybody's using raid.

Yeah,

depending on whether or not you opted for, does anybody

wrap up for RAID 10 these days?

I don't know.

I

I don't think so.

I think everybody does raid six, right?

Or, or, or something.

So raid, dual parody or whatever, and that has a right penalty, right?

So for a number of reasons, restores are often slower than

backup and you will never know.

Until you do what Prasanna.

You

Exactly.

And again, go ahead.

Well, I'm gonna bring, I'm gonna bring out a story.

Sorry.

Uh, going,

okay.

going back to going back to my first, you know, the first time that things were

really, really bad was that time when we had a new backup system and we had

used a compression feature on, on the way in, and it was software compression.

And long story short, when we went to, uh, there, there was a. There

was the, um, we went to do the first major restore after we needed it.

Right?

We, we didn't test restores, we only tested backups.

And uh, when we went to do it, uh, the, it was a DD, s and it was like,

blink, blink, long pause, right?

Blink.

Blink.

And once we called into support and they were like, yeah, it's working as design.

And basically we had not.

Tested this at all.

And not only was it slow, it wouldn't work.

It, it was just literally without going, without taking too much time,

it just literally wouldn't work.

Right.

And, uh, unless we like tripled the size of ram or something.

Right.

And, um, so yeah, you, you just do not know how your system is going to

perform until you go to do a restore.

Yeah.

And.

One thing similar to that story is you should also do a realistic restore test.

Don't just be like, oh, I'm gonna just restore a file, or

I'm just gonna restore a vm.

I'm good

Yeah.

Right?

Because that may not be a realistic scenario for when you have to

recover a full application suite or your entire environment.

So make sure you're doing the right little type of

Yeah, absolutely.

It should,

any

it should be whatever the, whatever the thing is that we're

setting the RTO for, right?

Uh, you don't have restore the entire environment, but you need to do

representative restore tests, right?

Um, entire servers, entire environments, entire recovery groups.

What's a recovery group Prasanna?

It is a group of things you need to restore in order for

your application to come back.

So it might be your database server plus your storage, plus your active

directory or uh, system, right?

Plus whatever else is needed in order to get that production application back

exactly.

And so, and, and so that becomes important too.

I was actually gonna comment on that, because there's an order

of operations you have to do, and so you have to account for that.

When you calculate your RTO, it's not like, oh, I can just restore my, uh,

database, have it up and running before I have active directory up and running.

It's not

Right.

And, and by the way, that one, one of the things that prompted this episode,

uh, Kaseya did a 2025 state of the backup industry, uh, and they said that more

than 60% of respondents believed that they could recover under, in, under a day.

However, only 35% could actually do that in reality, which is, that's

quite, that's quite a, a gap there.

Um.

Yeah.

Another interesting thing was that only 10% of businesses reported

no outages in the last 12 months.

Uh, which means that 90% tested their backup systems the hard way.

Uh, not quite as hard as, uh, our Alaskan friend, but, uh, which for those of

you that haven't heard that episode, uh, he tested DR system by deleting

the entire surfer for the entire.

Data center and then restoring, and that was his first test.

And it was like, gee, I hope it works.

Don't do it like that.

Um,

but hey, it worked out

yeah, exactly right.

Um, and remember, again, going back to the things that fit into the RTO, right?

You know, you, you also have to, to include things like

detecting that there's a problem.

Right, because the RTO clock starts the moment the outage happens, not

the moment the restore happens, right?

So, uh, the moment you have the outage and then you're like, what's going on?

Right?

Because so many times the, the symptom that gets your attention has nothing to

do with the thing that actually went bad.

Right.

Uh, I mean, it does have something to do with it, but it's not

the thing that went bad, right?

So you gotta figure that out.

You gotta understand how bad it is if it's a ransomware attack.

Again, you gotta figure out, you know, how bad this, you know, how big the scope is.

You might have to get approvals, um, you know, all these different things, right?

Yeah.

And well, just one thing to add to that, because I was thinking

about the, uh, what was the.

Company,

Rackspace.

with their hosted exchange.

Right.

I think one of the things to also consider.

Uh, when you're thinking about RTO is order to bring my app back up and

running, do I need to restore all my data?

an example?

Maybe I only need a subset of my data in order for my application to come up

and I can solely backfill all my old data that's archived or other things

like that, I can still get people up and running and ready to go without

waiting for everything to be done.

And so there might also be slight nuances depending on the application

of what the expectations are.

Yeah.

The other thing I would say regarding that Rackspace outage, if you're in

the middle of your recovery or you're about to begin your recovery, don't

change all the rules, right In, in their case, they're like, we tested how to

do this recovery, but you know, just before they went to do the recovery,

they're like, ah, what if we just move everything over to Microsoft 365?

Right?

And it's like, oh, well that would mean that we have to like.

Basically you, you can't, you can no longer restore the exchange

databases directly into the user.

Uh, you have to, um, you'll have to restore it and then migrate

the data over individually, which is a much bigger process.

Much, it's gonna just take much, much longer, and it ended up taking months.

You may recall, and there was a, uh, some lawsuits regarding that.

So make sure that whatever scenario in which you do, uh, you do the testing,

you, you, you have to do the testing.

So, um,

Yep.

I know we talked earlier about, okay, that 24 hour RTO for some businesses,

but there are some industries, right?

Where even like seconds make a big difference, right?

Yeah, definitely.

Yeah, definitely like financial trading firms, banking organizations, the more you

can attach a real number when you can say one hour of downtime costs us this much.

If, if you can do that, if the business can do that.

One, $1 billion.

Um, yeah, I'm sorry, I, I gotta do the, the pinky, right?

Um, if you can do that, the more you can do that, the, the, the, the

much more equipped you will be as a, you know, backup and dr. Person.

To be able to make enhancements to the backup and recovery system if needed.

Right.

So let's talk about, uh, some of the things that you

could do to close this gap.

Obviously, the first, the first thing is if you can have an

iterative discussion on, uh, okay.

You said you want one minute, we can do 10 hours.

Right.

Let's figure out, you know, let's get the, let's get the RTO set to somewhere near.

Um, you know, uh, realistic that we, that we can actually meet, right?

And you can, you can say, we're gonna set the RTO for now at this.

We're gonna move towards, uh, a better RTO at a later, a later time.

Um, any thoughts there?

Yeah, no, I think that makes sense because it also takes time to implement

new technologies because if, say for instance, your RTO is 10 hours based

on your existing infrastructure, and now they're like, oh, we need

an hour or 10 minutes, right?

You're now going to need to think of something very different that's

gonna elongate the time it takes, and so it really is important to

ask the question, do you need.

Yeah.

Yeah.

The help, just,

start with

yeah, everybody's gonna say zero and zero for your RTO and RPO, right?

So it is just, you gotta justify it and you gotta say, well, if it's

really worth $10 million every minute.

Then you need to give us, you know, whatever the number is.

Right.

Um, so then if we're gonna do testing, if we can automate that testing, the more we

can automate testing, the more the, you know, the better that things are gonna be.

Doing it very regularly, doing it small, it's sort of like, uh, the

same as opinions that I have on testing your, it's kind of like in

cybersecurity where you have a company that actively tries to send phishing.

You know, phishing tests to the users to see if, um, to

see if they fall for it, right?

Same thing here, where over there, more frequent, smaller bite-sized testing is

preferred to the once a year I have to do this and it takes two hours, right?

Keeping it on the mind, keeping a recovery mindset is really important.

So I think regular Dr. Drills is part of that.

And I think having the regular DR drills is important because if something

changes in your environment, you

Yeah,

rather than sort of that

exactly.

And then there's also the concept of chaos engineering, um, whi, which, you

know, like the chaos monkey, right?

You wanna talk about that?

Yep.

Yep, yep, yep.

Yeah.

So.

just try breaking things in your environment, see what happens and

see did I miss something that I wasn't backing up as an example.

Maybe you forgot a backup active directory, and now in order and something

happened to it, you lost all the data there and you realize, oh, I can't recover

my application because I don't actually have a backup of active directory.

And so you start to understand the dependencies in your

environment and point out sort of.

Issues that you might not foresee different failure scenarios or like if

the network goes down or The other thing is, it's not even just technology, right?

It might be even a person.

I know Curtis, you used to talk about at the bank doing

testing with someone who did not

Yeah, exactly.

Exactly.

Uh, well back then, I didn't write the book, but Yeah.

Yeah.

I wa I wasn't Mr. Backup yet.

I wa I was, I was Mr. Backup junior.

Um, and, and then, you know, the idea is if, if you do this, the whole, the

whole idea of doing this on a frequent basis to get better at it, to create

and, and, and improve your runbooks, to create and improve decision trees.

What do we do when this happens?

Right?

Um, we also, we didn't talk, uh, at all about, um, uh, tabletop exercises.

Those are, uh, obviously a great, uh, uh, tool here.

Uh, you know, do 'em at lunch.

Do 'em so that they're not like, so again, do them frequently in smaller.

We're not where the whole world isn't, you know, don't do 'em like just before

your, your performance review time.

Which makes them like much more stressful.

Do them frequently and, and, and have fun at it.

And, and then learn from it and improve your runbooks,

improve your decision trees.

Cross train your teams.

Do what we were talking about before.

Don't use the, don't rely on one person.

Uh, you know, you know, because that one person might not be available.

Uh.

You know, uh, at that time, right?

And then measure, again, measure and report reality.

Here's where we are.

Make sure that everyone is on the same page.

We've asked for this, we've agreed to this for now, we would like to get to here.

Here's where we are.

Report those gaps.

Uh, and then let, let business leadership decide.

What to do about that.

It is not your responsibility.

Right.

I do remember like

Yep.

bad because the backup system wasn't capable, but it's like, I'm not magic.

All I can do is make recommendations.

Right.

And I do remember, by the way, I do remember when I, and I had a shell

script that was doing everything right.

We had like, I dunno, like 50 servers and I was doing all this with like a, a Unix.

You know, shell script.

Right.

And at some point I couldn't, but, and all of it was based on that each

server could fit on a tape drive.

And then one day we bought a server that it didn't fit on.

50 tape drives, right?

On 50 tapes.

And, and it, it just, that and other servers that weren't

quite that bad, it just broke.

It broke my ability to do it right.

And I said, I'm just not, I can't do that.

And then, and I just went to the boss and I said, Hey, I can't do this.

And she said, well, aren't there like commercial products that do this,

that we can like spend money on?

Oh, because you flipping

I was, I was flipping, I was flipping out what I was doing,

and I, I remember feeling like a failure because I couldn't fix this.

Right.

I wasn't that good at scripting.

I, I, I don't think anybody could deal with the 50, you know, the right.

But I, I remember at the time feeling like a failure, and I guess I'm

saying don't try not to feel that way.

Right?

Go and give an honest assessment of where you're at.

And, um, even if you're the one that put you in that scenario, right.

Um, I, I remember another story of a guy that told me that he bought a

particular vendor's DDU product that had a 90% DDU tax, meaning that the restore

speed was 10% of the backup speed.

And, and he's like, I don't know what to do.

I'm like, well, you have to tell your boss.

And he's like, I'm the one that recommended the system.

It's okay.

You gotta be honest.

You get, because you can't get there from, you can't get there from here

if you, if you don't address that.

Right?

Yeah, so one question I wanted

Yeah.

Curtis.

There is a term though, right?

So you

Mm-hmm.

and then you have like, okay, you're doing these tests, you

actually figure out like, okay,

Yeah.

long it takes.

There's a term for that though, and I don't think it's

No, it's nowhere near.

Yeah,

it's

nowhere near as As.

Yeah, thanks.

Nowhere near as widely used as RTO.

Right?

And it's RTA recovery time actual.

Some people say recovery time reality, it doesn't matter.

Just have a different term.

Don't say.

Our RTO is an hour when what you're saying is this is how fast you can

recover your RTO is your objective.

Your other thing, I don't care what you call it, recovery time actual is good.

This is where we are.

The difference between your recovery time actual and your recovery.

Time objective is the gap that you need to address with whatever changes in

process, documentation, or quite possibly enhancements to your backup system.

Yeah.

All right.

I think, I think we've covered enough.

And then next, next week we're gonna recover recovery

point objective, which is.

It It is.

Yeah.

It's weird.

Like all the, yeah.

Uh, and this is going to be basically how much data we agree that we

can lose, which is something very different than how long the system is.

Yeah, exactly.

We should, we should recover in zero minutes and we should lose zero data.

We all agree.

That would be amazing.

Uh, it's also not gonna happen.

Well, thanks for, uh, thanks for joining me again.

I am.

Enjoy these.

I

Yeah, I'm glad.

Yeah.

more.

I think now, now that we, you've figured out your world and I figured out my

new world, uh, we, we should be good.

And, uh, thanks to the listeners you're, why we do this.

Uh, that is a wrap.