How to Set Realistic Recovery Point Objective (RPO) Goals
Want to know how much data you're really willing to lose? We're breaking down recovery point objective RPO - the agreement about how much data loss you can accept, measured in time. Most organizations have RPOs that are pure fantasy, claiming they can only lose an hour of data when they're backing up once a day. Curtis and Prasanna discuss why RPO matters, how ransomware scenarios can force you to accept more data loss than planned, and the difference between your stated RPO and your actual backup frequency. Learn practical strategies for rightsizing your backup schedule, using database transaction logs to minimize data loss, leveraging snapshot-based backup technologies, and protecting your SaaS applications like Microsoft 365 and Salesforce. From incremental backups to continuous data protection, discover how modern backup technology can help you meet your recovery point objective RPO targets without overwhelming your infrastructure.
You found the backup wrap up your go-to podcast for all things
Speaker:backup recovery and cyber recovery.
Speaker:In this episode, we're talking about recovery point objective or RPO, which
Speaker:is how much data that you're willing to lose when things go sideways.
Speaker:Spoiler alert, most people's RPOs are complete fantasy.
Speaker:I mean, you think you can only lose an hour of data, but
Speaker:you're backing up once a day.
Speaker:That's a problem.
Speaker:We'll break down what RPO really means, why it's measured in time and not, uh,
Speaker:the amount of data and how ransomware can totally mess up your carefully planned.
Speaker:Objectives.
Speaker:Plus I'll share some practical ways to rightsize your RPO.
Speaker:Talk about database transaction logs and explain why your SaaS apps need the
Speaker:same love as your on-premises systems.
Speaker:Let's talk RPO.
Speaker:By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.
Speaker:Backup, and I've been passionate about backup and recovery for over 30 years.
Speaker:Ever since I had to tell my boss there were no backups of
Speaker:that database that we just lost.
Speaker:I don't want that to happen to you, and that's why I do this.
Speaker:On this podcast, we turn unappreciated admins into Cyber Recovery Heroes.
Speaker:This is the backup wrap up.
Speaker:Hi, and welcome to the backup wrap up.
Speaker:I am your host, w Curtis Preston, AKA, Mr. Backup, and I have with me a guy who I
Speaker:called while laying upside down yesterday.
Speaker:Prasanna Malaiyandi, how's it going?
Speaker:Prasanna,
Speaker:I am good Curtis, and do you wanna tell the listeners what you were doing?
Speaker:Laying upside
Speaker:I was underneath my Tesla for the first time since I bar, uh, borrowed
Speaker:it since I bought, bought it.
Speaker:Two years ago, uh, I, I did a, I made a boo boo and I, um, I did a, you know, over
Speaker:here in California, I think it's worse here in California than other places.
Speaker:We have these, like, you know, when you go through an intersection, there's the big
Speaker:dips before and after the intersection, and you, you seem to, I, I just think.
Speaker:We don't get rain, but we get it.
Speaker:We get it in torrents.
Speaker:And so they have these like huge dips at the beginning of many intersections.
Speaker:And if you're not paying attention, you can easily bottem out.
Speaker:And I apparently, I found out after the fact that I apparently bottemed
Speaker:out so hard that the two bolts that held the little, um, they're
Speaker:little 10 mil, 10 millimeter bolts.
Speaker:But the two bolts that hold on this.
Speaker:Like what?
Speaker:What's that?
Speaker:It.
Speaker:Under tray.
Speaker:So under trade, it's like protects the undercarriage of the car.
Speaker:Uh, I just sheared them off.
Speaker:And then for apparently a while, it had been held on by two other 10
Speaker:millimeter bolts, which weren't screwed into anything other than the plastic.
Speaker:Uh, fascia, right?
Speaker:And so then at some point that, uh, you know, didn't work.
Speaker:And then, uh, so that came off.
Speaker:So anyway, so I had to rip all that off and put it all in.
Speaker:And it wasn't until I did all this, I, I bought a new shield and
Speaker:I went to go screw it in there.
Speaker:And by the way, that meant lifting up a Tesla, which for the record.
Speaker:Ain't no walk in the park.
Speaker:Yeah.
Speaker:that was interesting.
Speaker:But then I called you, then I called you.
Speaker:I was like, FaceTime.
Speaker:And you're like, uh, because I don't normally FaceTime you.
Speaker:You're like, what am I looking at?
Speaker:I'm like, uh, look at the underside of my car.
Speaker:It is really weird though, right?
Speaker:Like looking inside, something like that, that like, like I kind
Speaker:of know what I'm looking at when I'm looking at a gas car, right?
Speaker:But there's all these parts and none of them.
Speaker:Are familiar, right?
Speaker:I'm like, okay, I, I understand steering parts, right?
Speaker:Um, and, um, uh, and so I had to, and, and did, I did, ultimately I had to buy.
Speaker:What, what, what's the, what's the part called the, uh.
Speaker:bar.
Speaker:The, well, the, the stabilizer bar, but then the bushing for the stabilizer bar
Speaker:and then these two brackets that hold the bushing that hold the stabilizer bar.
Speaker:That's what I had to buy.
Speaker:And uh, amazingly I got them for $15 each from Amazon shipped and two days for free.
Speaker:Ugh,
Speaker:Nice.
Speaker:nice.
Speaker:Um.
Speaker:that is gonna be your job.
Speaker:That is gonna be my job, luckily.
Speaker:Yeah.
Speaker:Now, now that I've figured all that stuff out, it, it'd probably be 20 minutes.
Speaker:Um, the hardest part will be lifting up the car.
Speaker:Make sure that you have all the hardware before you take stuff apart.
Speaker:I, I, yeah.
Speaker:Okay.
Speaker:Yeah.
Speaker:What do you think you're dealing with here, Mr.
Speaker:just make sure that you have enough, 'cause I don't know what
Speaker:you actually got with the kit versus like what you might need when you
Speaker:All I, all I need is the bracket.
Speaker:I got all the other stuff right.
Speaker:The thing is that when I went to do this five minute job,
Speaker:I realized that I had to take, I had to do.
Speaker:Like I had to take off other parts to get to because I realized basically
Speaker:as I did this more and more, and I realized that by the end, by the time
Speaker:I was in my, my recovery point, uh, was very different than, uh, what I
Speaker:originally had, had, had envisioned.
Speaker:But, um, because, uh, just basically I lost so much more than I had originally
Speaker:planned to lose underneath my car.
Speaker:Yeah.
Speaker:So currently my car looks worse underneath than it did when I started.
Speaker:Um.
Speaker:But that's okay.
Speaker:Sometimes you gotta take a couple steps backwards in order to move forward
Speaker:Sure we'll do that.
Speaker:something like that.
Speaker:Hopefully you don't do that with a, with a, with a recovery point.
Speaker:So today we are talking about recovery point objective, which I would
Speaker:define very quickly as saying it.
Speaker:It is just how much data we agree we're allowed to lose as measured by time.
Speaker:Right.
Speaker:So it's not like what.
Speaker:Why would you ever lose data?
Speaker:Curtis isn't backup, supposed to be?
Speaker:Never lose data.
Speaker:Yes.
Speaker:Yes.
Speaker:We'll get to that.
Speaker:Uh, and also why, again, just like RTO, most people's RPOs
Speaker:are complete fantasy, right?
Speaker:Um, so.
Speaker:Uh, basically again, it's as measured by time, so it's not like we agree
Speaker:we're gonna lose, um, 10 gigabytes of data or 10 terabytes of data.
Speaker:We agree that we're gonna lose or allow to lose up to 12 hours of data, 36
Speaker:hours of data, whatever the number is.
Speaker:And again, just like with our TO it, uh, which we just did an
Speaker:episode on recovery time objective.
Speaker:If you didn't see that one, then go, you know, uh, and again, you can watch
Speaker:these either on YouTube or you can listen to 'em on your favorite pod catcher.
Speaker:Um.
Speaker:Is that different scenarios, different recovery scenarios.
Speaker:We'll probably call for different RPOs.
Speaker:Right.
Speaker:Uh, a ransomware scenario is probably, once again, possibly you're going to
Speaker:have to accept more data loss than you would in just a regular recovery.
Speaker:Right.
Speaker:Regular loss of a server or whatever.
Speaker:Right.
Speaker:Well, because you might.
Speaker:Find out that even your, some of your backups are corrupted, right?
Speaker:That, that you've been backing it up for a week and it was, some part of
Speaker:it was encrypted two weeks ago, right?
Speaker:And so you might, you might have to recover to some, to some older ba you
Speaker:know, from some older backup, right?
Speaker:Uh, again, that, that is something that, that can happen, uh, as a, as a recovery.
Speaker:Um, and
Speaker:I,
Speaker:go ahead.
Speaker:I have another question on RPO.
Speaker:Yeah.
Speaker:So it's the recovery point.
Speaker:You said it's the amount of data you agree to lose, right?
Speaker:Is it the amount of data you agree to lose since your last successful backup?
Speaker:Or is it the amount of data that you're willing, like, can
Speaker:you define that a
Speaker:bit clearer?
Speaker:great, great great question.
Speaker:So it's the amount of data that we, uh, agree to lose.
Speaker:Period.
Speaker:Right.
Speaker:Uh, as measured by time, what will determine the amount of data you
Speaker:actually lose is the last successful backup that you're able to use.
Speaker:That actually rhymed.
Speaker:Let me wrap that.
Speaker:Um, so.
Speaker:Your, your, your RPO happens, let's say, um, basically it's
Speaker:the, the time is measured.
Speaker:It's measured backwards.
Speaker:Uh, whereas our RTO is measured forwards from the, uh, outage.
Speaker:RPO is measured backwards.
Speaker:So from the point of the outage.
Speaker:How far back are we allowed to go and still consider it to be successful?
Speaker:So if I, if this is a database and we're using, uh, redo logs and transaction
Speaker:logs, hopefully you can actually restore right up to the point of failure, like
Speaker:right up to just before the point of failure, even in a ransomware scenario.
Speaker:Right.
Speaker:Yeah.
Speaker:Uh, because.
Speaker:Generally with databases, if you start encrypting it, it's
Speaker:gonna, it's encrypt everything.
Speaker:Or you know, the moment you encrypt any part of the database, the whole
Speaker:database is gonna crash, right?
Speaker:So if you've got transaction logs and those transaction logs are being
Speaker:protected, key thing there, right?
Speaker:And they're being shipped off to some other system that hasn't been
Speaker:attacked, then, um, which would be part of your recovery system, right?
Speaker:Yep.
Speaker:Then you should be able to just go back minutes, right?
Speaker:Uh, again.
Speaker:That's just the restore, right?
Speaker:It's gonna take a while to figure out which, which things we're gonna
Speaker:restore if this is a ransomware event.
Speaker:But if it's a file on the opposite end of that, if it's a file system,
Speaker:Mm-hmm.
Speaker:you may have sys, you may have files in there that have been getting
Speaker:encrypted over time for months.
Speaker:Right?
Speaker:Um, because the dwell time.
Speaker:Do you want to, you wanna define dwell time?
Speaker:Yeah.
Speaker:The dwell time is how long ransomware sits in your system before it starts
Speaker:doing something or before it's detected.
Speaker:Yeah, so the dwell time may be measured in months.
Speaker:There was, we, we covered one, uh, a little while ago that it was like a year.
Speaker:Right.
Speaker:Um, and.
Speaker:A.
Speaker:Yeah.
Speaker:Yeah.
Speaker:That, yeah, that was literal, that wasn't that long ago, but
Speaker:I was like, which one was that?
Speaker:That was like three weeks ago that we did that.
Speaker:Um, yeah, that was an interesting story, right?
Speaker:Where, where it happened over a year.
Speaker:And so if they, if they're just, if they're really trying to mess
Speaker:with you, they're going to encrypt little files here and there.
Speaker:Possibly ones with older, um, access times, right?
Speaker:That haven't been looked at in a while.
Speaker:Um.
Speaker:And so that's gonna be very complicated, right?
Speaker:And you're, now that I think about it.
Speaker:The RPO is almost an irrelevant concept there because we typically talk about RPO
Speaker:from a server standpoint or an application standpoint, or a file system standpoint,
Speaker:but for a file system that has been being encrypted over time, the RPO is actually
Speaker:going to be many, many little RPOs.
Speaker:Right, right,
Speaker:because you're always looking for what's the valid data and
Speaker:try to pull the newest data
Speaker:right.
Speaker:the
Speaker:Which may reso, which may actually be thousands, potentially tens of
Speaker:thousands of individual restorers rather than, uh, and hopefully you
Speaker:can script that, uh, in the, in the book, um, that we, that we are, we are
Speaker:finishing the editing of right now.
Speaker:Literally.
Speaker:Yeah.
Speaker:Uh, that would be learning ransomware response and recovery.
Speaker:I actually wrote a little script that could, that could basically
Speaker:comb your way through a file system.
Speaker:It's a very basic script, but it's just an idea that it could give you, if you
Speaker:could comb through the file system, find the files that are encrypted, and
Speaker:then find the oldest or the find the most recent version of that file that
Speaker:wasn't encrypted and restore that file.
Speaker:So you're, you're actually doing many little restores and
Speaker:hopefully you can automate that.
Speaker:Yeah.
Speaker:But the, the, the point is that your, your RPO is the, that amount
Speaker:of time that you agree that you can, you know, uh, how much you can lose.
Speaker:Right?
Speaker:The RPO is measured.
Speaker:Going backwards in time from the incident , we can say we're gonna lose three hours
Speaker:worth of data, whatever it is, one hour's worth of data, two weeks worth of data,
Speaker:whatever time you've agreed on that is what your recovery point objective is.
Speaker:Whether or not you can meet that or not would we would call that
Speaker:recovery point actual, right?
Speaker:Um, and so the difference would be, you know, again.
Speaker:The, yeah, the gap between the two,
Speaker:Yeah.
Speaker:potentially an issue, which you might need to look at now.
Speaker:One thing I wanted to ask you, Curtis, is like as a backup, if I was a backup admin,
Speaker:Yeah.
Speaker:I, I do not play a backup admin either on TV or on this podcast or
Speaker:anywhere else, just to be clear, right?
Speaker:But as a backup admin, am I the one just sort of going to be like, Hey, yeah, I
Speaker:think we can lose like one hour of data.
Speaker:Like
Speaker:Y
Speaker:who is making that decision?
Speaker:Yeah, great question.
Speaker:Just like RTO, the answer is absolutely not right?
Speaker:You should never be making any procedural decisions like that, right?
Speaker:Um, this is a, well, this is a policy decision, right?
Speaker:Um, this is something that must be determined by the,
Speaker:um, the, the business, right?
Speaker:Mm-hmm.
Speaker:the longer the RPO is the, the more.
Speaker:Work you're going to have to redo.
Speaker:So the question is, how possible is it that we can redo this data?
Speaker:Right?
Speaker:So if it's, um, I don't know if it's customer records.
Speaker:If it's a, if it's a customer, uh, database of orders, is there some
Speaker:other system that you have where you've, uh, you know, whenever you do
Speaker:an order, you do a PDF of the order, you email that PDF to the customers.
Speaker:How f how much effort is it going to take us to go back into all of our
Speaker:outgoing emails from the the CRM system?
Speaker:Look at all of the invoices for all the orders that we said we were going to
Speaker:send, and then double check those against the orders that, uh, and you probably
Speaker:don't have to double check it too hard.
Speaker:You can say, the incident happened today at noon.
Speaker:We had to recover to yesterday at midnight.
Speaker:So we have all the, all the emails between those two different times
Speaker:and, um, and then go and just reenter those orders manually.
Speaker:That is, there is a cost associated with that, number one.
Speaker:Number two, you may have systems where.
Speaker:There isn't a backup, right?
Speaker:You may have an e-commerce site that where customers can go to that site.
Speaker:Put in requests and then that issues, um, you know, an an order
Speaker:and now it's, no one's actually seen any of this stuff, right?
Speaker:No one's looked at this stuff and then an outage happens that
Speaker:that is irreplaceable data.
Speaker:You're never gonna get that data back.
Speaker:Right?
Speaker:Um, and so again.
Speaker:Th that will be measured both in terms of perception, uh, business perception,
Speaker:Mm-hmm.
Speaker:also there may be actual loss of revenue.
Speaker:Now, quite possibly what will happen is you will, um.
Speaker:Get a, a phone call from somebody going, Hey, man, where the hell's
Speaker:my, where the hell's my thing?
Speaker:I asked from my thing and, uh, it's not there.
Speaker:Or like, uh, let's say, you know, I order from Amazon a lot.
Speaker:I, I, I went back into my account and I know I ordered a butcher Majer
Speaker:yesterday and it's not even, not only do I not have it yet, it's uh,
Speaker:it's not even listed in my orders.
Speaker:What the hell happened.
Speaker:So you might get some of that business back, but it will, you'll suffer a.
Speaker:A severe reputational, uh, damage.
Speaker:Yeah.
Speaker:Yeah.
Speaker:And so this is the amount of data you could lose.
Speaker:Now, I'm sure if you went to the business, right, and this is
Speaker:coming from the business, right?
Speaker:They're probably gonna tell you, I can't afford to have any data loss.
Speaker:Yes.
Speaker:How as a backup admin, are you supposed to respond to that question?
Speaker:Well, you, you say, well, the first thing you say, well, our current ability.
Speaker:Mm-hmm.
Speaker:Right, based on testing, we've done testing, right?
Speaker:'cause you're always gonna be doing testing, right?
Speaker:Yes.
Speaker:Okay.
Speaker:yes.
Speaker:So hopefully we've done testing and we've, we've figured out that
Speaker:based on our current system, right?
Speaker:So if you're, if you're having this, this conversation for the first time, right?
Speaker:Uh, you know, I listened to this podcast and Curtis and PSA
Speaker:said, I need an RPO and an RTO.
Speaker:You ask it and they go, it's zero and zero, right?
Speaker:You say, okay.
Speaker:Good job.
Speaker:Uh, thanks for giving me a number that I can work with.
Speaker:And then you say, well, we can currently do three weeks, so.
Speaker:Let's meet somewhere in the middle, right?
Speaker:Um, and just like with our, with our to, we wanna see if we can pull them back.
Speaker:But you, you should be able to pretty much, I mean, short of zero, right?
Speaker:You will always lose some data and it will always take some amount
Speaker:of time to do the restore, even if it's an instantaneous restore.
Speaker:There's still some time, especially if we're talking a ransomware
Speaker:attack, because again, you're gonna spend most of your time figuring
Speaker:out what you need to restore.
Speaker:You say to them, okay, if what you want is zero, which I'm gonna translate into
Speaker:Or
Speaker:less, less than, than one hour, right?
Speaker:If you want less than one hour, RPO and RTO.
Speaker:Then we're going to need to do this.
Speaker:And this is, and, and I, I, I've gotten a ballpark number and
Speaker:it's gonna be $20 million, right?
Speaker:And then they go, okay.
Speaker:Yep.
Speaker:Right.
Speaker:Or maybe they go, holy crap.
Speaker:No.
Speaker:Uh,
Speaker:day is.
Speaker:yeah.
Speaker:Well, yeah, one, one day.
Speaker:One day's good.
Speaker:Either they adjust their expectations, right?
Speaker:Or, uh, they give you the money.
Speaker:Right.
Speaker:Or, or somewhere in the middle.
Speaker:Right.
Speaker:They, they adjust their expectations, but they give you less money.
Speaker:And you, you may be surprised, you know what they may do because it really,
Speaker:the RPO and RTO are determined by how much money, what's the financial
Speaker:impact to the organization going to be?
Speaker:And you have to, like, if it's just a reputational impact, you
Speaker:have to measure that in terms of.
Speaker:Yeah.
Speaker:Money.
Speaker:And if you say, look, um, we're a company that currently generates
Speaker:$50,000 a month in revenue, right?
Speaker:So we're, you know, that's $600,000 a year.
Speaker:Uh, we can't spend $3 million on a backup system, right?
Speaker:Uh, but if we're a company that does $50,000 in an hour,
Speaker:Yeah.
Speaker:then uh, we can justify pretty much anything.
Speaker:Yeah, the other thing to also remember is.
Speaker:From a backup technology perspective, as you start to reduce your
Speaker:RPO and RTO, it's not linear in terms of cost, It's exponential.
Speaker:Like to go from like 24 hours to one hour, down to one minute, down to one second,
Speaker:Yeah.
Speaker:like it's a significant increase in cost.
Speaker:Yeah, because you start, you start doing real time protection at that point, right?
Speaker:Um, you start talking about things like, you know, continuous data protection
Speaker:or near continuous data protection.
Speaker:Um, or, you know, full, full, just full, um, replication without
Speaker:really, because, you know, one of the things I often say is that like.
Speaker:Replication's great.
Speaker:And you could get a zero minute RPO or really close to it.
Speaker:The problem is it doesn't go backwards, right?
Speaker:So if you do need to go back even one minute, it's just
Speaker:simply incapable of that.
Speaker:So I'm not a fan of replication by itself as a, uh, as a protection
Speaker:mechanism, but if you, if you have replication that somehow has the
Speaker:ability to go back in time, which I would call continuous data protection.
Speaker:Or I'd say a one hour RPO, it's so much easier to do than a,
Speaker:than a one minute RPO, right?
Speaker:percent agree.
Speaker:Yeah.
Speaker:Because with a one hour RPO, you take one, you take hourly snapshots, you
Speaker:replicate 'em, you're good to go.
Speaker:There are.
Speaker:Myriad systems that will do that.
Speaker:Right.
Speaker:Starting from your former employer.
Speaker:Uh, you know, you know, they probably, you know, NetApp, uh, probably
Speaker:really perfected that I think.
Speaker:Um, you know, and, uh, but they're, but they're now, you
Speaker:know, a lot of fast followers that have that, that are doing that.
Speaker:Right.
Speaker:Um, and, um.
Speaker:But if you want to, if the, the number of companies that do true
Speaker:real time data protection down to the sub minute, that number is very
Speaker:small and the price is very high.
Speaker:Right?
Speaker:Yeah.
Speaker:Um, there's a lot of dead soldiers in that field, right.
Speaker:Companies that tried to do it
Speaker:yeah.
Speaker:backed off, or ultimately got acquired for, you know, basically
Speaker:it was like a furniture sale.
Speaker:Yeah, my former employer happens to be one of those who's
Speaker:very successful in that space
Speaker:Nice, nice.
Speaker:So the number one thing that determines your RPO is going to
Speaker:be your backup frequency, right?
Speaker:So if you are backing up once a day.
Speaker:twice a day.
Speaker:Mm-hmm.
Speaker:Yep.
Speaker:Yep.
Speaker:Which, as you know.
Speaker:Isn't always the case.
Speaker:Is it always the case?
Speaker:Well, and here's here.
Speaker:Okay, here's an important question that I've always had.
Speaker:Uhhuh?
Speaker:Okay, so you finished a backup yesterday, right?
Speaker:Say the backup.
Speaker:it was a snapshot based backup.
Speaker:It started at midnight yesterday,
Speaker:Yeah.
Speaker:right?
Speaker:And it takes two hours to transfer the data.
Speaker:Yeah,
at 2 00:22:25
00 AM your backup, your recovery point Objective.
at 2 00:22:29
Is midnight
at 2 00:22:30
yeah.
at 2 00:22:31
Right, because that's when
at 2 00:22:32
Well, your recovery point.
at 2 00:22:34
You're sorry,
at 2 00:22:34
Your recovery point is midnight.
at 2 00:22:36
Uh, you said it started at midnight and then it replicated it.
at 2 00:22:39
Yeah.
at 2 00:22:39
yeah.
at 2 00:22:40
Okay.
at 2 00:22:40
Now
at 2 00:22:41
Well that's assuming that because Are we taking it every hour?
at 2 00:22:46
No.
at 2 00:22:46
What today.
at 2 00:22:47
Oh, once a day.
at 2 00:22:48
Okay.
at 2 00:22:48
Yeah.
at 2 00:22:48
Yeah.
at 2 00:22:49
Okay.
at 2 00:22:49
So now the next backup will happen at the next midnight.
at 2 00:22:54
Right?
at 2 00:22:56
And until it shows up, which, let's just say it takes two hours.
at 2 00:23:01
The recovery point you use is a previous night spend night
at 2 00:23:04
Correct.
at 2 00:23:05
So technically, even though your backup frequency is set for 24 hours, RPO may
at 2 00:23:12
actually exceed your backup frequency.
at 2 00:23:15
Your RPA may exceed your backup frequency.
at 2 00:23:19
Yes.
at 2 00:23:19
Your RPA.
at 2 00:23:20
Yes.
at 2 00:23:20
Uh, so yes.
at 2 00:23:24
Right.
at 2 00:23:25
And because that's why I'm saying like the best you're gonna be able to do.
at 2 00:23:28
Right?
at 2 00:23:28
Uh, it really depends on when that.
at 2 00:23:30
When that actual incident happened.
at 2 00:23:32
So it's gonna be based on when the incident happened, it's gonna be based on
at 2 00:23:36
whether or not last night's backup worked.
at 2 00:23:38
Yep.
at 2 00:23:39
Um, do you do backups on the weekend?
at 2 00:23:40
I hope so.
at 2 00:23:41
Right.
at 2 00:23:42
Uh, because I, I've worked places where they, their last
at 2 00:23:47
backup was Thursday night.
at 2 00:23:50
Right.
at 2 00:23:51
And now it's Monday morning and they're gonna do their next backup Monday night.
at 2 00:23:56
Yeah.
at 2 00:23:57
If you have an outage on Monday and you did any work over the
at 2 00:24:00
weekend, you're gonna lose Friday, Saturday, Sunday, and Monday.
at 2 00:24:05
Right.
at 2 00:24:05
Um, so it, again, it's determined by your backup frequency and then, and
at 2 00:24:09
then any failures, uh, which again, only you, uh, can prevent forest fires.
at 2 00:24:15
Sorry, that's, that came out only you were gonna know what
at 2 00:24:18
your, what your actual, uh.
at 2 00:24:20
Yep.
at 2 00:24:21
You know, real recovery time or, uh, recovery success rate is right.
at 2 00:24:26
Um, and also, uh, you know, if, if backups get corrupted, uh, anything
at 2 00:24:32
like that, especially if backups get corrupted by, um, you know,
at 2 00:24:36
ransomware or anything like that.
at 2 00:24:38
Right.
at 2 00:24:38
Um, that's a, that's a good question.
at 2 00:24:41
So how do people actually test their RPO in order to determine their RPA
at 2 00:24:52
So, yeah, so good, good, good question.
at 2 00:24:55
Yeah, I, I, it's a little different than RTO, right?
at 2 00:25:00
And because really it's just, it's.
at 2 00:25:07
You, you don't really test it like, like you, because you, uh,
at 2 00:25:11
that's a, that's a great question.
at 2 00:25:13
You, it's, it's going to be the result of whatever your, your RTA is.
at 2 00:25:17
Right?
at 2 00:25:17
But you, you, again, it, it's more a discussion like how, how frequent are
at 2 00:25:24
our backups and how frequent are they?
at 2 00:25:27
Do they fail?
at 2 00:25:28
Yep.
at 2 00:25:29
Right.
at 2 00:25:29
Um, and then what you should do is you, you just like.
at 2 00:25:34
You, you report on what your compliance is, right?
at 2 00:25:38
Yeah.
at 2 00:25:38
And if, if the compliant, if the number starts creeping up or it
at 2 00:25:42
starts, like, hopefully you should, you should say, look, if a backup
at 2 00:25:47
fails more than once, then like all it should be all hands on deck, right?
at 2 00:25:51
Because it's bad enough that we're gonna lose, let's say, 24 hours worth of data.
at 2 00:25:55
Well now we're talking 48 hours.
at 2 00:25:57
And if it fails again, now we're talking 72 hours.
at 2 00:25:59
This is a huge amount of business data that you're losing.
at 2 00:26:03
So really.
at 2 00:26:04
It's not so much you can test, it's just, it's something you can monitor.
at 2 00:26:08
I think you can just monitor how well you're frequently backing
at 2 00:26:11
up and how well it's working.
at 2 00:26:13
and it looks like that's such a low bar compared to actually doing
at 2 00:26:17
like the recovery time testing
at 2 00:26:19
Yeah.
at 2 00:26:20
you should be able to do this
at 2 00:26:22
I.
at 2 00:26:22
easily.
at 2 00:26:23
Like there should be no excuse for you not to know what your RPA is.
at 2 00:26:26
Correct.
at 2 00:26:27
No excuse.
at 2 00:26:28
Um, and again, the better thing you can do to do RPA is to switch to,
at 2 00:26:32
you know, well, well, let's, we'll get to get to that in a second.
at 2 00:26:35
Um, yeah, absolutely.
at 2 00:26:36
Right.
at 2 00:26:37
So the, the first thing, again, this is like, uh, it's like
at 2 00:26:41
the 12 step process, right?
at 2 00:26:43
The first thing is to acknowledge that you're powerless over your RPO.
at 2 00:26:46
Okay.
at 2 00:26:47
Sorry.
at 2 00:26:48
So, so acknowledge you're an honest assessment.
at 2 00:26:51
Right of, um, of where you are.
at 2 00:26:55
Right.
at 2 00:26:55
You, you, you say you don't wanna lose an hour's worth of data.
at 2 00:26:57
We currently back up once a week.
at 2 00:26:59
Uh, this is a problem, right?
at 2 00:27:00
You have to do that.
at 2 00:27:01
Then you can rightsize the frequency.
at 2 00:27:04
You, you, you know, how quickly can you do that, right?
at 2 00:27:07
Maybe, maybe it's such, maybe it's, it's like, look, we currently
at 2 00:27:11
back up once a day, right?
at 2 00:27:14
Can we potentially back up, let's say.
at 2 00:27:17
Like, I don't know, during the day, right before the day.
at 2 00:27:20
Right after the day.
at 2 00:27:21
Um, you know, it depends on how your business works, right?
at 2 00:27:25
Um, could you potentially just tweak your, how frequently you can do it?
at 2 00:27:29
Um, and if you've got an incremental base backup system, remember that
at 2 00:27:35
if many cases, if not most cases, four backups throughout the day.
at 2 00:27:41
Take roughly the same amount of time as one backup once a day, right?
at 2 00:27:46
Unless what we're talking about is backing up the same data multiple
at 2 00:27:48
times because it's been, you know, changing throughout the day.
at 2 00:27:51
Right?
at 2 00:27:52
Databases.
at 2 00:27:52
Yeah.
at 2 00:27:53
Um, but like with databases, what you can do with databases is just
at 2 00:27:56
back up the transaction logs,
at 2 00:27:58
Yeah.
at 2 00:27:58
make sure that the transaction logs are getting backed up and sent to immutable
at 2 00:28:02
storage, uh, throughout the day.
at 2 00:28:04
That's the way you don't have to back up the whole database just
at 2 00:28:07
to get those transaction logs.
at 2 00:28:08
It may take longer to recover, but at least you won't lose the data.
at 2 00:28:11
Right.
at 2 00:28:12
Yeah.
at 2 00:28:12
Oh, that's a good idea.
at 2 00:28:13
Yeah.
at 2 00:28:14
Um, and then of course, again, backup validation.
at 2 00:28:17
Do the testing, see how long it takes, um, you know, you know, all
at 2 00:28:21
of those different technologies.
at 2 00:28:23
And then potentially consider, um, uh, a change in backup technology.
at 2 00:28:30
Right.
at 2 00:28:30
Again, either CDP or near CDP, uh, you know, the, the, um.
at 2 00:28:37
The, these are things that are your friend.
at 2 00:28:39
Generally speaking, many if not, most of those are storage based,
at 2 00:28:45
meaning that you will need to go to a different type of storage system in
at 2 00:28:50
order to get snapshot based back up.
at 2 00:28:52
That's not a hundred percent true, but there are systems like data core, right?
at 2 00:28:56
And I, I'm sure there are others where it can work with your existing
at 2 00:28:59
storage, but in most cases what people are doing is they're saying,
at 2 00:29:02
we're gonna buy Product X, right?
at 2 00:29:04
And, and we're gonna get snapshot based backup, we're gonna do
at 2 00:29:08
snapshots plus replication.
at 2 00:29:10
And just a just one final note on, on the RPO and sort of changes into technology.
at 2 00:29:17
Make sure you're taking into account.
at 2 00:29:19
Your SaaS applications, they're, they're, the RTO is gonna be very
at 2 00:29:24
different from SaaS apps, right?
at 2 00:29:25
Especially if the app itself is down, but you are, just make sure that you're also
at 2 00:29:31
looking at your SaaS apps like Microsoft 360 Fives and Salesforce where you're
at 2 00:29:35
generating data throughout the day.
at 2 00:29:37
Are there ways that you can incrementally back that up as well throughout the day?
at 2 00:29:41
The more modern backup technology that you're using, the easier it will
at 2 00:29:46
be to meet your RPO, uh, and because many, if not most modern backup
at 2 00:29:54
applications or SaaS backup applications.
at 2 00:29:58
They're doing deduplication based, replication based, very minimal
at 2 00:30:04
incremental backups throughout the day, stored in such a way that you
at 2 00:30:07
could very easily restore right up to the point of failure, assuming we're
at 2 00:30:11
not talking about ransomware, right?
at 2 00:30:13
Um, so just make sure you're taking all of the different parts of your
at 2 00:30:18
environment into, um, into play.
at 2 00:30:22
Any thoughts?
at 2 00:30:24
no, I think that's, yeah, I was actually wondering, 'cause in the RTO
at 2 00:30:28
episode, we didn't bring up SaaS app, so
at 2 00:30:31
Uh, yeah.
at 2 00:30:33
Well, because yeah, no, that's a good point.
at 2 00:30:35
Yeah.
at 2 00:30:36
I mean, again,
at 2 00:30:37
Yeah.
at 2 00:30:37
just whatever you have, whatever your environment is, you should be testing
at 2 00:30:42
recovery and of, of that thing.
at 2 00:30:44
Right.
at 2 00:30:46
Um, and, um,
at 2 00:30:48
not special.
at 2 00:30:49
what's that?
at 2 00:30:50
SaaS apps are not
at 2 00:30:51
They're not well, they are special and, and that people think they're
at 2 00:30:55
special, but they're not special.
at 2 00:30:58
They're just the same.
at 2 00:30:59
They have you, you are as responsible for that data.
at 2 00:31:03
And by the way, Microsoft finally gave in, they're now offer a
at 2 00:31:07
backup service at an extra cost.
at 2 00:31:10
To me, that's admitting the fact that.
at 2 00:31:13
You need a backup service.
at 2 00:31:15
Uh, and I would prefer, and again, nothing against Microsoft, right?
at 2 00:31:18
They, they do a great job with Microsoft 365.
at 2 00:31:21
I would still personally use a third party for the backup.
at 2 00:31:24
I would, and, and that's Salesforce.
at 2 00:31:26
Salesforce has a backup service.
at 2 00:31:27
I would use somebody else for the backup service.
at 2 00:31:30
Um, and that's not just because I used to work for one of the companies that
at 2 00:31:34
made, had one of those backup services.
at 2 00:31:36
It's just, I just, you know, sometimes.
at 2 00:31:40
Yeah.
at 2 00:31:40
When we read these stories about things that happen at vendors, we're like, oh my
at 2 00:31:44
God, I can't believe they did that thing.
at 2 00:31:47
Plus that thing, plus that thing.
at 2 00:31:49
And then of all of those things, it's like when, when I think about
at 2 00:31:52
like what happened at OVH in France.
at 2 00:31:54
And you're like, oh, it's that, that thing plus that thing.
at 2 00:31:57
Plus that thing.
at 2 00:31:58
So not only did they have like these container based storage things,
at 2 00:32:01
and not only did they have the, you know, and they were sharing power and
at 2 00:32:05
they were sharing, you know, and you know, the backup system was sitting.
at 2 00:32:08
Right.
at 2 00:32:09
You know, they said it was physically separate and by physically separate,
at 2 00:32:11
they meant it is over there.
at 2 00:32:13
Right.
at 2 00:32:13
It's, it's on the other side of the, the other side of the container.
at 2 00:32:17
You're just like, all of this logic.
at 2 00:32:19
When, when, when the logic is bad.
at 2 00:32:23
That bad logic can extend to, um, you know, um, and so again, not to pick on
at 2 00:32:30
Microsoft, but they're not perfect, right?
at 2 00:32:32
Um, this is a company when Microsoft 365 went down simply because somebody forgot
at 2 00:32:38
to renew the Cate certificate, right?
at 2 00:32:41
Uh, again, they're not perfect, right?
at 2 00:32:43
Um, so, uh, and the people that you have that administer the apps are not perfect.
at 2 00:32:48
So I, again, I would prefer to have it as a third party app, but.
at 2 00:32:51
Anyway, I digress.
at 2 00:32:53
All right.
at 2 00:32:54
Well thanks for chatting about RPO.
at 2 00:32:57
Thank you Curtis.
at 2 00:32:58
And hopefully everything turns out okay on the Tesla
at 2 00:33:01
Yeah.
at 2 00:33:02
uh, I might expect, uh, FaceTime, I'm guessing tomorrow maybe, maybe not
at 2 00:33:08
I'll, it'll be showing you the, the picture of a, the beautiful
at 2 00:33:12
underside of a completed Yeah.
at 2 00:33:13
Project.
at 2 00:33:15
All right.
at 2 00:33:16
Uh, thanks folks for listening.
at 2 00:33:18
Uh, I mean, if it wasn't for you, you know, I don't know why we do this.
at 2 00:33:21
So, uh, that is a wrap.