The Disastrous Dedoose Crash: Lessons Learned

This week on the backup wrap-up we dive into the disastrous data loss

event that hit dedoose a cloud-based research data management platform.

Back in 2014.

Dedoose lost access to their primary data and their backups simultaneously

resulting in the loss of over three weeks worth of customer research.

We discussed what caused the crash as well as some questionable backup practices.

These practices resulted in a ton of data loss for doctoral researchers who

lost countless hours of painstaking work, they will never get back.

We discuss what dedoose did right, and what they could have done

better so that we can all learn.

If you're new to the pod, I'm w Curtis Preston, AKA Mr.

Backup.

And there's a reason I'm so passionate about this topic.

When I first started in backups over 30 years ago, my company lost an important

database that I couldn't restore.

Since then I've dedicated my career to making sure that would never again happen

to me or anyone who will listen to me.

We turn unappreciated backup admins into cyber recovery heroes.

This is the backup wrap-up.

Welcome to the show.

W. Curtis Preston: Welcome to the show.

I'm your host, w Curtis Preston, AKA, Mr.

Backup, and I have with me my tax non-ad advisor, Prasanna Malaiyandi.

How's it going?

I, I've needed you a lot over the last few days is

It is that time of the year, Curtis, you know,

I wouldn't expect anything else.

For, for our international listeners who don't know , april

15th IS a US' day that you have to file your yearly tax income tax

with the government, with the IRS

W. Curtis Preston: You can file an extension, but any money that you

owe has to be paid by that time.

This was, I'm gonna say one of the, if not the most complicated years that

I've ever done tax, because halfway through the year I was laid off.

And so then I went, you know, self-employment and I went like nine

different ways of self-employment.

And um, you know, and I had, I had some carryovers from previous years and I had

some, I had some of this, some of that, and I'm like, I think I'm gonna be okay.

I might not be okay.

Turned out I was way okay.

I was very okay.

But you may recall that there was a day there or so where I was not having any.

I recall that.

And, and for people who may not understand the complexities of the US

tax system, they're probably thinking, oh, don't you just like take your

income and they just take a check for based on a certain percentage.

Yeah.

That's not how the US system works.

W. Curtis Preston: Yeah.

And well, and then when you add in the complexities of, um, some of the

different types of self-employment that I did, there's some very specific

deductions that you have to take and then you have to really record stuff

to, to, to make those deductions.

And, and, uh, there was, um.

It was just,

I remember the picture you sent of just like, Hey, here

are the things I have to be tracking in order to be able to do this deduction.

I was like, oh my gosh, Curtis, there must be a better way to

W. Curtis Preston: Yeah.

I, and, and I, I will say I did figure out a better way for that one.

I've only done that, that particular one for like seven years.

And, uh, I never had figured out a better way.

And actually this year, for the first time, I figured out a better way.

So 2024 will be better for

W. Curtis Preston: 2024.

And also, uh, I, you know, uh, you helped me, uh, you know,

come up with a spreadsheet.

I mean, I did it on my own, but, you know, there was some,

definitely some advice in there.

And man, you're right, the US tax code is super complicated.

Um, you know, because, you know, we have a progressive tax system, so like

they tax this percentage on that amount and this percentage on that amount and,

and, uh, so that, so it's not just.

Figure out how much you made and take X percentage of it.

It is so not that.

And of course, I live in California and we have a state income tax,

so we have that in addition.

And just in my tax bracket from the lowest, you know, there

are, what, what did we find it?

Six, six different layers of that.

It starts at 1%, then 2%, 3%, 4%, you know, all the way up to like 9%.

I think, um, you know, once you go.

12.3.

Yeah.

Is

W. Curtis Preston: Oh, well, no.

Well, I, well, I know it goes past that, but like, but like, you know,

if you make over a certain amount, you end up paying 9% on that final, uh,

on the, on the marginal, that's what's called the marginal tax rate, uh,

So it's like every dollar you make

additional gets taxed at that rate.

And yeah.

And then the other thing challenge is you have kids.

They have different deductions, credits and deductions change

every year, depending on what gets passed, what doesn't get passed.

Your brackets don't stay fixed because some are adjusted based on the rate

of inflation, but some are based on one inflation rate, others is

based on a different inflation rate.

So it's a giant mess.

You basically need a PhD in order to be able to do this stuff and keep up with

W. Curtis Preston: Yeah.

Yeah, but through, but through it all, I have contacted my

financial non-ad advisor.

Prasanna Malaiyandi

let's just say you are not the only person

who talks to me about, uh, taxes.

W. Curtis Preston: Is that right?

I, that, that does not surprise me.

'cause you're a, you're a smart, you're a smart little person.

It's just random bits of knowledge that I have in my head.

W. Curtis Preston: yeah.

I love, I love how you can just, you can just quote the marginal tax rates for

different, for different levels of income.

Um, yeah.

Yeah.

And you're like, oh, yeah, with the self-employment tax, you know,

you could deduct half of it, right?

I'm like, no, I didn't know that.

Yeah.

And of course, I, I bet a significant portion of our listeners go, yeah,

no, that's why I have a guy, right?

I just give the money to the guy and the guy just tells me, um, you know, uh, yeah.

Um.

Prasanna Malaiyandi: useful to learn, right?

Because then you could figure out the impact of certain things that

you're gonna be doing or like how you should be withholding.

And honestly, like it's great to have a guy, but the guy

charges money and it's expensive

W. Curtis Preston: charge money.

Yeah.

I, I think one meeting with the guy, and by the way, it

could also be a girl, right?

Just, you know, one meeting with the guy and you, uh, you've spent

like the entire thing of just what I spent with TurboTax for the year.

And that's just one meeting, right?

Um, yeah.

So.

My guy's free, except he is not allowed to be my guy.

He's, he is my non-ad advisor, legally is not giving me financial advice.

Um, yeah, our financial advice is get a guy.

Um, anyway.

Prasanna Malaiyandi: Well, at least it's done.

W. Curtis Preston: At least.

Yeah, it is done both last year and this year.

I have filed my estimated taxes for Q1, like a week early.

I

Look at that, Curtis.

And I'm hoping that a lot of people who have complicated

taxes don't wait till the end.

W. Curtis Preston: Yeah.

Yeah.

I'm just so super happy to be done with my taxes, both for the last

year and for this, this quarter.

I'm so super happy, super excited.

Uh, but we're gonna talk about something not so exciting.

We have yet another episode of, you know, cloud disasters and this one.

This

I don't know about you.

W. Curtis Preston: what, go ahead.

I don't know about you, but I've kind of enjoyed this series.

I know it's sort of taking joy in other people's misery, but

I've found it to be kind of, I.

Like interesting, right?

To kind of look and figure out, hey, what did people do?

What happened?

Because these are things that could happen if you do something wrong or

misconfigure something, or maybe it's not even your fault and you just ended up

getting attacked by a malicious user and

W. Curtis Preston: Yeah, I, I.

I will say that this story, there's one particular part in the middle of it that

is the crux of what happened that is so bad backup design that we're gonna,

we're gonna get to right at the core.

At the core, they had some other mistakes that, you know, the, the same mistakes

that everybody else makes and, um, and, and sadly it appears that this was

one where they stepped on their own.

Toe or whatever, right?

Um, but this was not an attack.

This was just, uh, you know, some of the headlines, you know, use phrases

like fat fingering and stuff like that.

Uh, we are of course talking about a company who I'd never even heard of prior

to this story, and it's called Dedoose.

At least I think that's how it's pronounced because they are a, um,

uh, you know, what do they describe?

They're, they're a cloud-based, qualitative and mixed methods research,

data management and analysis application.

That's a mouthful.

I.

And they were in Manhattan Beach, California, which for the record is

right up the road from me and right down the road from our friend, uh,

Jeff Rochlin, uh, that's literally like right next door to him.

And they, they were designed to help researchers organize, analyze,

and collaborate on qualitative and mixed methods research data.

And one particular group of people that used this platform a lot were people

doing their doctoral dissertation.

I.

And so this is gonna

a bunch of research.

W. Curtis Preston: bunch of research and that, that you're just getting

ready to file to, to, to, you know, to what, what do you call, do you,

what do you do with your dissertation?

You file it, you.

You defend.

Yeah.

Um, present and defend.

Right.

Um, and you're just at that moment where you're about to do that.

And, um, the um, uh, and, and this, then this bad thing happened.

So they were, they had some competitors, uh, I'll just finish this part.

They had some competitors, Atlas Ti NVivo.

And Max does, they all have these, these interesting names, but they were

apparently the only SaaS based, um, you know, web-based, uh, portal, which made

them the only game in town for this, for doing it the way that they were doing it.

Yeah.

And my guess is people, like if you're an academic, you're probably

like, yeah, I'll just throw it into spreadsheets or throw it into Microsoft

Access and just run it locally.

Um, but that doesn't really scale.

And then you have to manage it, and then you have to worry about how

do I protect it and back it up, and what happens if my laptop crashes?

How do I sync it over to OneDrive or wherever else it has to go?

And yeah, so I could see the appeal of this company.

W. Curtis Preston: Yeah, absolutely.

You mentioned it was a cloud provider.

They are hosted on top of Azure,

W. Curtis Preston: Yes, they are hosted on Azure.

And this was one of the reasons that we wanted to cover this because we were

trying to get a story from each of the major cloud providers, and this was the

biggest story that we could find on Azure.

But I don't know if the fact that they were on Azure was

really part of the story.

It almost was as we're gonna cover it in the story.

But, um, but in the end, I think this was all them.

Prasanna Malaiyandi: Actually, I take it back.

It's Azure, not Azure.

W. Curtis Preston: Oh, it depends on who you ask.

Um, we could go on, we could go on Microsoft copilot and

ask them how to pronounce it.

Do you want to describe what, what actually happened?

Yeah, so everything was going fine, right?

They had the service up and running, and then sometime in 2014

they encountered an event, right?

And it was basically a failure of a service.

Running on top of Azure, which initially they said was a Microsoft service that

had an issue, but turns out it was just their service that they had been

operating that just leverages Microsoft component infrastructure to run on.

W. Curtis Preston: right.

And so, uh, what ended up happening is that failed and they

basically lost access to their databases, which hosted the data for the end users.

They also happened to lose access to their data itself and

the backups at the same time.

And so a lot of bad things happened all at once.

W. Curtis Preston: Yeah.

I mean, this is why we back up, right?

Right.

Because bad things happen.

Um, what I, what I don't, just don't fundamentally understand is

they go, they describe it as we were in the middle of our database

encryption and backup process.

What does that mean to you?

I read that as they're encrypting the database

before they write out a backup.

W. Curtis Preston: Right.

But

And so my,

W. Curtis Preston: place

no, I

W. Curtis Preston: as an event.

my guess is, well, it's almost like streaming, right?

So I think as they're reading data out, they're encrypting it and writing

W. Curtis Preston: Okay.

Okay.

It's just that it's just that the event damaged the database.

That's why, that's what I don't understand is why would a backup

event damage the database?

Well, it's like the old days of Oracle, right?

And I know you've been in this situation so many times, right, Curtis, where

it's like, Hey, you did a backup, you took down my production database, right?

They might have been an issue.

And I know that in all your scenarios, Curtis, that's

always been not the case, right?

W. Curtis Preston: Right.

That backup doesn't take down production, right?

Maybe they just didn't code for it properly, or maybe there

was just some race condition,

W. Curtis Preston: Yeah.

some reason while they were doing a backup.

Our production database also had a failure.

W. Curtis Preston: Right.

And I think this was good that they were encrypting the backup, but

based on what I'm hearing, what I'm inferring from this is that this was

a homegrown backup system, right?

Because they, they, they seem to.

Make a big deal about the fact that they're encrypting and, and

then, and then, you know, and then backing it up, it's like, okay,

well that's sort of standard.

You know, that's SOP for most people, right?

But, um, so they had a failure of the primary database as they were backing

it up, they were backing up and this, this was the crucial part that I

was talking about earlier, is that.

It's apparently they didn't have enough space or whatever that they

ba they're overriding the last good backup of the database as they're

backing it up and slash or they're only backing up like once a month because,

a bunch of incrementals

W. Curtis Preston: well, but if they were, but if they were doing incrementals,

they would've been able to restore to a better point in time because.

So maybe we should talk about that point in time that they

actually were able to restore four.

W. Curtis Preston: Yeah.

So after, after, yeah.

I guess we, we kind of jumped ahead, so we're doing the research afterwards.

Yeah.

So, um, yeah.

So let's back up to, you talked about the event.

So after that event, they, they went out and they did what we tell people to do.

They put out a blog post and, uh, they're like, here's the situation.

Uh, we believe we can go back to, this was May of 2014 . We know for a fact

we can, that the March backup is good.

We think we can also restore the April backup.

We unfortunately believe that we are going to be unable to restore.

The, the, the May backup.

And so every, all of the changes and anything that you put in the system for

the last two to three weeks will be gone.

So I am, I am inferring from that, that they only backed up once a

month because otherwise they would've been able to get a lot closer

and it wasn't just the database, like I was reading Reddit

posts where they're like, Hey, we talked to a service person or a support person.

They basically said if you created a new account, even in that May

timeframe, you're basically out of luck.

Right.

So it

W. Curtis Preston: Like the entire application, right?

So it's possible that they weren't overriding the last good backup.

It's just that the last good backup was from over a month ago.

Or it could be that like their more

recent backups were corrupt.

W. Curtis Preston: Uh, well, we don't have any information to suggest that,

don't have any information.

Yeah.

But

W. Curtis Preston: I I also don't understand why if it, if all backups

were equal, why didn't they immediately try restoring the one from April?

They, they, they said, we know for a fact we can get the one from March.

We think we're gonna get to one.

And they go and, and, and they made a comment about, you know, we realize

that this is, it reminds me a little bit of the Rackspace event where.

What we can do, the quickest is we can get the March one up and running.

Why wasn't the April one also just as good as the March one?

Right.

So I'm not, I'm not sure what happened there, but there was something that

was not as good about the April backup.

Uh, but, but it did appear that they eventually were able to get

that up, but they lost two to three weeks worth of, of effort.

Yeah.

Yeah.

And even for the April backup, they didn't actually restore it.

So using the way back machine right, we were able to go back and

find the original blog post for

W. Curtis Preston: Right.

Why did we have do that Prasanna?

Why did we have to use the, the, the internet archive?

because the company took down that post and created a

new post at that same, for that same date, that shed different light on

what had happened and their steps.

W. Curtis Preston: Interesting, right?

Um,

so, so from that or may, uh, blog post, they basically

said, or soon thereafter, right, I think it was a couple weeks later, they

were like, Hey, we're able to get you back up and running for your March and

for April, we were able to restore the data, but they didn't wanna merge the

data back into the original database.

So they actually created a staging site where people had to go to manually

in order to be able to pull down their data if they want to access it.

W. Curtis Preston: this is so much like the Rackspace event, right?

And so what that, what that tells me, you know, and again, we're, we're, we're not

taking joy in this, but what that tells me is that they didn't have a regular

tested procedure because they didn't, they didn't know that the April one was good.

If they knew the April one was good, they would've just restored the April

one, because by restoring the March one.

They may, you know, you couldn't just do an incremental restore

on top of the March database.

So, um, they created this extra work for

themselves and their

and especially their entire thing was we want to get people

up and running as quickly as possible.

Right.

That's why they did the March one.

But like you said, Curtis, if April was good, they should have just used April and

it would've been the same amount of time.

W. Curtis Preston: There had to be something up with the April one.

You know, you could, you'd sort of have to read between the lines.

Um, there was something up with the April one and then,

we've talked about this before.

We urge people not to do homegrown backup and when I said this last time,

you said sometimes you have no choice.

Yeah,

W. Curtis Preston: I would like to, I would like to.

say it again.

W. Curtis Preston: Well, I'm, say it again, but I would like to point out the

fact that they were running on Azure, and Azure has backup features built into it.

Right?

They could have taken a snapshot of their entire environment every day, every hour.

They could have done that.

They would've had essentially, you know, you know, a copy of

their database from two hours ago.

You know, they could have, and that wouldn't have been homegrown.

I'm just saying they could have done that.

Well.

The other thing they could have done is why don't they

have a disaster recovery copy?

Like if this is a SaaS service, wouldn't you have expected a Dr.

Copy to be available somewhere where even if the production instance went down for

some reason, they should have been able to come back up even if it wasn't a backup.

W. Curtis Preston: Yeah, so this, this is where, you know, we often talk about

that, that if your data is sitting on a SaaS service, I think it's your

responsibility to back it up, or at least to make sure that it's backed up.

You can, you can make an argument as to whether or not you should use

their backup service if they have one.

I think we've got three stories now that show that sometimes their backup

service is as good as their it, um, and.

The, so you, I think you should be backing up your own data, but

generally speaking, some of these SaaS services, they will have a dr.

Copy that isn't any good for you if you do something stupid, but, but if

they do something stupid or there's a fire, or there's a, an attack,

they've got a copy that they can restore the entire environment that

that appears to be what they had here, but it was just woefully out of date.

Um, and so the vast majority of their customers did get their data back, but

there were a few, and I wanna, I want, I want to just point out a few that

they were, uh, quoted in the story.

There was, uh, Margaret Fry, a postdoctoral researcher at Harvard

University, lost about 60 annotated texts and over a hundred hours worth of coding

work related to her research on AIDS and sexual attraction Southern Africa.

Jason Richardson, an associate professor at the University of Kentucky and his

colleagues lost around a hundred hours of work on a project, analyzing job

advertisements for school principles to assess how well Kentucky prepares

aspiring school administrators.

Uh, there's one comment on Dedoose Facebook page that the company had managed

to destroy their wife's $10,000 research project, so there were real people.

And these weren't just companies, these were just, these were just regular

people trying to get their doctorates done and uh, they lost, you know,

yeah, but, but to their credit though, dedoose actually

had no, I don't know if it was before this issue or after the event, but

they did say they have functionality.

Actually, no, it was even before because for staging, this is

how you get your data out.

There is an export functionality.

W. Curtis Preston: Right.

That a user could use to pull all their data

out of Dedoose for doing, like what you talked about, a local backup.

So it is possible that these users could have been using that to protect

themselves from these situations, and Dedoose is still around, and so I highly

recommend people who are using Dedoose.

Maybe you should periodically export your own data out.

W. Curtis Preston: Can I, can I edit you and take the word maybe out there?

Yes.

W. Curtis Preston: Yeah, so you should periodically back up your work no

matter where it's residing, you know, a, a periodic, you know, inefficient

backup that has a poor recovery point.

Actual is still better than, than nothing.

Right.

Um, yeah.

I remember, I, I have a, a niece that came to me with her, she had a laptop

and she had her masters on this laptop.

And the laptop, the drive was making those noises.

You know

Oh, the click, click, click, click,

W. Curtis Preston: Yeah.

The hard drive should not make.

We were able to get off her master's thesis, uh, off of that

drive before it just went up in smoke and it was killing me.

'cause I was like, you don't have a backup of her.

That's really important work.

Uh, you know, I feel for these people At the same time, you know, uh, my,

my statement is gonna remain the same.

That if your data is sitting in, in a SaaS service somewhere, it's

your responsibility to, to make sure that it's being properly backed up.

Yeah.

W. Curtis Preston: and and I still say even if the SaaS vendor offers

backup, which this one apparently did, um, I still think that you

should do it some other way.

Um, yeah.

Well, do we know if they offered backup or

was this just sort of their own

W. Curtis Preston: right.

It this was their, this was their internal dr.

Right.

Um, what I don't know is whether or not they advertise this as part of the, um.

You know, now the CEO of Dedoose said this is a horrible moment for our company.

We have never lost data or had a breach.

It's impossible to say how many customers have lost data because

of the firm doesn't monitor how customers are using their accounts.

Only users can access, can assess losses in their projects, not Dedoose.

Yeah, it was a, it was a

Yeah, if I

W. Curtis Preston: a.

Oh, oh yeah.

Like, and especially researchers, like coming back and getting

that data is so difficult.

And then just being like, yep, it's gone.

Just like that puff of

W. Curtis Preston: And, and some of that data is irreplaceable.

Yeah.

W. Curtis Preston: It's interviews of subjects.

Um, you know, I don't even want to try to, when you're in the middle of

something that complicated, there is some data that is simply irreplaceable.

Um, and, um, so it, you know, it's, it's a real shame what happened.

Did you wanna talk about what they said they're going

to do and what they've actually done?

W. Curtis Preston: yeah.

That's a great, yeah, that's a great segue.

So they said they're now going to be doing seven things and when, when I

say now we, we mean, uh, 10 years ago.

Right.

Um, after the event they said they were going to, it sounds like they

went from not much to way overboard.

Right.

Of course, I would never say that any backup system is overboard.

But here's what they said they were gonna do.

They were gonna develop a database mirror, slave, and Azure.

A database mirror slave in Amazon S3, keeping a mirror copy of the entire

blob storage, you know, including all the stuff in an encrypted volume

storing nightly database backups on the V-H-D-V-H-D, virtual hard drive, I guess,

and Azure blob storage and Amazon three.

So they're gonna keep three nightly backups in three separate places mirroring

all Azure file data into an S3 bucket.

Carrying out weekly restore exercise, uh, for the database backups and

a monthly bare bones restore.

That all sounds great.

Now I do, I do wanna comment, um, that anything that refers to mirroring to me.

Isn't a backup, right?

Um, that is a dr.

Copy that will only be useful if what happens is like a fire, right?

If there's any sort of attack, any of those mirrors are totally worthless

because, or if, if you do something stupid like drop a table, all those

mirrors are gonna immediately, uh, you know, get corrupted as

but it would've helped in their current,

in the event that it happened.

W. Curtis Preston: Uh, I'm gonna say it's unclear as to whether

or not it would help in it,

Okay.

Sorry.

It could help.

Yeah.

W. Curtis Preston: It could have helped.

Yes, yes.

Um, mirroring is not bad just because they're putting it here and

they're listing all these things.

Um, you know, it's a shame that they, that they had to wait.

Uh, after the thing.

What I also would've liked to seen in that list is I would've liked

to have seen that one or more of these copies, uh, was immutable.

Right?

Um, and, uh, and maybe they are, I don't know, you know?

It's a great list, but it's also one of those

things like once a horse has left the barn, like what do you do?

You know, it's sort of like, Hey, uh, we're just gonna throw

everything at this and just say, this is everything we're going to do.

W. Curtis Preston: yeah, it

than what

W. Curtis Preston: yeah.

what is actually makes sense from a business perspective,

because sure, you could do all these things, but just imagine the cost

you would be spending for backup.

Right?

Your backup copies are probably more than your production

copies at that point, right?

W. Curtis Preston: Yeah.

It depends on, depends on the way they're doing them.

Yeah.

Yeah.

Yeah.

Right.

W. Curtis Preston: Yeah.

To take, to take your, to take your analogy, you know, the horse is laying

out dead in the field, and you're like, you know what we're gonna do?

We're gonna lock the, we're gonna lock the barn door now, and then we're

gonna put another padlock on top of that lock, and then we're gonna put

a fence around the area in front of the barn, just in case the lock fails.

And then, uh, you know, and then we're gonna have a guy

standing, standing there with a.

With a tran gun, uh, to shoot the horse.

If the horse comes outta the bar, it's like, yeah, that's all great, but your

horse is dead, so you need another horse.

Yeah,

W. Curtis Preston: we're some, I'm, I'm sorry that was a little

morbid, but I apologize to the horse lovers in the, in the group.

but the other thing we found though, so that's what they

had promised 10 years ago, but they do have in their, what is this called in

their security article on Dedoose, sort of what they've actually done, right?

So.

What they have done is they are using redundant storage volumes on Azure.

W. Curtis Preston: mm-Hmm.

All project data backed up in full on, on a nightly basis.

A volume is onsite with two being offsite and replicated across geographic regions.

Um, and then the storage volume is encrypted and mirrored in

real time to Amazon S3 storage.

They've automated everything such that on a weekly basis, it'll download the most

recent backup from each storage volume.

Verify they're using the correct version of the backup file, do a full

test restoration of the database, and email reporting to all backup

and restore process results to key members of the Dedoose team.

W. Curtis Preston: Yeah, it's, it's still, to me, it still reads

like a homegrown system, but, um.

You know, because like one of the things like, you know, I, I did, I did a lot of

homegrown systems over the years, right?

Like one of the things like with email reporting is.

D does your system report if it doesn't run right, is is there,

is there a thing that, that's, you know, like what happened?

You know, 'cause sometimes you don't notice that you're not getting an email.

Right?

Hey, has anybody noticed that?

We don't get the, we stopped getting the backups, you know,

uh, like three weeks ago.

Um, usually when you, when you discover that is right after the, um, the

bad thing happens, whatever, it's

but even though it might be sort of homegrown from a

automation process perspective, like we don't have enough information.

I think the fact that there are at least.

Doing nightly backups, right?

Verifying it on a weekly basis like these are, and also making sure that

the data is stored offsite, right?

I think maybe it's going a bit for like, they may not have to do like multi-cloud.

Maybe that's going a stretch too far, but I could see that they really are

worried about data integrity, so,

W. Curtis Preston: There's no such thing as a stretch too

far for backup and recovery.

My friend.

well,

W. Curtis Preston: They're doing the thing that I wished everybody did, but

nobody does because it's too expensive,

Yes.

W. Curtis Preston: right?

Um, so I, I can't fault 'em for doing that,

Yeah.

So it looks like they are doing the right things,

W. Curtis Preston: mm-hmm.

um, from what we can gather.

So it looks like they have learned and they're still around, right?

People still continue to use 'em.

They do have an active subreddit.

They are continuing to release new features.

W. Curtis Preston: Mm-Hmm.

Yeah, and I think, I think what was in their favor is that they

were the only SaaS game in town.

And for a lot of people that, that, that's probably a significant.

Reason to use their service.

Right.

Um, I would also hope that this is a significant reason for anyone to do a,

a regular export of their data, right.

When I'm using a service like that, where I tend to do is I do, I do an export.

Like if I'm doing a manual export, I'm doing it after significant events

like finishing my taxes, right?

I did, you know, I did an export.

I, I did a PD, I didn't do exports of.

My intuit data as I was doing it right.

But once I finished my return, I printed a PDF of that.

Um, I have a paper copy.

I have a PDF copy that's on my laptop.

It's backed up using my backup software.

And uh, I put it in, um, Google Docs, right?

So it's all over the place, right?

Um, but I don't do that every day.

It's just you do it when you, when you get to that moment where you're

like, ah, oh, this is really good.

I hope I don't lose

Prasanna Malaiyandi: a good stopping point.

W. Curtis Preston: It's a good stopping point.

Yeah.

Um, it's like also I use voice recognition software, right.

And sometimes it crashes and you lose whatever you said, uh, in that thing.

And so I just got into the habit of anytime I finish, like

a page, I just hit, you know?

I just say, um, yeah.

Uh, 'cause I, I can actually say.

Um, I almost said, Hey, Siri, you know, click file save.

That's what I say.

Click file, save.

And, um, and I can do that or you can just type control s right?

Uh, just do you know, this is, I guess this is another lesson to learn

is, is that should be a part of your mental process when you do stuff and

it's storing that data in some place.

At the end, at that stopping moment, do you, do you think to yourself, Hey,

is this data backed up in a way that protects me from all stupid things?

You know

W. Curtis Preston: are you

funny when you're, when you're talking

about the control save, right?

Or Control S to save it.

So I'm so used to working on Excel spreadsheets that I would always do

like Control S, control S, to save

W. Curtis Preston: Right, right,

Even when I use Google shoots, I still do it anyway,

W. Curtis Preston: right.

just out of the force of habit.

W. Curtis Preston: Yeah.

That's funny.

, Prasanna Malaiyandi: So, lessons learned.

Lessons

W. Curtis Preston: Yeah.

Yeah.

So we, you know, so obviously we talk about, you know, it's still your

responsibility to backup up SaaS.

The other thing is a month RPO?

A month backup frequency.

That's, that's unacceptable.

That's just, I can't imagine that being acceptable, uh, to

in, in any world, by the way.

That is determined by your business, right?

I, I have met at least one company where a month long RPO was fine.

They were a paper mill, right?

They were, they were like, we don't, the, the, the, the computers, they just, they

don't have anything that we care about.

Right?

They were fine with a month.

That should be determined by your business.

But a month seems really long to me.

Um, and the other thing is, if your backup system.

Is this is, we don't know if this is actually the case,

but I'm inferring from it.

If your backup system overwrites the last good backup as part of

the process because you're so starved for storage, that is bad.

That is bad.

Bad, bad, bad, bad.

That is basic

storage is cheap.

W. Curtis Preston: Bad.

Well, it's cheap, but I'm just saying it's cheap.

Right?

So much.

So much we do or don't do is because it's, it's inexpensive.

Right?

Well, given that they are keeping all these

copies and dealing with cross site replication and everything else,

W. Curtis Preston: Mm-Hmm.

Well, they, yeah, they're not doing that now.

Right.

I'm just saying there are people, there are people that their backup system

is overriding the last good backup, and I just, that, that is a, that is

a really, basically, it means that if you're in the middle of the backup and

something happens, you have no backup.

Yep.

W. Curtis Preston: Unless you're backing up the backup.

If you're backing up the backup right, then, then, then, right.

But if this is your only backup and you're overriding it with the next

good backup because you're too cheap to buy enough storage for multiple

backups, um, I dunno what to tell you.

Yeah.

Please do not please step away from the keyboard and have someone

else design your backup system.

W. Curtis Preston: step.

That

is true though.

W. Curtis Preston: perfect.

It is true.

Yeah.

Yeah.

Yeah.

W. Curtis Preston: Yeah.

Uh, can you think of any other, um, lessons from this one?

I know that we talked about how Dedoose may not have

had the best backup systems in place,

but this is also one of those things like it was.

Unfortunate that they had these multiple issues that led to such a big failure.

But it is good that they did see, okay, here are gaps in our system.

Here's how we're gonna fix it.

They were transparent with the users and it looks like they've gotten

their act together, which is good.

Right?

So I think it's just a matter of making sure they're continuing to stay up to

date and reevaluating their architecture and infrastructure of their backup.

Especially like, I don't know how they deal with ransomware.

W. Curtis Preston: Right, right,

Like are they really using separate accounts,

like when they're replicating to different buckets, different regions,

or is it all like the same username, password, or whatever across the two?

Right.

W. Curtis Preston: You're killing me

Right.

Because hopefully they're keeping their Azure or their AWS system isolated.

W. Curtis Preston: Right, right.

And hopefully again,

than backups.

Yep.

W. Curtis Preston: somewhere, along the way they're using immutable storage.

Right.

And we, and we do like the way that they, that they had the blog.

They were, they were immediately, um, you know, transparent with what was going on.

They were, they were a little bit of, uh, you know, mea culpa, right?

They're like, Hey, this is our fault.

You know, this is all us, you know, uh, we're so, so sorry.

Uh, I'm a little disappointed that they deleted the blog after the fact.

Um,

Put a link to the.

W. Curtis Preston: referenced in mo.

What's that?

Yeah, we will put a log to a, a link to the archive version of the blog.

'cause the internet never forgets.

Um, yeah.

So.

All right, well thanks for helping me work our way through

another cloud disaster Prasanna.

No, thank you Curtis.

This was fun.

W. Curtis Preston: Yeah.

Fun, fun, fun and a sad way that unfortunately involved

a, you know, a deceased

horse.

but.

W. Curtis Preston: Um,

Uh, poor horsey.

W. Curtis Preston: yeah.

Um, well subscribe folks so that you don't miss all of this amazingness.

Uh, and, uh, that is a wrap.