Human error: Why We Backup

Human error has replaced hardware failures as the primary driver of data loss and restore operations in modern IT environments. This episode explores real-world examples of how both end users and administrators create the need for backup and recovery operations through accidental deletions, configuration mistakes, and poor processes. W. Curtis Preston shares war stories from his decades in the industry, including incidents involving accidental directory deletions, source code stored in temporary folders, and tape library disasters. The discussion covers how technology improvements like RAID and solid-state drives have made hardware more reliable, shifting the focus to human-related incidents. The hosts also examine insider threats and the importance of implementing proper controls around privileged access. Learn why understanding human error patterns is critical for designing effective backup and recovery strategies that account for the reality of how data actually gets lost.
You found the backup wrap up your go-to podcast for all things
Speaker:backup recovery and cyber recovery.
Speaker:In this episode, we look at the reality of modern data centers.
Speaker:Human error has become the number two reason that we do restores today.
Speaker:Number one, of course, would be cyber attacks.
Speaker:Back in the day, you know, a hundred years ago when I first started, it was
Speaker:really all about hardware failures, hard drives, crashing, servers dying.
Speaker:But now it's people making mistakes.
Speaker:Sometimes it's a user accidentally deleting a file, or maybe it's an admin
Speaker:wiping out an entire directory structure.
Speaker:Humans are the weakest link.
Speaker:I've got, I don't know, four or five war stories that'll make you cringe.
Speaker:And we talk about insider threats as well, and why your backup systems
Speaker:need to be designed with human error and human frailty in mind.
Speaker:Trust me, these stories will remind you why we do what we do.
Speaker:By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.
Speaker:Backup, and I've been passionate about backup and recovery ever
Speaker:since I had to explain to my boss.
Speaker:While there were no backups of the production database that
Speaker:we had just lost, I don't want that to ever again happen to me.
Speaker:I don't want it to happen to you.
Speaker:That's why I do this.
Speaker:On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.
Speaker:This is the backup wrap up.
Speaker:Welcome to the show.
Speaker:Hi, I am w Curtis Preston, AKA, Mr. Backup, and with me, I once again have
Speaker:my security deposit consultant persona.
Speaker:Molly, Andi, how's it going?
Speaker:Persona,
Speaker:I am good, Curtis, although I thought you would've said you're a person who
Speaker:was successful at DIYing something for probably the first time in their life.
Speaker:a guy that, a guy that finally joined the ranks because, because you're
Speaker:completed now with your, with your.
Speaker:I am.
Speaker:If people
Speaker:Yeah,
Speaker:episode, I, uh, had to redo some.
Speaker:stuff for
Speaker:yeah,
Speaker:system, and it was a little painful.
Speaker:yeah,
Speaker:lie.
Speaker:Um, and especially because I had to replace a shutoff valve, which meant I had
Speaker:to shut off the water for the entire house
Speaker:yeah.
Speaker:I was repair, replacing that shutoff valve.
Speaker:And then it turned out that I was leaking and blah, blah, blah, blah, blah.
Speaker:So, eventually got around to it, turned out, um, make sure you wrap the
Speaker:Teflon tape in the right direction.
Speaker:Yes, yes, yes.
Speaker:You figured that out, didn't you?
Speaker:Yes.
Speaker:And also having the right tools helps, especially a pipe wrench.
Speaker:Yes.
Speaker:More than one, right?
Speaker:Yeah.
Speaker:So once I did that, then actually doing all the PVC gluing and
Speaker:That part was easy, right?
Speaker:So I remember purple stuff first, then the blue stuff.
Speaker:Yeah, yeah,
Speaker:I should say.
Speaker:yeah.
Speaker:Oh, you use clear stuff?
Speaker:Yeah.
Speaker:Oh, that explains why I saw the blue.
Speaker:I saw the purple.
Speaker:Yeah.
Speaker:Okay.
Speaker:But it worked and it held, and it's knock on wood.
Speaker:So far so good.
Speaker:So
Speaker:Well, I'm glad to hear that you can now sprinkle your yard once again.
Speaker:I of course don't have that challenge 'cause I have a fake yard.
Speaker:well it's not a yard, it's just for the plants and all the drip irrigation, so.
Speaker:Oh, gotcha, gotcha.
Speaker:It's a yard, it's just, it's not grass.
Speaker:saw grass.
Speaker:Yeah.
Speaker:Right, right.
Speaker:especially last weekend.
Speaker:No, over the weekend, I think the temperature at 87 here in the Bay
Speaker:Area and today it's 61, so yeah.
Speaker:Yeah, we actually, we had a heat wave here as well.
Speaker:Um, it actually got up to like 90 at one point,
Speaker:Dang.
Speaker:Today it was cold and rainy, so
Speaker:Yeah.
Speaker:I think may.
Speaker:Gray back, no Uns shining outside.
Speaker:Hmm.
Speaker:I don't know.
Speaker:Who knows?
Speaker:You never know in San Diego, right?
Speaker:Especially in May.
Speaker:Right.
Speaker:May Gray and June gloom.
Speaker:For those that don't live here, it's pretty much overcast all the
Speaker:time for two months of the year.
Speaker:But it's why the temperature is,
Speaker:Even Keter.
Speaker:Yeah.
Speaker:Yeah.
Speaker:Um,
Speaker:dude.
Speaker:come here.
Speaker:What?
Speaker:ahead, finish.
Speaker:Well, I just said people come here and they, um, they're like, why?
Speaker:Where?
Speaker:I'm like, this is like, do you do no research on the places you go to?
Speaker:Right,
Speaker:Well, it's like the time that, or I think that twice that I've been up to
Speaker:Seattle and both times it was nice and sunny and so in my mind it's imprinted
Speaker:that Seattle is always nice and sunny.
Speaker:Yeah.
Speaker:Oh yeah.
Speaker:I, yeah, I spent, I spent four months in Seattle in 1998, and
Speaker:it was the most perfect four months in the history of Seattle.
Speaker:It rained like two days during those four months, and it was just, and
Speaker:yeah, I could, I could have moved there, but it's not normally like that.
Speaker:exactly.
Speaker:Yeah.
Speaker:But, um, well, uh, today, speaking of.
Speaker:Humans making mistakes today.
Speaker:I thought we'd talk a little bit about human error.
Speaker:Um, and, uh, as, as I often, uh, you know, find myself doing, I'll talk about
Speaker:back in the day, back in the day, um.
Speaker:The reason when I started my IT career, the, the number one reason
Speaker:we were doing restores was, um, actual hardware failure, right?
Speaker:Because I am Prera, right?
Speaker:Um, we, we had mission critical servers running on individual hard drives,
Speaker:Which is bonkers.
Speaker:A
Speaker:right?
Speaker:Right.
Speaker:do you think, what is, what is it like this, isn't there like a satellite
Speaker:that's been out there for like 40 years
Speaker:Oh yeah, yeah,
Speaker:and do you think that still also runs on a single disk?
Speaker:I,
Speaker:yeah.
Speaker:Def absolutely.
Speaker:Yeah.
Speaker:crazy.
Speaker:Yeah.
Speaker:Well, it might, do they have, do they have disc drives?
Speaker:I don't know.
Speaker:No.
Speaker:I mean, it's, if it's been there that long.
Speaker:Uh, but yeah, it, it's, I mean, I'm pretty sure RAID was invented before I actually
Speaker:saw it in the data center, but in my data center, which was a, a big, you know,
Speaker:$35 billion company, we didn't have any.
Speaker:Right.
Speaker:And, um,
Speaker:Was it
Speaker:and go ahead.
Speaker:was it the difference though between raid on open systems versus raid on mainframes?
Speaker:Yeah, I don't, yeah, I don't know.
Speaker:Um, it just, it certainly wasn't, you know, I remember the first
Speaker:time and I talked about it a few episodes the first time I saw
Speaker:hot swappable disc drives, right?
Speaker:Where you, you had the raid arrays, right.
Speaker:Um, and all of that sort of, it sort of all went down right around
Speaker:that timeframe, but for a few years.
Speaker:We were, we were losing entire databases or servers because an
Speaker:individual R drive went down.
Speaker:And so we, we were restoring that stuff all the time.
Speaker:And in fact, um, my first, once I left the bank and I went into
Speaker:consulting, um, one of the reasons I got so good at Bare Metal Recovery
Speaker:was that I was doing it all the time.
Speaker:Um, because I was at a. Uh, the headquarters of a
Speaker:large oil and gas company.
Speaker:And, um, they had all these servers that had not been, uh,
Speaker:maintained in a really long time.
Speaker:And so we were doing crazy things like installing patches and rebooting them
Speaker:with significant portion of the time.
Speaker:They just didn't come back up.
Speaker:So,
Speaker:makes perfect though, right Curtis?
Speaker:yeah.
Speaker:And so, yeah.
Speaker:Um, and so my, my, my point is that back then.
Speaker:A significant portion of the time, the reason you were doing a restore, it
Speaker:was because the actual hardware failed, failed, and that is just pretty much.
Speaker:Gone.
Speaker:I, you know, uh, between the fact that we now use solid state a as hard
Speaker:drives, and the fact that we use that, that no one, no one does anything
Speaker:important on, on an individual.
Speaker:I mean, your laptop, your phone, these are individual drives, but not
Speaker:nothing in a, in a server, right?
Speaker:Um, and.
Speaker:and also probably just reliability of components has also gone up.
Speaker:Absolutely.
Speaker:The result of all of that is that at this point the, there are,
Speaker:generally speaking only two reasons.
Speaker:We restore things and both of them had to do with people, right?
Speaker:Either people purposefully did something to damage the systems or they accidentally
Speaker:did something to damage the systems.
Speaker:Are you with me?
Speaker:Yep.
Speaker:Yeah, so
Speaker:failure.
Speaker:it is not a hardware failure, right?
Speaker:So either bad people, you know, bad actors, you know, it's a si,
Speaker:it's a, it's a ransomware attack, it's a cyber attack of some sort.
Speaker:Uh, or stupid users
Speaker:Would you consider like the OVH
Speaker:or stupid admins.
Speaker:So, so that is still, that is still a human error in my opinion.
Speaker:I mean, okay, so what, we can add that as a third category, which
Speaker:is natural disasters, right?
Speaker:Um, so if, if a fire happens, you know, um, yeah, I, we'll, we'll put a fire in
Speaker:a natural disaster, although that fire, I'm not sure was a natural disaster.
Speaker:But yeah, so that, that, that is a third reason, you know, I, I'll take that.
Speaker:But, but, but even that, the reason they end up having to do a restore,
Speaker:I think, was human error, right?
Speaker:Meaning that they, they just configured everything all wrong.
Speaker:Um, and I, and that's my story, and I'm sticking to it.
Speaker:so let's talk about the accidental humans
Speaker:Yeah.
Speaker:Yeah.
Speaker:'cause we talk a lot about the, the, the bad actors.
Speaker:Yeah.
Speaker:Um, so let's talk about the, you know, and, and basically what this
Speaker:kind of boils down to is, uh, either stupid users or stupid admins.
Speaker:Right?
Speaker:I, so there are two stories I want you to tell,
Speaker:Okay.
Speaker:The first one is, uh, restoring a document.
Speaker:The, the document called resume,
Speaker:Yeah.
Speaker:Yeah.
Speaker:And
Speaker:Uh.
Speaker:second one I want you to talk about is the file server with the user home directories
Speaker:Lord.
Speaker:Yeah, yeah.
Speaker:Sorry, sorry.
Speaker:Uh, sorry.
Speaker:Ganda.
Speaker:We're gonna call you stupid admin.
Speaker:Oh, okay.
Speaker:Um, yeah.
Speaker:So that, that was, you know, back in the day when, when, you
Speaker:know, I got, we, we, you could.
Speaker:Hmm.
Speaker:We, we had 12,000 employees and we had a help desk, and you could call in.
Speaker:This was, you know, pre-web, pre, you know, everything.
Speaker:You, you, you lost a file.
Speaker:You called into the help desk, and the help desk would maybe try to
Speaker:help you find the file, but if not, they would issue a restore request.
Speaker:We got a lot of restore requests, very regular.
Speaker:Another reason why I got good at doing this, because unlike a lot of places, we
Speaker:did new backups just to do them, we did restores on a very regular basis, right.
Speaker:There was this person who called and, um, they, they, they asked to have a file
Speaker:restored and it was called Resume Doc.
Speaker:And we were like, really?
Speaker:Like that's the what happened to Resume doc?
Speaker:Well, I was updating it.
Speaker:Where are you?
Speaker:Is that right?
Speaker:Yes.
Speaker:I was updating my re resume dot doc and um, I, I, you know,
Speaker:I fat fingered it basically.
Speaker:And, uh, and I, you know,
Speaker:to resume?
Speaker:yeah.
Speaker:What were you resuming?
Speaker:Yeah, trying to resume your career.
Speaker:That might've been, you know, depending on who we told that might've
Speaker:been a resume producing event.
Speaker:Uh, uh, yeah, so that was, that one was kind of funny.
Speaker:But, but I think that the, the, the, the, the story that really drives
Speaker:home the, you know, because, you know, admins are not perfect, right?
Speaker:And you, um.
Speaker:You, you know, you do things, you do things to, to make things better,
Speaker:but sometimes you do the wrong thing.
Speaker:Right?
Speaker:I mean, I think of like the KPMG.
Speaker:The K-P-M-G-I think is like the, that's like the, the best one that I had nothing
Speaker:to do with, where they're just trying to delete one guy's chats and, and they
Speaker:were, they were being stopped by Microsoft Retention policies and so they made a new
Speaker:retention policy that had no retention and then they thought they moved the
Speaker:one guy into it, but instead they moved.
Speaker:The entire company into it, and they deleted it was like 150,000 people's
Speaker:chats, uh, like, you know, like that.
Speaker:And, and, um, and there was no backup.
Speaker:Right?
Speaker:But, uh, but yeah, this other, this other story, uh, is, is a good one.
Speaker:And you know, Gonda, we love you.
Speaker:Um, so she, the, we had a file server.
Speaker:I remember it was HP FS oh one, HP file server.
Speaker:You know, one, and we had a lot of employees and we had a lot of turnover.
Speaker:And over time, the, and, and this is, this is, you know, old school Unix.
Speaker:You had home one, you know Curtis.
Speaker:Right?
Speaker:And after the, the person left the company, we did not have an
Speaker:off-boarding process would be the, uh, or if we had one, it didn't
Speaker:include deleting the old data.
Speaker:And so somebody said, we think we have a lot of.
Speaker:Of directories out there.
Speaker:And, and we knew that we had all of this on tape and then we knew that we
Speaker:kept all of our tapes for seven years.
Speaker:And so we knew that if, if anybody ever actually needed
Speaker:this data, we could get it back.
Speaker:And so they hired, uh, my friend to, uh, to programmatically work their way
Speaker:through the home directory and then delete any directories that were not.
Speaker:From a valid user.
Speaker:Okay.
Speaker:And so the directories were named after the user.
Speaker:And so you would see, you know, the c Preston directory.
Speaker:And then she would look and, you know, she would just traverse the tree and she
Speaker:would find the, the directory name, and then she would look to see if there was a
Speaker:user by that name and the password file.
Speaker:And then if the user wasn't there, she would delete it.
Speaker:And, um, the um, but.
Speaker:There was, she made one small, one small calculation or miscalculation,
Speaker:and that is that there was a two level directory structure, so it was, it was
Speaker:like Home one slash a slash Alfred,
Speaker:Mm-hmm.
Speaker:know, home one slash c slash c Preston.
Speaker:And um, so she worked her way down the tree.
Speaker:She got the home one A.
Speaker:Is there a username, a. In the, in the tree, in the password file.
Speaker:Nope.
Speaker:All right.
Speaker:Delete a
Speaker:Yeah.
Speaker:Oh, no.
Speaker:And she, and she was like a third of the way through, uh, the directories
Speaker:when people started, they, they, they were unable to log in 'cause
Speaker:they had no home directory.
Speaker:And, uh, we got a whole bunch of calls to the thing.
Speaker:And luckily she, she stopped it out.
Speaker:And I, I still remember she called me that day and her, her first words
Speaker:were, she's like, Curtis, um, how were the backups of HP FSO one last night?
Speaker:And I'm like.
Speaker:Um, they were fine.
Speaker:Why are you asking?
Speaker:And she, uh, we proceed to restore all the home directories.
Speaker:Um,
Speaker:yeah, that's, uh, that's a scary thing because imagine if
Speaker:you didn't have those backups,
Speaker:yeah,
Speaker:right?
Speaker:That's a third of the company's data.
Speaker:AOG or users on that
Speaker:yeah, yeah.
Speaker:Um, I,
Speaker:And your data too, because
Speaker:what's that?
Speaker:And your data too, because you're in the cs.
Speaker:That's right.
Speaker:Yes.
Speaker:I was gone.
Speaker:Um, uh, the, uh, you, you probably would've been safe, you know, with
Speaker:the, with the being and the peas there.
Speaker:Um, there's another, another great, uh, sort of, um, uh, do
Speaker:you have, do you have a story?
Speaker:want you to tell another story,
Speaker:Oh, okay.
Speaker:Okay.
Speaker:So there's another great, um.
Speaker:Story.
Speaker:It, it, it actually involves the deletion of backup data.
Speaker:But it, it, it drives home this issue of basically users
Speaker:make or admins make mistakes.
Speaker:And this has to do, this has to do with your, your former employer, uh, legato.
Speaker:Um, and they had, they introduced a feature.
Speaker:Uh, two features actually in, in, it seemed like it was version five maybe.
Speaker:And one of the problems was when you were, when, when you, when you
Speaker:made, when you expired tapes and you were gonna reuse those tapes,
Speaker:you would relabel those tapes.
Speaker:And by that I mean you would put an electronic label on the front of the
Speaker:tape, and by doing that, you would, you effectively erase the rest of the tape
Speaker:because it puts an end of data marker at the end of the, the, you know, the, um.
Speaker:The label and then you, you can't get past that with a tape drive.
Speaker:And so the um.
Speaker:But the problem was even if you had like several tape drives, um, the process,
Speaker:it would label the tapes one at a time.
Speaker:So if you had a whole bunch of tapes, it, it took a long time.
Speaker:And then also when you would, uh, label it, it would, uh,
Speaker:confirm the deletion each time.
Speaker:Okay.
Speaker:you sure you wanna relabel the state?
Speaker:Yeah.
Speaker:And so they introduced the fast and silent.
Speaker:Uh, option where they would use every available tape drive in the
Speaker:library and they would not ask.
Speaker:Okay?
Speaker:So if you check those two boxes, then, um, basically you would, um, you know, you
Speaker:would relabel whatever tapes you selected.
Speaker:But this was accompanied by a bug.
Speaker:You no.
Speaker:And the bug was that if you, if, well, if you had a list of tapes.
Speaker:In a, in a tape library, and you double clicked on one of those tapes, you would
Speaker:expect that what would pop up would be a dialogue that would include just
Speaker:that tape that you just clicked on.
Speaker:That's not how it worked.
Speaker:If you clicked on the tape, what would pull up was slot one to slot,
Speaker:however many slots there were.
Speaker:Oh no.
Speaker:Right.
Speaker:So if you wanted to relabel just this one tape, you would double click it.
Speaker:And if you weren't paying attention, it would pop up a dialogue box.
Speaker:And if you click yes, fast and silent, go, what you just told it to do was
Speaker:relabel every tape in the tape library.
Speaker:Oh no.
Speaker:And I was there when, uh, I. When, uh, it was actually a Legato employee, uh, that
Speaker:was at the, it was a, it was a healthcare company up in, up in the LA area.
Speaker:And he, uh, he managed to relabel this customer's tapes, all of them.
Speaker:Um, again, I think he got like halfway through the tape library
Speaker:before he realized what he had done.
Speaker:there's no going back, right?
Speaker:Mm-hmm.
Speaker:Mm. Because once you put that, again, once you put the electronic label on
Speaker:the front, it puts an end of data marker after that, and you can't get past that.
Speaker:Um, I hope the customer had copies of those tapes, but the
Speaker:customer was not very happy.
Speaker:I would not be either.
Speaker:I.
Speaker:yeah.
Speaker:So
Speaker:Yeah.
Speaker:we've been talking about this for a while, but there is a third story
Speaker:or fourth story I want you to tell.
Speaker:Oh, you have another story.
Speaker:All right.
Speaker:want you to tell the story about, uh, temp and source code.
Speaker:That's a good one.
Speaker:That's a good one.
Speaker:Yeah.
Speaker:So, so for those that, that, that don't live in the Unix world, you know,
Speaker:temp, uh, you know, slash TMP or slash TMP depending on which os you were
Speaker:talking about, which, you know, which distribution was typically where you,
Speaker:you just put garbage files, right?
Speaker:You, you, it's temp, it's temporary.
Speaker:Right?
Speaker:And, um, the, um, and, and you just use it to, for garbage files, right?
Speaker:Uh, for scripting and stuff like that.
Speaker:And, um, the, uh, and, and then on one, some of the distributions and
Speaker:specifically on HPUX, which is what we were running, temp, was actually in Ram.
Speaker:Mm.
Speaker:And so when you rebooted the box, anything in temp would go, bye-bye.
Speaker:Yeah.
Speaker:Right?
Speaker:It's temp, that's the point of temp. And we, uh, one day I got a, um,
Speaker:a restore request for a directory structure in temp, and I said.
Speaker:What, what is this?
Speaker:And they go, oh, well, it's a, it's a source code tree that we've been
Speaker:working on, this group of consultants, like 20 of us or something.
Speaker:We've been working on it for a few months and it was in temp and
Speaker:this, the box, it got rebooted or you know, it just, it got deleted.
Speaker:They said like, they didn't know how it got deleted.
Speaker:It got deleted.
Speaker:I'm like, it's in temp. And they're like, yeah.
Speaker:I'm like, we don't back up Temp. We, we never have backed
Speaker:up temp. It's temp, right?
Speaker:It's like, it's like backing up your garbage can, like what,
Speaker:what, what is the point of that?
Speaker:Right?
Speaker:And um, and so they're like, well, you, you don't understand like, heads are gonna
Speaker:roll if we lose this, this source country.
Speaker:And I'm like, not my head like, yeah.
Speaker:was the genius who decided to put it in temp?
Speaker:Yeah, some.
Speaker:It was a developer.
Speaker:It was a developer, like a whole team of developers, and nobody was like,
Speaker:should, is this where we should put this?
Speaker:So know we're, we've been focused on like human error doing things accidentally,
Speaker:Yeah.
Speaker:right?
Speaker:And it's not necessarily just an end user who could be at fault.
Speaker:It could be an admin, it could just be company processes are broken.
Speaker:Right?
Speaker:But it's still like.
Speaker:I think the key here is asking questions before things go wrong, so you know how
Speaker:your data's being backed up and where it's
Speaker:Right, right.
Speaker:And, you know, and, and, and a lot of the, a lot of these problems, uh,
Speaker:could be fixed by versioning, right.
Speaker:Snapshots and things like that.
Speaker:So we don't have to use the quote unquote backup system, but I, I
Speaker:do think that this is like the number one reason that we do that.
Speaker:We do restores right.
Speaker:Do you think it's still the case today?
Speaker:Absolutely.
Speaker:Well, I mean, I think right now the number one, well again, the number one
Speaker:reason we do restores are, are, is humans.
Speaker:Right?
Speaker:We probably do more restores having to do with cyber incidents than
Speaker:we do stupid user, uh, incidents.
Speaker:But they, but they still happen.
Speaker:They still happen all the time,
Speaker:Do
Speaker:right?
Speaker:a lot of the user.
Speaker:Because you mentioned like the help desk, right?
Speaker:People would call in, they would ask for things.
Speaker:Do you think though, with a lot of the technologies out there that users
Speaker:are now doing sort of self-service restores where they don't need to be
Speaker:embarrassed by, Hey, I accidentally deleted this file, my resume dot
Speaker:doc, can you please bring it back?
Speaker:Yeah, I think, well, I think a couple of, I think a couple of things there.
Speaker:One is, uh, there's a lot of shadow it going on where people are doing their
Speaker:own backups or something like backups.
Speaker:Uh, and so they, they use that instead of having to, um, you know, call a help desk.
Speaker:And then the other is, I do think that there are a lot of technologies that
Speaker:have been deployed in data centers.
Speaker:Snapshots being the primary one.
Speaker:That have allowed users to essentially do their own restore, right?
Speaker:They know I can go to ~Snapshot and I can go in there and I can get whatever
Speaker:I want from whatever timeframe I want, and I, and I can get it from an hour ago,
Speaker:Yep.
Speaker:Just before I did the stupid thing.
Speaker:Uh, so I do think that.
Speaker:There's probably less of a reason for people to do, to call into an actual
Speaker:help desk or to go to a website and say, Hey, I need this file restored.
Speaker:Um, because there's a lot of self service.
Speaker:What's that?
Speaker:What's that?
Speaker:or the trash bin.
Speaker:Right.
Speaker:Yeah, yeah, yeah.
Speaker:have a trash bin where you can
Speaker:Yeah.
Speaker:your data.
Speaker:Well, the trash bin is usually only helpful if you delete files.
Speaker:If you corrupt the files, trash bins aren't usually helpful.
Speaker:Yeah.
Speaker:Um, and, you know, thi this, this is one of the things that
Speaker:I used to hear about Salesforce.
Speaker:I go, well, Salesforce has a, you know, it has a, again, a trash bin.
Speaker:And I'm like, right.
Speaker:But when you modify files, when you modify, uh, what they would call, um.
Speaker:Objects.
Speaker:call it records, actually, an object.
Speaker:They, they use the term object very, in my opinion, very weirdly to them,
Speaker:an object is like the user table.
Speaker:Oh, that's
Speaker:That's an object.
Speaker:I never understood that, but whatever, uh, I would call it a record, right?
Speaker:When you modify a record, there is no place where the previous
Speaker:version of that record is stored.
Speaker:Yep.
Speaker:so, um, and I remember one time where I, um.
Speaker:I screwed up my, we had a Salesforce database that had a couple million records
Speaker:Hmm
Speaker:and I screwed up every million of 'em by accidentally.
Speaker:Um, first off, it should be a federal crime that when you have an Excel
Speaker:spreadsheet and you click on a column
Speaker:That
Speaker:and then you say, sort.
Speaker:column and not everything.
Speaker:That should be, that should, whoever came up with that feature
Speaker:needs to be tarred and feathered.
Speaker:How is that?
Speaker:What, in what world is that what anyone wants to do?
Speaker:Right.
Speaker:I mean, I, I think now when you do it, it it, it'll, it'll say, did you mean to
Speaker:sort, you know, and you just have to push.
Speaker:You've got, yeah, no, I want to sort the whole thing, but like,
Speaker:why would that ever be the case?
Speaker:You know what I mean?
Speaker:It should be like, it should be the, like, what they did in AWS with o
Speaker:with open buckets where you have to like really, really, really try
Speaker:really hard to do an open bucket.
Speaker:It should be like that.
Speaker:Um, and if you, if you actually want to do a sort by just that
Speaker:column, you should have to really, you should have to really say so.
Speaker:It is like provide a blood oat and
Speaker:Yeah.
Speaker:Yeah.
Speaker:Because, you know, once you do that, uh, and so what I, what
Speaker:I did was I accidentally sorted the, uh, I, I downloaded the phone
Speaker:number column, which was, um.
Speaker:All the phone numbers, they're formatted, like all different.
Speaker:And so I figured out, I, I wrote a program to basically take out all
Speaker:of the formatting and then put back the formatting the way I wanted it.
Speaker:Right.
Speaker:Except I sorted it.
Speaker:Oh,
Speaker:And so when I uploaded that back to Salesforce using the data loader, I
Speaker:think that's what that was called.
Speaker:It's been a while.
Speaker:Um.
Speaker:It.
Speaker:I just put all the wrong phone numbers to all the wrong places,
Speaker:but luckily I had downloaded
Speaker:Uh
Speaker:table prior to doing that.
Speaker:Luckily, I had a backup there.
Speaker:Um.
Speaker:So do you wanna even touch on the second category?
Speaker:Because I think that's basically what we talk about on the
Speaker:podcast, like 97.2% of the time.
Speaker:Oh, the, the, um, the, um, the cybersecurity stuff?
Speaker:Yeah.
Speaker:Um, just, just a little bit.
Speaker:I do wanna specifically talk about the insider threat 'cause I don't
Speaker:think we talk about that as much.
Speaker:Right.
Speaker:Um, and I do think that you need to.
Speaker:Design your backup systems to be resilient against the insider threat.
Speaker:And we don't, and I don't think that people think about that enough.
Speaker:what's the
Speaker:think,
Speaker:thread?
Speaker:Thread?
Speaker:basically, it's someone who it's, it's a, it's a person who, um.
Speaker:Works in on the inside, they are one of your employees or contractors
Speaker:who have full access to whatever system we're talking about, and
Speaker:either they have been compromised or their user ID has been compromised.
Speaker:Right?
Speaker:Um, it, so typically what we mean, we're referring to them
Speaker:being compromised, right?
Speaker:They're, they're, you know,
Speaker:a bad actor inside the
Speaker:they're a bad actor inside the company.
Speaker:The, the most famous or infamous one,
Speaker:so this is a story from 2002, and the story that I'm looking at is actually
Speaker:a very ugly thing here on justice.gov and it talks about basically, um.
Speaker:I'll just read the opening sentence here.
Speaker:A disgruntled computer systems administrator for UBS was charged
Speaker:today with using a logic bomb
Speaker:Mm.
Speaker:cause more than $3 million in damage to the company's computer network and
Speaker:with securities fraud for his failed plan to drive the company's stock down
Speaker:with the activation of the logic bomb.
Speaker:You know the other story,
Speaker:Yeah.
Speaker:doing the one about unify where they claimed that they were being
Speaker:hacked and it was an insider?
Speaker:All right.
Speaker:Right.
Speaker:That,
Speaker:years ago.
Speaker:yeah.
Speaker:Yeah, yeah, yeah.
Speaker:So, you know, if you have somebody that's on the inside, they can do a lot of damage
Speaker:to your systems without, um, without you, before you're able to stop them.
Speaker:with great power comes great responsibility.
Speaker:Yeah.
Speaker:Well, and I will also say with great power comes great restriction
Speaker:because, because the more powerful a person's role is, the more controls
Speaker:should be placed around that power.
Speaker:Right.
Speaker:Um, and you should.
Speaker:You know, everything that you do should be logged and, you know, and, and,
Speaker:and the more destructive things you should be able to have to, you should
Speaker:have to get, you know, multi person authentication, you know, otherwise
Speaker:known as four eyes authentication.
Speaker:Um, and, um.
Speaker:Because, and, and this is just, just, you have to think about that because not
Speaker:only are you protecting from the insider threat, you're protecting from a bad
Speaker:actor, getting a, getting access there.
Speaker:There's been a lot of stories right with that.
Speaker:There was of course the LastPass incident, which is, you know, was the
Speaker:nail in the coffin for me regarding LastPass, uh, where basically they,
Speaker:the, the hacker was able to gain access to a backup of the vault.
Speaker:Because they had a, they had, they had, they weren't able to decrypt the vault,
Speaker:but they were get a, they were able to get access to the vault because there
Speaker:was this backup script that had hard coded, um, you know, credentials in it.
Speaker:And then once they gained there, there was a, there was an initial breach
Speaker:and then that breach gave them access to see the script, which then gave
Speaker:them access to the credentials, which gave them access to the backup, which
Speaker:gave them access to, uh, the vault.
Speaker:Which was, I think then used for targeted crypto attacks.
Speaker:was used for targeted crypto attacks against people.
Speaker:Yeah.
Speaker:Um, and so really all we're, you know, we're just talking about, just
Speaker:realize that human error is still one of the val, the biggest reasons that
Speaker:we do anything in the backup world.
Speaker:And, and when we talk about this just.
Speaker:Try to implement the concept of least privilege.
Speaker:Try to implement the concept of minimizing the blast radius, minimizing the ability
Speaker:for somebody to do something or, you know, the, the more dangerous something is,
Speaker:the more controls you can put around it.
Speaker:Uh, or should put around it.
Speaker:so going to a sentence, you just said,
Speaker:Yeah.
Speaker:humans are always at risk,
Speaker:Humans.
Speaker:Yeah.
Speaker:Right.
Speaker:Yeah.
Speaker:So.
Speaker:Humans are always de-risk, I would say.
Speaker:Yeah.
Speaker:So what if people are like, Hey, forget about humans.
Speaker:Let's switch over to ai.
Speaker:No, no, no.
Speaker:I, I wanna play a thought experiment,
Speaker:uh.
Speaker:What, uh, would your views change?
Speaker:Would your, you would recommend, because for humans, right?
Speaker:We talk about putting controls, doing multifactor authentication, multi
Speaker:authentication, All of these things, if people start to move towards ai.
Speaker:How does this change?
Speaker:I know this episode is talking about human error, right?
Speaker:But just to leave
Speaker:don't, I don't think I'm ready to answer that question.
Speaker:We'll, we'll cross that bridge when we come to it.
Speaker:How's that?
Speaker:Okay.
Speaker:I, well, I'll just say this in general, I'm a fan of automation and AI can
Speaker:enable automation, but it's gotta be put into a, it's gotta beral.
Speaker:and
Speaker:Yeah.
Speaker:else
Speaker:Yeah.
Speaker:Again,
Speaker:may be
Speaker:with
Speaker:but
Speaker:power
Speaker:yeah.
Speaker:great.
Speaker:You know, responsibility.
Speaker:Right.
Speaker:And so you want to, you want to, um, um, sorry.
Speaker:I just found myself thinking about, um,
Speaker:Spider-Man.
Speaker:Stanwell Stan Lee.
Speaker:I wonder if Stan Lee thinks about d the fact that his Spider-Man line would be
Speaker:quoted at a IT podcast decades later.
Speaker:Um.
Speaker:The, uh, yeah.
Speaker:So the more power the AI has, the more control you gotta put around it.
Speaker:Because it could do things right.
Speaker:It could be used to do things.
Speaker:Yeah.
Speaker:All right.
Speaker:Well, this has been fun, sad, depressing, uh, et cetera, but that's why we back up.
Speaker:And,
Speaker:And, um,
Speaker:and that's why we're human.
Speaker:yes.
Speaker:And I, I wanna wish you, you, you have an upcoming trip.
Speaker:Uh.
Speaker:I
Speaker:Yeah, wanna wish you a good trip, Bon voyage.
Speaker:Thank you.
Speaker:Thank you.
Speaker:I hope our listeners get to have a trip coming up.
Speaker:You know, maybe you have some fun.
Speaker:At least somebody will have some fun.
Speaker:And, uh, with that, that is a wrap.
Speaker:The backup wrap up is written, recorded, and produced by me w Curtis Preston.
Speaker:If you need backup or Dr. Consulting content generation or expert witness
Speaker:work, check out backup central.com.
Speaker:You can also find links from my O'Reilly Books on the same website.
Speaker:Remember, this is an independent podcast and any opinions that
Speaker:you hear are those of the speaker and not necessarily an employer.
Speaker:Thanks for listening.