Microsoft 365 Backup: The KPMG Disaster That Proves You Need It (Encore)
Microsoft 365 backup is something a lot of people still think they don’t need, and KPMG learned the hard way why that’s wrong. In August 2020, one admin tried to change the Teams retention policy for a single user and instead wiped out the private chat history of 145,000 employees. Microsoft couldn’t bring it back.
This is an encore episode. We brought it back because it’s one of the most popular episodes we’ve ever done, and not just in downloads. People listened to most of it, and some came back for more. It’s the first of seven episodes in our Cloud Disasters encore series. I recorded a new intro that ties it to November’s massive Microsoft outage. An outage is annoying. Losing your data is a whole different story.
Prasanna and I walk through what happened at KPMG and how it happened. The feature that deleted all that data was a retention policy. Some people point to that exact feature as the reason you don’t need Microsoft 365 backup. Retention policies are built for compliance and e-discovery. They’re good at keeping data you’re supposed to keep, and they’re good at making data disappear on schedule. What they can’t do is restore your stuff to a point in time. Get them wrong and they’ll delete everything.
We also talk about:
- the retention lock checkbox that would have saved KPMG’s old chats, and what it does to your storage bill
- why this could just as easily have been every email or SharePoint file in the company
- the legal mess when “our admin made a mistake” lands in the middle of pending litigation
- a Valentine’s Day phishing test that made an entire company feel loved for about five minutes
If you’re on the fence about Microsoft 365 backup, listen to this one. If you’re not, send it to someone who is.
Chapters
00:00 Encore intro: an outage vs. losing your data
01:34 Imagine every Teams chat disappearing
03:22 Welcome to the show
05:24 Why hardly anyone backs up Microsoft 365
09:01 What happened at KPMG
11:29 How Microsoft 365 retention policies work
15:19 Retention is for compliance, not recovery
18:08 How the deletion probably happened
21:51 What KPMG should have done
24:27 What Reddit and The Register had to say
26:27 It could have been email or SharePoint
28:04 Why not just delete the user?
29:07 Adverse inference and the legal fallout
31:24 The cloud isn’t magic
32:49 Humans are still the #1 reason we back up
33:52 A Valentine’s Day phishing test gone wrong
36:16 Advice for smaller companies
If you use Microsoft 365, you probably had a pretty bad Thursday.
Speaker:November 12th, Outlook, Teams, Copilot, even Intune and Defender went down in more
Speaker:than 70 countries, and it took Microsoft until the weekend to get everything back.
Speaker:Microsoft says it was caused by a network upgrade and not an attack,
Speaker:so at least that's some good news.
Speaker:But here's the thing.
Speaker:That was an outage.
Speaker:It's annoying, it's expensive, but at least your data comes back.
Speaker:But it did remind me of a story that we covered a few years ago where
Speaker:the data didn't come back when KPMG permanently lost the Teams chats for
Speaker:145,000 people because of one little mistake, and Microsoft says there
Speaker:was really nothing they could do.
Speaker:Two years later, they're still saying that backup of Teams chats is in development.
Speaker:So I decided that it's time for a, an encore series of our cloud disaster
Speaker:episodes, seven stories that, uh, talk about what happens when you
Speaker:assume the cloud has your back.
Speaker:I'm gonna show you the first one now.
Speaker:I will say I just took a quick look at the video and, uh, if you're
Speaker:watching this on YouTube, you're gonna get to take a look at heavier me.
Speaker:Uh, this is me from before my weight loss . So I'm about to gain about 110 pounds.
Speaker:It's kind of hard to look at for me, so, just know that I did this
Speaker:on purpose and, uh, I'm happy I did it and, um, I feel a lot better.
Speaker:I hope you enjoy the episode
Speaker:Imagine this.
Speaker:You're in the middle of a really important conversation in Microsoft teams.
Speaker:You spend several minutes typing up a response that
Speaker:will finally make your point.
Speaker:But the moment you hit enter the message disappears.
Speaker:And it turns out.
Speaker:So did everyone else's responses?
Speaker:Then you find out that every single conversation in Microsoft, 365 is gone.
Speaker:That's what happened to 145,000 KPMG employees, right in
Speaker:the middle of the pandemic.
Speaker:The Microsoft 365 admin was trying to delete a single user's chats
Speaker:when they actually deleted the entire company's data instead.
Speaker:And today's episode, we'll dive right into this jaw-dropping data disaster
Speaker:and see how it serves as a powerful reminder of the importance of backing
Speaker:up your cloud and SAS environments.
Speaker:We'll be discussing how Microsoft retention policies are not backups.
Speaker:This story proved my point.
Speaker:It's, Microsoft's turn.
Speaker:If you hear this story and still think you don't need to back up Microsoft 365.
Speaker:I just don't know what to say.
Speaker:Hi, I'm W, Curtis Preston.
Speaker:AKA Mr. Backup.
Speaker:30 years ago, my employer lost their purchasing database.
Speaker:When it turned out, my backups were broken.
Speaker:I vowed that would never happen to me again.
Speaker:And now I try to do the same for those who read my books and listen to my podcast.
Speaker:I want to turn you the unappreciated backup admin into a cyber recovery hero.
Speaker:This is the backup wrap up.
Speaker:Welcome to the show.
Speaker:I'm your host, w Curtis Preston, AKA, Mr. Backup, and with
Speaker:me, I have what I hope to be.
Speaker:My personal non-ad advisor on taxes
Speaker:Is it that time of year, Curtis?
Speaker:is that time of year.
Speaker:Although I have to say I have.
Speaker:I, this will be a complicated year, right?
Speaker:Due to a variety.
Speaker:A variety of things, right?
Speaker:The interesting year of employment and non-employment and contractor
Speaker:work that I did last year.
Speaker:So let me get, are your taxes like already done?
Speaker:No comment.
Speaker:Okay.
Speaker:I just, you just seem so put together with this whole tax thing that I
Speaker:figured that by now you'd be done
Speaker:so I was just waiting for the last of the paperwork, but I am one of those
Speaker:crazy people who pre-plan everything.
Speaker:Yeah.
Speaker:You got spreadsheets and stuff.
Speaker:so usually I know more or less where I'll land based on pay stubs and it's
Speaker:just the final bits of like interest and dividends and other things like that.
Speaker:You probably wouldn't be sleeping at night if you're in my
Speaker:situation right now because I'm.
Speaker:Yeah.
Speaker:Yeah.
Speaker:I don't know how you do that.
Speaker:I, so I should say that 99.99999% of the people are like you,
Speaker:I think you're right.
Speaker:Yeah.
Speaker:In fact, I know based on my work with Intuit back 25 years ago, that like they
Speaker:were saying, if I recall correctly, it was like something like 90% of their
Speaker:business on TurboTax occurs within the last three days before April 15th.
Speaker:Doesn't surprise me.
Speaker:One challenge too is there's a lot of fraud.
Speaker:So someone takes your social security number files.
Speaker:A fake tax return.
Speaker:Gets the refund.
Speaker:that's, that is really hard to recover from.
Speaker:Yeah, you know what else is real hard to recover from?
Speaker:What?
Speaker:deleted data in a SaaS provider that you don't have a backup of.
Speaker:See that segue?
Speaker:You're proud of that segue.
Speaker:Look at
Speaker:good.
Speaker:So this will be a part of our continued series on why
Speaker:you should back up the cloud.
Speaker:And the cl there, there's a bunch of different parts of the cloud.
Speaker:Today we're gonna be talking about a SaaS provider, possibly
Speaker:of the biggest SaaS providers,
Speaker:I think they're one of the most valuable companies in the world right now.
Speaker:So that would be Microsoft.
Speaker:And I remember in the, my days at Druva, this was what we saw as like one of the
Speaker:biggest greenfield environments because hardly anybody backed up Microsoft 365.
Speaker:don't need to back up Microsoft 365 Curtis,
Speaker:You reach
Speaker:over Strangle me through the video.
Speaker:Yeah and the thing was, it was super difficult for me as a person who has
Speaker:dedicated their career to backups.
Speaker:I really don't understand the people that have migrated to a SaaS provider
Speaker:have migrated any portion of their.
Speaker:IT infrastructure to some type of other provider of any kind, whether
Speaker:it's a, I, a s, past SaaS, whatever, and didn't check this box, right?
Speaker:Didn't ask how does backup and recovery work there?
Speaker:What's the thing that didn't look in the service agreement, didn't do any
Speaker:testing of any backup, let's delete.
Speaker:Something and then restore it and,
Speaker:happens.
Speaker:like, I've known for a long time that people hate backup.
Speaker:Nobody wants to do
Speaker:I am sorry Curtis.
Speaker:yeah.
Speaker:Nobody, yeah.
Speaker:Nobody wants to be the backup person.
Speaker:That's how I got my job.
Speaker:That's how I got my first job, was the, Ron Rodriguez didn't
Speaker:want to be the backup guy anymore.
Speaker:And so that's how I got my job.
Speaker:And that's how many of our listeners got their jobs.
Speaker:And it's how.
Speaker:Much of the cloud migration has happened, right?
Speaker:It's people move to something like Microsoft 365, and I do know for a fact
Speaker:that some Tams technical account managers at Microsoft 365 have uttered words to
Speaker:the effect that they don't have to worry about backup, and that's one of the
Speaker:reasons that they should migrate to 365.
Speaker:They're wrong.
Speaker:They don't do it in writing,
Speaker:This isn't unique to Microsoft 365 a SaaS or path, because do you remember back
Speaker:in the day with virtual machines, right?
Speaker:And VMware, right?
Speaker:They had the same message as well, right?
Speaker:And it wasn't until people were like, Hey, I'm now using this for
Speaker:production workloads and I really should care about how I back it up.
Speaker:And then that spurned okay, Veeam came along with other backup vendors, right?
Speaker:Figuring out how to optimize it and make it better.
Speaker:But the initial ones you never had backups.
Speaker:Yeah.
Speaker:Backup never leads the charge in anything.
Speaker:And but it's still depressing to me sometimes.
Speaker:And in this case, there are people, I can think of one particular person
Speaker:I won't mention in my name, but I can think of one particular person that
Speaker:is a fellow book author that, just really doesn't seem to see the need.
Speaker:To backup 365, even in his blog post about the incident that
Speaker:we're gonna talk about today.
Speaker:He said, some people will probably tell you that this is an example of why you
Speaker:need to back up Microsoft 365, but I wanna tell you that even if you had backups
Speaker:that wouldn't have helped in this case.
Speaker:And he's right, but doesn't change the
Speaker:Yeah.
Speaker:So enough with the preamble.
Speaker:What are we talking about here?
Speaker:What happened back in 20, August of 2020, what happened?
Speaker:Isn't that right around the pandemic, Curtis, where aren't we all locked down?
Speaker:Not going
Speaker:we were all, yeah.
Speaker:So while we were all locked down,
Speaker:Yeah.
Speaker:Of little companies.
Speaker:so this was a large very large company called KPMG, which is a
Speaker:consulting slash auditing firm.
Speaker:They do both,
Speaker:And they decided one day, Hey, I'm going to go change a retention policy for my
Speaker:Microsoft teams for a particular user.
Speaker:right.
Speaker:And they went and made that change.
Speaker:And apparently they didn't realize it affected all users.
Speaker:And by the way, that small company you're talking about, it's like 145,000 people,
Speaker:45,000 people
Speaker:And so it.
Speaker:So it impacted the retention for those 145,000 people.
Speaker:And so all these folks who are basically like, auditing companies
Speaker:and have a bunch of sensitive chat messages and other things.
Speaker:Yeah.
Speaker:All those messages went byebye.
Speaker:Yeah, just poof.
Speaker:And I'm sure that they reached out.
Speaker:We don't, it's not in the story, but I am absolutely sure that they
Speaker:reached out to Microsoft, said, is there anything you can do?
Speaker:And the answer is no.
Speaker:Because what you experienced is a feature, not a bug.
Speaker:yeah.
Speaker:And
Speaker:The, go
Speaker:no, go finish.
Speaker:The true irony of this is that the feature that they were using.
Speaker:That resulted in the deletion of data is the feature that people, including
Speaker:the guy to which I was referring, they pointed this feature as a substitute
Speaker:for backup, that you should use this fe.
Speaker:That if you had, if you properly use this feature, you wouldn't need backup.
Speaker:But apparently if you improperly use the.
Speaker:Yeah.
Speaker:So maybe Curtis, why don't you talk a little bit about the
Speaker:feature itself and what it does?
Speaker:Yeah.
Speaker:So the feature that we're talking about is retention policies,
Speaker:and they are complicated.
Speaker:They're, the last time I checked, if you went and looked at the justice feature
Speaker:in the documentation, there's 25 pages of documentation on this one feature.
Speaker:That Microsoft 365 has, and basically what it's designed to do is to, in
Speaker:general, the idea is to make sure that things that should be around
Speaker:are around and subsequently things that shouldn't stay around, don't.
Speaker:A typical retention policy might be 90 days.
Speaker:We set a retention policy on email 90 days, which means that after 90 days, what
Speaker:sent email a received email of any kind, again, based on the policy gets deleted.
Speaker:You could also set it to be a year, you could set it to be seven years,
Speaker:whatever.
Speaker:You can set different retention policies for email, for SharePoint, for OneDrive.
Speaker:You can set all these different policies.
Speaker:You can also set a single policy.
Speaker:Everything in Microsoft 365 is kept for.
Speaker:End days and no longer, you can set it at a group level, you can
Speaker:set it at an individual level.
Speaker:And the reason why and it has an optional and what I'm gonna call, and
Speaker:I really mean it feature which is a checkbox that says once you set it.
Speaker:can't change your mind.
Speaker:So once you say Everything in this environment should be
Speaker:kept for a year, you can't.
Speaker:Then 60 days later, change your mind.
Speaker:Say I'm sorry.
Speaker:I meant something Way less than a year.
Speaker:Sorry.
Speaker:You've turned it on for a year.
Speaker:That means any data created stays for that year.
Speaker:If I created new data after changing into 60 days, new data would stay for 60 days.
Speaker:Old data would stay for one year.
Speaker:right, and so what can happen if you use this policy is and the reason
Speaker:why I say this is those that are proponents of using retention policies.
Speaker:Instead of backup, they basically say, look, Microsoft is really resilient.
Speaker:They've got delayed copies and all this kind of stuff in there.
Speaker:And so really like Microsoft will protect the Microsoft environment, right?
Speaker:And that if they have to recover the Microsoft environment, you will get
Speaker:recovered as a consequence of that.
Speaker:So you don't, these are the proponents of this is not me, this is,
Speaker:other folks.
Speaker:They'll say, you don't have to worry about that.
Speaker:What you need to worry about is a ransomware attack or a
Speaker:massive accidental deletion, like what happened at KPMG, right?
Speaker:Yeah.
Speaker:And so you can say, um.
Speaker:Everything gets kept for X amount of days, and you can check the box
Speaker:that says and I can't change my mind.
Speaker:And then after, 30, 60, 90 days when you find out what that does to your storage
Speaker:and you're like, holy cow, I didn't realize this was going to do this to
Speaker:my storage, and subsequently my bill.
Speaker:Yeah.
Speaker:You can't change your mind.
Speaker:You can change your mind, but you can't change your mind for that period.
Speaker:It's not retroactive, right?
Speaker:You told it, you know that you wanted to keep it and you wanted to
Speaker:do the, the the compliance feature.
Speaker:Since you just mentioned compliance, so retention though, right?
Speaker:It's typically associated with a compliance policy.
Speaker:And even if the user goes and deletes that email before the retention period
Speaker:is up, because it's intended for compliance, that email is still kept
Speaker:around until the retention policy expires.
Speaker:Yeah.
Speaker:When I delete an email, all it does is set the don't show it to Curtis
Speaker:Yeah.
Speaker:It's still there.
Speaker:The and the other thing is that.
Speaker:It's designed you said it's designed for compliance.
Speaker:It's designed, it's the complete opposite of backup.
Speaker:So you know how we talk a lot about backup versus archive.
Speaker:This is much more about archive and retrieval and e-discovery than it is about
Speaker:recovering Curtis's email box because
Speaker:To no point in time.
Speaker:At no point in time.
Speaker:Yeah, at a point in time, what it can do is you can go then do an e-discovery case.
Speaker:First off, you need different permissions.
Speaker:It's different than the default system admin permissions.
Speaker:You need separate permiss permissions to do this, and then you can
Speaker:go in and say, please extract all of the email that Curtis.
Speaker:Has ever received or received in this time period.
Speaker:And then they can throw that in your inbox all in one big pile, not in the folders
Speaker:or whatever, that you had it before.
Speaker:And by the way, not just email, but the same is true in SharePoint or whatever.
Speaker:It's just a big pile of stuff.
Speaker:And then they put it back, and then you gotta go figure out
Speaker:what you don't need, right?
Speaker:and usually you use retention policies because keeping data around
Speaker:is actually sometimes a bad idea
Speaker:this is a, it's a legal, it's a legal issue, right?
Speaker:And so you say, look, we're not gonna keep data longer than a year.
Speaker:Period.
Speaker:I know that when I worked at Druva, they had implemented a data retention
Speaker:policy on Slack, for example.
Speaker:They only would allow Slack messages for I think it was like 90 days
Speaker:Yeah.
Speaker:So a couple of things about retention policies.
Speaker:One is it's not good at restore it any more than a backup tool is good
Speaker:at e-Discovery, this is an e-discovery tool that's not good at backup.
Speaker:It's not good at Restore.
Speaker:And so
Speaker:if you got a large ransomware attack.
Speaker:It wouldn't be good at that, right?
Speaker:The other thing is that it can result in significant increases in your cost, right?
Speaker:And your storage, as opposed to just using a backup tool, which would
Speaker:also have an increase in cost, but a different increase in cost, right?
Speaker:And more, and a more predictable increase in cost.
Speaker:And then, and to the point of this story, it can sometimes really screw you up.
Speaker:yeah.
Speaker:If you don't understand and aren't careful,
Speaker:Yeah, so as I understand it, so the way that they would've done
Speaker:this, and this, by the way, this is inference not information.
Speaker:When you've got 145,000 people and you have one person whose private
Speaker:chats you wanna delete, and you've got a retention policy that says
Speaker:for 145,000 people, we keep.
Speaker:Private chats for six months whatever the number is, right?
Speaker:90 days, you can't delete those private chats as long as that person is in
Speaker:the in that policy.
Speaker:So you have to create a different policy that says private chats
Speaker:get kept for one day, and then you move that user into that policy.
Speaker:Yep.
Speaker:Apparently they just, they did the wrong
Speaker:They just edited the policy.
Speaker:they create, yeah.
Speaker:They edited the policy and then poof instantly.
Speaker:And again, they didn't have the flag.
Speaker:See if they had the flag turned on.
Speaker:And
Speaker:Their old data would've been fine.
Speaker:Yeah, the old data would've been fine.
Speaker:They wouldn't have been able to mess it up like this.
Speaker:So one message could be, if you're gonna use retention policies,
Speaker:you better turn on that flag.
Speaker:'cause they all, they're all they are all you have.
Speaker:It wouldn't save you though from new messages created, so you
Speaker:No, but at least you wouldn't have deleted six months.
Speaker:I think he probably figured this out in about five minutes,
Speaker:But would you though,
Speaker:because now anything new that you type would be following that new
Speaker:policy and maybe you don't realize it.
Speaker:Yeah, I just think it would be less of a tragedy than I do think it's
Speaker:important to say that had they, there are no APIs for restoring.
Speaker:Teams messages.
Speaker:There are APIs and there are ways to get these messages.
Speaker:They are, and some of them are frankly, very klugy.
Speaker:I remember participating in some of these conversations, when I was
Speaker:at Druva, getting some of this data is very klugy because it wasn't
Speaker:really designed to be gotten right.
Speaker:If you had done, if they had done that, they wouldn't have been able
Speaker:to restore the team's messages, but
Speaker:would've had it at least.
Speaker:they would've at least had it.
Speaker:And if there was anything important in there, which there shouldn't be,
Speaker:there shouldn't be important stuff and private messages.
Speaker:But remember, we live in a world where people use their recycle bin for storage.
Speaker:You've seen that,
Speaker:Yep.
Speaker:You are like, Hey, your hard drive is filled, and you click
Speaker:the recycle bin whatcha doing?
Speaker:You just deleted my storage area.
Speaker:Yeah, that actually was the last thing that happened to
Speaker:me at the bank was deleting.
Speaker:I rebooted a server
Speaker:It was
Speaker:Somebody rebooted a server and it was hp where HP that slash TMP was in ram and
Speaker:we found out a whole bunch of developers had stored their code tree there.
Speaker:And they were like, this is really important.
Speaker:I'm like, then you shouldn't have put it in temp. And they were very
Speaker:angry because we didn't back up temp
Speaker:Who would?
Speaker:It's Tim.
Speaker:temp
Speaker:and we lost months of development work.
Speaker:So we live in that kind of world, right?
Speaker:And people do sometimes put important stuff in there and anything that was
Speaker:important there were in messages, which shouldn't have been, but anything that
Speaker:was there would've been instantly deleted.
Speaker:Opening the door when you're out to space,
Speaker:Don't do it.
Speaker:don't do it.
Speaker:So let's go back to this, right?
Speaker:So they lost a bunch of messages because someone accidentally
Speaker:set the retention policy.
Speaker:What should they have done?
Speaker:What could they have done to make this less painful?
Speaker:I'd say a couple of things.
Speaker:One, the first and most obvious to me was if they had a backup, then this
Speaker:would've been less painful, right?
Speaker:They would've been able to at least retrieve, important data.
Speaker:I. And by the way, not all products at Backup 365 are able to get
Speaker:anything out of teams, teams messages.
Speaker:Some have to different degree, different levels of success.
Speaker:But they would've had the possibility of getting that data back.
Speaker:And number two is, it is a little surprising that an admin would
Speaker:do something of that level.
Speaker:I. With a mouse.
Speaker:Without any controls or any checks or anything else like
Speaker:someone else at least approve or look over, Hey, are you doing the right thing?
Speaker:yeah.
Speaker:You would think that something of that level, I guess he wasn't
Speaker:thinking or she wasn't thinking that they were just thinking one person.
Speaker:It's just one guy.
Speaker:I'm just doing one guy.
Speaker:But that's where a process becomes so critical, and documenting process and.
Speaker:Yeah.
Speaker:So they could have, they also could, had they enabled the
Speaker:retention lock, that's what it's called, the retention lock feature.
Speaker:They would've at least kept the data from before.
Speaker:And they would've minimized the pain if they turned on the retention lot feature.
Speaker:But I'm not a big fan of the retention lot feature because.
Speaker:Its potential significant increase in in storage costs.
Speaker:and especially, it's so hard to predict how much space you have or would be
Speaker:consuming before you enable that feature.
Speaker:Like any sort of capacity planning or sizing, calculators, I've seen even for
Speaker:general storage, they're not accurate because your workload is going to
Speaker:change and it's going to depend on your environment and how people use the tools.
Speaker:And so it's just a best guess.
Speaker:And until you actually turn it on, start using it, you're never gonna know.
Speaker:Yeah, I would think that if you're gonna use the retention
Speaker:policies instead of backup, right?
Speaker:Which you shouldn't do, but if you did, should turn it on for, let's say three
Speaker:months and see what it does to your storage and then make a decision at that
Speaker:point as to whether or not you wanna.
Speaker:Essentially make this permanent.
Speaker:And then when you see what it does to your storage, then maybe have a
Speaker:conversation with a backup vendor and say, Hey, can you do this with
Speaker:less of an impact to my environment?
Speaker:Right.
Speaker:Now some people, I read with great delight.
Speaker:I read the comments in Reddit, there's a Reddit thread
Speaker:about this, and there's also.
Speaker:A comment thread from the register article.
Speaker:I, it's one thing I love about the register is that they have
Speaker:comments and boy do people get in.
Speaker:What's
Speaker:We'll, the links in the show notes.
Speaker:Yeah.
Speaker:Thanks to those.
Speaker:Yeah.
Speaker:And of course you got the standard stupid cloud, it's where you want to put, I'm
Speaker:like, dude, this could have been exchange,
Speaker:right?
Speaker:This could have been an exchange environment.
Speaker:Then the other, of course, is why didn't they go, why
Speaker:didn't they go to their backup?
Speaker:People just completely ignorant that and I, that's the proper word.
Speaker:People like, they're like, oh, that's pejorative.
Speaker:It's the proper word.
Speaker:They don't know that Microsoft 365 doesn't have backup.
Speaker:If you are not providing it, it doesn't have it.
Speaker:Given how big KPMG is and what sort of things that they provide to their
Speaker:clients, that's a little shocking.
Speaker:If they didn't know that there's no backups.
Speaker:I'm sorry.
Speaker:That's just bonkers.
Speaker:It is bonkers, but it's a bonkers that I run into all the time.
Speaker:Yeah.
Speaker:so I know that we're looking at this as people lost a bunch of data.
Speaker:And people lost their chats.
Speaker:It was probably a lot of sensitive information or
Speaker:important information was lost.
Speaker:I was just going in my head about what, that must be amazing because
Speaker:it's like you hit a reset, you don't have to worry about deleting
Speaker:conversations or anything else, right?
Speaker:It's like clean slate.
Speaker:It's like you start a new job somewhere.
Speaker:It's no previous chat messages, no baggage, no nothing that
Speaker:you have to worry about.
Speaker:Yeah.
Speaker:The other thing, when I think about this.
Speaker:Again, when I talk about this other person, they're like
Speaker:it was just private messages.
Speaker:You shouldn't have been putting anything of value in private
Speaker:messages.
Speaker:That is that people do it all the time.
Speaker:But they're right.
Speaker:They're right.
Speaker:You shouldn't do.
Speaker:But this could have just as easily been email,
Speaker:Or SharePoint.
Speaker:right?
Speaker:So retention policies are global.
Speaker:If you want them to be right, you can make them as granular as you want.
Speaker:Retention policy for one person, which is what they were trying to do, they could
Speaker:have just as easily deleted all email
Speaker:in the environment.
Speaker:And They would've had no backup, they would've had no recourse.
Speaker:'cause the whole point of retention policies, by the way, this is maybe
Speaker:something I should have mentioned before.
Speaker:The point of them is that if you say don't keep it longer than
Speaker:X, it gets rid of everything.
Speaker:It, it keeps it longer than, if you set it to 90 days it deals with recycle bin.
Speaker:But wherever that entity is, that record, that email,
Speaker:Once it's 90 days, it's boom.
Speaker:Gone.
Speaker:it, boom, it's gone.
Speaker:And in this case, they set 90 to one or zero.
Speaker:And that basically that means that there was no recycle bin
Speaker:for them to go get this out of.
Speaker:And that means if they did the same thing with email, they would've wiped out
Speaker:the entire organization's entire email
Speaker:infrastructure
Speaker:Or if they did that with OneDrive then all the documents are gone.
Speaker:So here's my question though, is assuming that they wanted to, because.
Speaker:Given that they wanted to set this user's retention policy to zero, that pretty
Speaker:much means they want all the data gone.
Speaker:Right.
Speaker:Why didn't they just delete the user?
Speaker:Because it would, the data would've stayed.
Speaker:Oh, okay.
Speaker:Yeah.
Speaker:Again, that's the whole point of retention policies.
Speaker:And I don't know, we can.
Speaker:We can theorize as to what did this user say in their private messages
Speaker:that they've really wanted gone.
Speaker:We can theorize, by the way, there are theories I should just mention
Speaker:them as long as we're talking about When I look, when you look at the
Speaker:thread, there, there were pending legal matters where these private messages.
Speaker:Perhaps could have been helpful, but now they're not available and this story would
Speaker:at least be a legally defensible reason why it deleted our admin screwed up.
Speaker:This wasn't spoilage, this wasn't purposeful deletion because of a
Speaker:lawsuit, but it also violates if they had a lawsuit that included this data.
Speaker:It could potentially create an adverse inference thing.
Speaker:And I'll just explain that for those that don't, basically if you're being
Speaker:sued or being prosecuted for something, if the judge believes that you have
Speaker:either destroyed evidence or have behaved poorly the, basically the judge
Speaker:can infer from that behavior something adverse to your case, and they could
Speaker:literally, like in the case of a jury.
Speaker:They can literally turn to the jury and say whatever the plaintiff said
Speaker:was in those private messages, just go ahead and assume it was there
Speaker:because why the hell else would they delete the private messages?
Speaker:That's what's called an adverse inference instruction.
Speaker:And or if if the judge is by themselves, they can just do it,
Speaker:but, and I do wonder though if given the fact that they weren't
Speaker:doing backups of this data, I wonder if that does qualify for that.
Speaker:Because yeah, you can blame that the admin was, did something by mistake,
Speaker:but there's no reason you shouldn't have backups of business data, you know?
Speaker:I agree with you there.
Speaker:They possibly could say even if we had backups, we wouldn't have had the
Speaker:data or whatever, because it's teams,
Speaker:But you have the data, you could extract it as on, you'd maybe not
Speaker:restore it directly back into teams, but at least you have the data.
Speaker:Yeah.
Speaker:I think that any adverse inference that could be inferred from their behavior
Speaker:is in the sole discretion of a judge.
Speaker:And so the judge d different judges are going to have different levels of do they
Speaker:believe that people are inherently good?
Speaker:They're, who knows what it this was four years ago.
Speaker:We could probably find out, but I didn't care that much about the part of it,
Speaker:but I just wanted to say that this was.
Speaker:Part people were specifically pointed to specific cases that this is in
Speaker:the register thread, by the way, for anyone who wants to look it up.
Speaker:This certainly is a nice way to get rid of a bunch of data with a nice story.
Speaker:That doesn't sound like you're trying to do it purposefully.
Speaker:I'm not saying that's what happened, but I'm saying some people
Speaker:So here's the thing.
Speaker:In summary, Microsoft 365 isnt magic.
Speaker:The cloud isn't magic.
Speaker:You need to back up the cloud somehow, right?
Speaker:Just it's not like people thought that, especially certain people
Speaker:thought that I was saying this just because I work for dva, right?
Speaker:I don't work for Duv anymore.
Speaker:The, it's not magic, right?
Speaker:It, it needs some kind of backup.
Speaker:I don't, it doesn't have to be a third party.
Speaker:I would prefer it be a third party.
Speaker:But
Speaker:So you look at so Microsoft is starting to talk about snapshots
Speaker:and some sort of recovery.
Speaker:I don't yet know how that will all play out, but in general and if it meets.
Speaker:The
Speaker:The usual, like 3, 2, 1 requirement.
Speaker:'cause again, going back to the 3, 2, 1 rule, having three copies of your
Speaker:data, two of which on, different media with different risk profiles, one of
Speaker:which being stored somewhere else.
Speaker:The thing with 365 and a lot of SaaS providers is that they don't
Speaker:do the two or the one, right?
Speaker:It don't do any of this.
Speaker:They don't have a separate copy of your data.
Speaker:That's basic.
Speaker:It's not magic.
Speaker:The cloud doesn't solve all new problems.
Speaker:In fact, it creates new ones and I'm pro cloud.
Speaker:How are you feeling Curtis, about this?
Speaker:I know.
Speaker:I was in a pretty good mood.
Speaker:The beginning of this.
Speaker:You to.
Speaker:and I think this just goes back to humans make mistakes.
Speaker:You gotta back it up.
Speaker:Humans are the number one reason we back up.
Speaker:It's, that's been more, that is more true today than it ever was 30 years ago.
Speaker:The number one reason we were doing restores would be 'cause
Speaker:somebody deleted a file, right?
Speaker:They fat fingered a file, they fat fair, and get the current version of
Speaker:the file and they accidentally saved it.
Speaker:Yep.
Speaker:That was the number one reason we did 30 years ago.
Speaker:And every once in a while we would also lose a server.
Speaker:Because remember back then we didn't even have a raid, So if you lost a hard drive,
Speaker:was very
Speaker:you lost a server, right?
Speaker:You move 30 years up and it's still stupid.
Speaker:User errors and ransomware, which is another human caused
Speaker:reason that we do restores.
Speaker:Because also isn't most of the ransomware attacks started with
Speaker:phishing more than anything else.
Speaker:Yes.
Speaker:Yes.
Speaker:it's an employee who accidentally clicks on a link that they shouldn't be clicking.
Speaker:I saw I'll tell a little story that I saw today on on, tikTok and it was a guy
Speaker:who he was like, he said, you would not believe what my wife's company did today.
Speaker:The video was actually made yesterday, so it's, this would be Valentine's Day.
Speaker:SO for those of you that don't know many companies, if not most companies do
Speaker:automated phishing tests, they send you.
Speaker:Emails that are essentially phishing emails to see if you will click on
Speaker:them or to see if you will notify them.
Speaker:And he said they sent an automated email or they sent one of these testing
Speaker:emails to everyone in the company that said you have a Valentine's Day delivery
Speaker:from Edible Flowers at the front desk.
Speaker:And
Speaker:Click here.
Speaker:And then there was a click, right?
Speaker:And he said for a few minutes.
Speaker:Everyone in that company felt they were loved that they found out that
Speaker:it was a
Speaker:phishing email.
Speaker:Instead of getting love, what they got was yelled at by it.
Speaker:And
Speaker:goes, accept.
Speaker:He said, accept my wife because my wife knew.
Speaker:There's no way I'm spending $200 on a, on some cantaloupe that's
Speaker:made to look like a flower.
Speaker:That is hilarious.
Speaker:But yeah, that's just, the sad thing is, like I see both
Speaker:sides of this argument, right?
Speaker:Phishing emails.
Speaker:They, they come they are timed two things, right?
Speaker:They would a bad guy would do this.
Speaker:They would leverage Valentine's Day, they would leverage Christmas
Speaker:at the company or something like that, right?
Speaker:exactly right.
Speaker:And so on one hand I totally understand why they did what they
Speaker:did, but it's also messed up.
Speaker:And then this guy that posted it, there were a lot of comments and people were
Speaker:like, yeah, we were told we got a bonus.
Speaker:We were told we got this.
Speaker:We were told we got that.
Speaker:And it was all a phishing test.
Speaker:That's no.
Speaker:It was a phishing
Speaker:test.
Speaker:Like it was the.
Speaker:And so instead of getting a bonus or Edible flowers, they get to go for a
Speaker:one hour training session on fishing.
Speaker:Yeah.
Speaker:So you need to back up your stuff.
Speaker:You need to back up Microsoft 365.
Speaker:In this case, this is just the best, this is the current best example we have.
Speaker:If we have a new one, we'll make another episode.
Speaker:Last.
Speaker:Oh, one last thing to add.
Speaker:Realize KPMG is a humongous company and they had issues.
Speaker:If you're a smaller company right?
Speaker:Don't feel bad that you're like, you haven't done anything yet, right?
Speaker:But now that go figure out how you are gonna back up your SaaS applications.
Speaker:If you're using Microsoft 365, figure out how you are gonna back
Speaker:it up so you don't end up like KPMG.
Speaker:Absolutely.
Speaker:I will poorly quote Maya Angelou.
Speaker:We did what we did when we knew what we knew, and now we know better.
Speaker:We can do better.
Speaker:It's a great quote even if I didn't quite get it right, but that's Maya Angelou.
Speaker:May she rest in peace.
Speaker:Thanks persona.
Speaker:No, this was fun.
Speaker:This was fun and I hope you sound a bit more cheerful now towards the
Speaker:end versus about five minutes ago.
Speaker:So that's a good sign.
Speaker:I was angry
Speaker:and thanks for those of you that are listening and look forward to some more.
Speaker:You should have backed up your stuff, episodes coming here in the
Speaker:coming weeks, and that's a wrap.
Apple Podcasts
Spotify
Castro
RSS Feed