Salesforce.com's Permission Slip-Up (Another Cloud Disaster)
In this episode of The Backup Wrap-Up, Curtis and his co-host dive into the chaos caused by Salesforce's accidental "modify all" permission change in 2019. They explore the fallout from this real-world SaaS disaster, including how Salesforce scrambled to restore proper permissions and the frustration felt by impacted customers.
Curtis and his co-host discuss the crucial role third-party backups could have played in mitigating the impact of this incident, and why relying solely on a SaaS vendor's recovery capabilities can leave organizations vulnerable. They also share practical advice on how listeners can avoid similar cloud disasters by implementing a comprehensive backup strategy for their SaaS applications.
Whether you're a Salesforce user, a SaaS enthusiast, or simply interested in the world of data protection, this episode offers valuable insights and entertaining anecdotes that will help you become a Cyber Recovery Hero. Tune in to learn, laugh, and discover how to safeguard your organization's critical data in the cloud.
Links
- Original SF post: https://issues.salesforce.com/issue/a028c00000qQ53kAAC/user-profiles-and-permission-sets-related-to-pardot-licensed-orgs-were-modified-by-salesforce
- Second post: https://salesforce.stackexchange.com/questions/262830/salesforce-bug-enabled-modify-all
- Big deal: https://appomni.com/blog_post/2019-blog-modify-all/
- SF follow up: https://help.salesforce.com/s/articleView?id=000384056&type=1
Mentioned in this episode:
Blank Midroll
Speaker:
what happens when a SAS giant accidentally grants modify all permissions to
Speaker:
every user in every customer org.
Speaker:
Spoiler alert.
Speaker:
It's not pretty.
Speaker:
Join me and my co-host as we explore the fallout from this
Speaker:
real world cloud catastrophe.
Speaker:
We'll discuss how Salesforce scrambled to restore proper permissions.
Speaker:
The frustration felt by locked out customers and the crucial role
Speaker:
third-party backups could have played.
Speaker:
You'll learn why relying solely on your SAS vendors, recovery capabilities
Speaker:
might leave you high and dry and how having your own backups can save the day
Speaker:
when things go sideways in the cloud.
Speaker:
If this is your first time listening.
Speaker:
Hi, I'm W.
Speaker:
Curtis Preston also known as Mr.
Speaker:
Backup.
Speaker:
My career in backup began over 30 years ago when my backups failed
Speaker:
and my company was unable to restore their purchasing database.
Speaker:
I vowed that would never again happen to me.
Speaker:
And it's my goal to do the same for you.
Speaker:
I want to turn you the unappreciated backup admin.
Speaker:
Into a cyber recovery hero.
Speaker:
This is the backup wrap up.
Speaker:
W. Curtis Preston: Welcome to the show.
Speaker:
I'm your host, W.
Speaker:
Curtis Preston, AKA, Mr.
Speaker:
Backup.
Speaker:
And with me, I have my election primary worker anxiety consultant
Speaker:
Prasanna Malaiyandi: How's it going, Curtis?
Speaker:
Yeah.
Speaker:
You're doing, it's that time of year, or I guess every couple years
Speaker:
where the election happens and
Speaker:
W. Curtis Preston: no, there's no word for like two years.
Speaker:
Is there?
Speaker:
It's a, it's a, that time of biannual, I
Prasanna Malaiyandi:
It's weird that bi counts as both, like half as well as two
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Don't get, don't get me started on English.
Prasanna Malaiyandi:
All right.
Prasanna Malaiyandi:
Um, semi or bi, right?
Prasanna Malaiyandi:
So I will once again be an election worker for the upcoming
Prasanna Malaiyandi:
California Presidential primary, and tomorrow is to set up day.
Prasanna Malaiyandi:
This year I am running an 11 day vote site.
Prasanna Malaiyandi:
Wow.
Prasanna Malaiyandi:
Crazy.
Prasanna Malaiyandi:
You'll be a busy, busy man.
Prasanna Malaiyandi:
W. Curtis Preston: I will be,
Prasanna Malaiyandi:
well, that's a different point because ask me how many people I
Prasanna Malaiyandi:
think I'll see in the first 10 days.
Prasanna Malaiyandi:
I am gonna say 21.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, I think that might be high.
Prasanna Malaiyandi:
It, uh, because what happens is everybody comes on election day.
Prasanna Malaiyandi:
I mean, I'm glad we have early voting, right?
Prasanna Malaiyandi:
I, I, I really am.
Prasanna Malaiyandi:
I, I believe in access and, and I even like the 11 day sites because
Prasanna Malaiyandi:
there are some people that have jobs that just really mess up a week.
Prasanna Malaiyandi:
So four days isn't just.
Prasanna Malaiyandi:
It just isn't enough for some
Prasanna Malaiyandi:
people.
Prasanna Malaiyandi:
I, I believe in access to elections.
Prasanna Malaiyandi:
It, it's just that, you know, everybody comes on election day and then we go,
Prasanna Malaiyandi:
yeah, well we've been here for 11 days.
Prasanna Malaiyandi:
And they're like, what?
Prasanna Malaiyandi:
I would normally go vote, like when they used to
Prasanna Malaiyandi:
have the neighborhood polling place.
Prasanna Malaiyandi:
I used to go vote in person on the day of the election.
Prasanna Malaiyandi:
I wouldn't go ahead of time.
Prasanna Malaiyandi:
I would just go like early in the morning and I'd just go be done and come back.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Uh, so we're there, uh, by, and by the time this episode airs, the
Prasanna Malaiyandi:
primary Will Al will already be over.
Prasanna Malaiyandi:
I love participating in the process and I will answer any.
Prasanna Malaiyandi:
All election questions that anybody has, and I'll say the same thing
Prasanna Malaiyandi:
that I say every time this comes up.
Prasanna Malaiyandi:
If you have any doubt as to the integrity of your election process,
Prasanna Malaiyandi:
do one of, or both of two things.
Prasanna Malaiyandi:
One, volunteer as an election worker.
Prasanna Malaiyandi:
That is you.
Prasanna Malaiyandi:
You get so much insight into the process and how it works.
Prasanna Malaiyandi:
Number two, be an observer.
Prasanna Malaiyandi:
You
Prasanna Malaiyandi:
are legally allowed to observe every single portion of you
Prasanna Malaiyandi:
the election process, right?
Prasanna Malaiyandi:
You know where, where the votes are initially cast, where they are received,
Prasanna Malaiyandi:
how they are counted, you can view the incredibly boring way in which the,
Prasanna Malaiyandi:
there is this, well, not just the accounting room,
Prasanna Malaiyandi:
but there is this process.
Prasanna Malaiyandi:
The most boring part of the process is when they do a 1% manual count.
Prasanna Malaiyandi:
So they take 1% of the cartons.
Prasanna Malaiyandi:
That, uh, you know, the, the ballot cartons that, that, that are gonna contain
Prasanna Malaiyandi:
anywhere from 20 to 200 votes, you know, and they sit there at a table with like
Prasanna Malaiyandi:
four people and they read it one by one, and then those four people tally it up.
Prasanna Malaiyandi:
And then they compare numbers and the numbers all have to match and they have
Prasanna Malaiyandi:
to match what the machine said to box it.
Prasanna Malaiyandi:
My wife has done that process, but, oh my lord.
Prasanna Malaiyandi:
It's like, it's like watching paint
Prasanna Malaiyandi:
I, I, I was just gonna think her, you're
Prasanna Malaiyandi:
probably gonna be like, okay.
Prasanna Malaiyandi:
One ballot, two ballot, three ballots, like counting sheep.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
But anyway, uh, so, you know, so I'm excited to participate in the process, but
Prasanna Malaiyandi:
I, I do have a certain amount of anxiety as I was alluding to a certain amount of
Prasanna Malaiyandi:
anxiety because there are people, right.
Prasanna Malaiyandi:
People who need people
Prasanna Malaiyandi:
to yell at.
Prasanna Malaiyandi:
I think the other thing to note is it's not like they're
Prasanna Malaiyandi:
just throwing you to the wolves, right?
Prasanna Malaiyandi:
So you go through training, right?
Prasanna Malaiyandi:
W. Curtis Preston: go through a lot of training.
Prasanna Malaiyandi:
Right, right.
Prasanna Malaiyandi:
Yeah,
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
And, and, and you know, and I'm experienced The other
Prasanna Malaiyandi:
people are experienced.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
There's no, yeah, it's not wolf throwing and,
Prasanna Malaiyandi:
and, and they have lots of support.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
So there, there, there's a, there's a, a phone number, the poll worker hotline.
Prasanna Malaiyandi:
Which I have
Prasanna Malaiyandi:
saved as a contact in my phone.
Prasanna Malaiyandi:
I just have them
Prasanna Malaiyandi:
as, I have them as R-O-V-R-O-V,
Prasanna Malaiyandi:
uh, the Registrar of Voters.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
That's their first and last name, and I just call 'em, you know?
Prasanna Malaiyandi:
yeah, but you don't need to have the experience that you did
Prasanna Malaiyandi:
because when you first started all this, you didn't have that experience either.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
You were
Prasanna Malaiyandi:
W. Curtis Preston: right, Yeah, yeah,
Prasanna Malaiyandi:
right, who was learning the ropes as well, and so you
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
So you're, you're speaking to the people that I'm saying participate?
Prasanna Malaiyandi:
Yes.
Prasanna Malaiyandi:
I I think you should participate.
Prasanna Malaiyandi:
Um, and, uh, by and large it is a very easygoing, peaceful process.
Prasanna Malaiyandi:
Every once in a while there are some challenging people
Prasanna Malaiyandi:
and, uh, you just pass those over to your more experienced
Prasanna Malaiyandi:
site manager, which is me.
Prasanna Malaiyandi:
So it's a little bit of anxiety.
Prasanna Malaiyandi:
But, um, anyway, let's get on to what you know.
Prasanna Malaiyandi:
This is part of this series that we have, uh, called Cloud Disasters, and
Prasanna Malaiyandi:
this is yet another cloud disaster.
Prasanna Malaiyandi:
The cloud is just computers that somebody else is running, and in this case it, it's
Prasanna Malaiyandi:
a database that someone else is running.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: And we get people all the time that wanna argue, oh, well
Prasanna Malaiyandi:
I don't need to back up Salesforce.
Prasanna Malaiyandi:
I don't need to back up Microsoft 365.
Prasanna Malaiyandi:
It, it is part of the service.
Prasanna Malaiyandi:
It's not
Prasanna Malaiyandi:
right.
Prasanna Malaiyandi:
Um, it's just not, it's not in your, uh, service description to, to go look at it.
Prasanna Malaiyandi:
If you don't believe me
Prasanna Malaiyandi:
And even if it was part of your service description,
Prasanna Malaiyandi:
you don't know if you could trust them
Prasanna Malaiyandi:
W. Curtis Preston: Uh,
Prasanna Malaiyandi:
yes.
Prasanna Malaiyandi:
Even if it was Yeah.
Prasanna Malaiyandi:
You know, and that, that, that's one of the stories.
Prasanna Malaiyandi:
We're gonna get to
Prasanna Malaiyandi:
that, right?
Prasanna Malaiyandi:
The, um, the O-O-O-O-V-H-O-V-H,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
The OVH story proofs.
Prasanna Malaiyandi:
Uh, we have a story, literally every comment that we, we don't make this,
Prasanna Malaiyandi:
we don't just make this stuff up.
Prasanna Malaiyandi:
You can't make this stuff up, right?
Prasanna Malaiyandi:
We, we have stories behind every one of the recommendations that
Prasanna Malaiyandi:
we make, and this series is about telling these stories and this.
Prasanna Malaiyandi:
Is a good one.
Prasanna Malaiyandi:
Do you wanna, do you wanna sort of, uh, look, first off, everybody should
Prasanna Malaiyandi:
know what Salesforce is, right?
Prasanna Malaiyandi:
Um, but you
Prasanna Malaiyandi:
know, just, just in case you don't, Salesforce is, I
Prasanna Malaiyandi:
think the OG SaaS app, right?
Prasanna Malaiyandi:
I'm sure there was another before, but they were the, the first one
Prasanna Malaiyandi:
that really took off their actual phone number is one 800 no software.
Prasanna Malaiyandi:
I don't know if you, if you knew
Prasanna Malaiyandi:
No, I didn't know
Prasanna Malaiyandi:
W. Curtis Preston: um, yeah, I, that's, that.
Prasanna Malaiyandi:
I remember that from, from many days gone by.
Prasanna Malaiyandi:
And they are A-C-R-M-A customer relationship management software.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And I remember using one of their competitors back in the day
Prasanna Malaiyandi:
when I had my own company and.
Prasanna Malaiyandi:
Uh, oh my Lord.
Prasanna Malaiyandi:
Is it so much easier to use Salesforce, especially when
Prasanna Malaiyandi:
you have multiple salespeople
Prasanna Malaiyandi:
that are, um, you know, all interacting with a variety of leads?
Prasanna Malaiyandi:
And again, to preface this story, I'm gonna explain how this works in a big org.
Prasanna Malaiyandi:
I've been a salesperson and most salespeople are, uh, commissioned.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
And they are, they're gonna attack any lead that you give them.
Prasanna Malaiyandi:
And they, because they're commissioned to do so,
Prasanna Malaiyandi:
and the only thing that prevents them, you know, you, you give these leads to
Prasanna Malaiyandi:
this person, these leads to this person.
Prasanna Malaiyandi:
And the only thing that prevents Steve from jumping all over
Prasanna Malaiyandi:
Janet's leads is permissions.
Prasanna Malaiyandi:
In a large database like Salesforce, you assign permissions, you create
Prasanna Malaiyandi:
groups of leads, and you give permission to Steve or to a certain team.
Prasanna Malaiyandi:
There's different ways to do it, but you divvy out these leads.
Prasanna Malaiyandi:
By way of permissions.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
And that would, that prevents Steve from going over and, you know,
Prasanna Malaiyandi:
stomping all over, uh, Janet's leads
Prasanna Malaiyandi:
and, um, uh, but then something happened.
Prasanna Malaiyandi:
So why don't you, so when, when, when did this happen and what happened?
Prasanna Malaiyandi:
Prasanna Malaiyandi: So it was back in 2019.
Prasanna Malaiyandi:
So it was a ways ago, and what ended up happening is Salesforce ran a script.
Prasanna Malaiyandi:
And what the script did is it allowed everyone in an organization
Prasanna Malaiyandi:
to be able to modify and access all records in that organization.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And so in your example of Steve and Janet, Steve could
Prasanna Malaiyandi:
now see everything Janet had could go stomp on it and be like, Hey, by the way,
Prasanna Malaiyandi:
Janet, you're actually not as far along as you said you were, or change a dollar
Prasanna Malaiyandi:
amount of the lead and other things like
Prasanna Malaiyandi:
W. Curtis Preston: Or Steve could also delete all of
Prasanna Malaiyandi:
Janet's leads, if that's what, if
Prasanna Malaiyandi:
Steve is a very bad person, if he wanted to go delete all her leads or just
Prasanna Malaiyandi:
delete, you know, uh, like any interaction that she had with the clients, right?
Prasanna Malaiyandi:
If
Prasanna Malaiyandi:
you were.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: A nefarious person.
Prasanna Malaiyandi:
Yeah, the notes, right?
Prasanna Malaiyandi:
If you were a nefarious person, you could have done a lot of damage to
Prasanna Malaiyandi:
other people in the organization, uh, or you could steal their leads.
Prasanna Malaiyandi:
Just
Prasanna Malaiyandi:
reassign those leads
Prasanna Malaiyandi:
to you.
Prasanna Malaiyandi:
Um,
Prasanna Malaiyandi:
or you could be doing just some random housekeeping,
Prasanna Malaiyandi:
innocuous housekeeping stuff like, Hey, I'm just gonna go clear out all my old
Prasanna Malaiyandi:
leads older than like two years old that I haven't touched and realize that
Prasanna Malaiyandi:
you might be stomping on Janet's leads.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, exactly.
Prasanna Malaiyandi:
Um, so Salesforce, interestingly enough, Salesforce, according to, and we're
Prasanna Malaiyandi:
gonna put links to this, we have the, a link to the original post that was made
Prasanna Malaiyandi:
by Salesforce, as well as a link to a follow-up post that they made several
Prasanna Malaiyandi:
months later as a, uh, postmortem.
Prasanna Malaiyandi:
What they learned, but what we know from their posts is they did not
Prasanna Malaiyandi:
notice that they had done this.
Prasanna Malaiyandi:
A customer called and said, Hey, this is odd.
Prasanna Malaiyandi:
Everybody apparently can modify everybody's leads.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
And, and by the way, just, just to put a, a point on that, it, it's
Prasanna Malaiyandi:
sort of like in file permissions.
Prasanna Malaiyandi:
If you have modified permissions, you have all of the others,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
Uh, you have read, write, you know, modified delete.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So, um, the, and by the way, they, they appear, it, it appears that
Prasanna Malaiyandi:
they only had this privilege.
Prasanna Malaiyandi:
To records.
Prasanna Malaiyandi:
They didn't have the, they at least didn't grant this permission to be able to
Prasanna Malaiyandi:
modify things like configurations, right?
Prasanna Malaiyandi:
So they couldn't go in and basically delete Janet or change Janet's permissions
Prasanna Malaiyandi:
as a person, as a user, but they could go in and access and do everything.
Prasanna Malaiyandi:
To her data.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So it's important to, to just mention that.
Prasanna Malaiyandi:
Anyway, so they didn't notice that they did this.
Prasanna Malaiyandi:
Customers called in and then they very quickly, uh, they had,
Prasanna Malaiyandi:
you know, what I would call, you know, an oh shit moment, right?
Prasanna Malaiyandi:
And they're like, holy cow.
Prasanna Malaiyandi:
That maintenance script that we ran, it appears that it did.
Prasanna Malaiyandi:
Um, you know.
Prasanna Malaiyandi:
A
Prasanna Malaiyandi:
W. Curtis Preston: Uh, A lot more than we had intended to do, and they
Prasanna Malaiyandi:
realized they had really messed up.
Prasanna Malaiyandi:
And so the first thing they did was just say, okay, just shut down everything.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, which, which I think was probably the best thing they could do at the
Prasanna Malaiyandi:
time, even though that would, of course immediately at Cal, all their customers.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Well I do wonder if I agree that that's sort of like the nuclear option, right?
Prasanna Malaiyandi:
But I do wonder if maybe they could have at least, uh, I guess I was just thinking
Prasanna Malaiyandi:
could they have removed the modify all and just given like view only, but then
Prasanna Malaiyandi:
some orgs, it still might have been bad to allow Steve to see Janet's leads
Prasanna Malaiyandi:
W. Curtis Preston: well, I, I, the problem was, I, I don't think, you
Prasanna Malaiyandi:
know, based on the, the, the records and stuff that we have, I'm not sure
Prasanna Malaiyandi:
they even know the, knew the extent
Prasanna Malaiyandi:
of the damage that they had caused
Prasanna Malaiyandi:
it's like a ransomware attack where
Prasanna Malaiyandi:
you just pulled a network.
Prasanna Malaiyandi:
Cable.
Prasanna Malaiyandi:
W. Curtis Preston: Exactly.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Um, there's a man, there's a great scene in, in, you know, one of my
Prasanna Malaiyandi:
favorite shows, alias, where, uh, he goes running into the server room
Prasanna Malaiyandi:
and he literally is like flipping
Prasanna Malaiyandi:
power switches, you know, they're downloading all the files
Prasanna Malaiyandi:
up the server and he is just flipping all the power switches.
Prasanna Malaiyandi:
Uh, it, it was pretty much like that.
Prasanna Malaiyandi:
And so they, they, um.
Prasanna Malaiyandi:
Uh, that was their first response.
Prasanna Malaiyandi:
And then, uh, then what did it do?
Prasanna Malaiyandi:
So then after that, well, so that shut
Prasanna Malaiyandi:
it down, but it was only for the organizations that were impacted.
Prasanna Malaiyandi:
And
Prasanna Malaiyandi:
W. Curtis Preston: by by the way, I just want to interject.
Prasanna Malaiyandi:
What we now know is that the, IM, the organizations that were impacted
Prasanna Malaiyandi:
was any user or any organization that had used Pardot, which is their.
Prasanna Malaiyandi:
Marketing automation.
Prasanna Malaiyandi:
, it's the Salesforce equivalent to Marketo.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So this is the thing that's gonna email your customers
Prasanna Malaiyandi:
and things like that, right?
Prasanna Malaiyandi:
Um, so anyone who had ever used or was currently using Pardot, that
Prasanna Malaiyandi:
turns out to be, who was impacted?
Prasanna Malaiyandi:
So after that, let's see, what did they do?
Prasanna Malaiyandi:
W. Curtis Preston: I'm not exactly sure exactly when, what happened
Prasanna Malaiyandi:
'cause we don't exactly have a timeline 'cause there's multiple
Prasanna Malaiyandi:
posts and multiple articles and,
Prasanna Malaiyandi:
you know, we even have a, there's a stack exchange thread that we could
Prasanna Malaiyandi:
follow during this, uh, uh, outage.
Prasanna Malaiyandi:
Yeah, even though we don't know that timeline, Curtis,
Prasanna Malaiyandi:
I think the one thing we can just sort of take away from all these articles
Prasanna Malaiyandi:
is they did try to fix it themselves.
Prasanna Malaiyandi:
They weren't like, Hey users, we have nothing to do.
Prasanna Malaiyandi:
Good luck.
Prasanna Malaiyandi:
Go pound sand.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
It looks like they were internally trying to do things to fix this and
Prasanna Malaiyandi:
looking at various technologies or resources that they might have had, but.
Prasanna Malaiyandi:
As we know, that takes time.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
And as a user, at that time, they weren't really forthcoming about, Hey, we're
Prasanna Malaiyandi:
trying things internally either, right?
Prasanna Malaiyandi:
They were, they didn't wanna give users hope.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, well, I'm not sure if they were communicating,
Prasanna Malaiyandi:
you know, it does show that they mass emailed some users.
Prasanna Malaiyandi:
All we have access to is what they said publicly and publicly.
Prasanna Malaiyandi:
Again, around this time they had this post where they said, Hey, we messed up.
Prasanna Malaiyandi:
We gave modify all.
Prasanna Malaiyandi:
And here are a couple of ways that you can potentially fix this if
Prasanna Malaiyandi:
you want to fix this yourself.
Prasanna Malaiyandi:
And, 'cause that was obviously a question that people asked is, Hey,
Prasanna Malaiyandi:
can I, can I fix this on my account so that I can get my account back online?
Prasanna Malaiyandi:
And the you, you know, that the response just really infuriates me
Prasanna Malaiyandi:
because I gave them two options.
Prasanna Malaiyandi:
And they had to do with the sandbox.
Prasanna Malaiyandi:
They basically said, if, you know, if you made a sandbox, which is something
Prasanna Malaiyandi:
that you could do regularly, which if you don't know a sandbox is, uh,
Prasanna Malaiyandi:
you know, it's a place where you can play with your data and, and,
Prasanna Malaiyandi:
mess
Prasanna Malaiyandi:
a clone,
Prasanna Malaiyandi:
W. Curtis Preston: then It's fine.
Prasanna Malaiyandi:
It's what?
Prasanna Malaiyandi:
It's like a clone.
Prasanna Malaiyandi:
W. Curtis Preston: It is a, yeah, it's a clone that you can
Prasanna Malaiyandi:
automatically make with Salesforce.
Prasanna Malaiyandi:
You, you know, it's, some people actually treat it like a backup.
Prasanna Malaiyandi:
I don't because, uh, it's all in the same place, but, so it doesn't
Prasanna Malaiyandi:
conform to the 3, 2, 1 rule.
Prasanna Malaiyandi:
But it, um, the, but they said, you know, if you happen to have a recent backup.
Prasanna Malaiyandi:
You could go and get their permissions from there.
Prasanna Malaiyandi:
'cause remember, they're, they don't have to restore the data.
Prasanna Malaiyandi:
They didn't mess up the data.
Prasanna Malaiyandi:
They messed up the permissions of the, of the data of the
Prasanna Malaiyandi:
And just to correct you, you meant
Prasanna Malaiyandi:
to say sandbox not backup in
Prasanna Malaiyandi:
that statement, correct?
Prasanna Malaiyandi:
W. Curtis Preston: oh, did I, did
Prasanna Malaiyandi:
I say, did I call
Prasanna Malaiyandi:
Ouch.
Prasanna Malaiyandi:
You are correct.
Prasanna Malaiyandi:
I meant to say samples.
Prasanna Malaiyandi:
So if they, if they're saying if you happen to have a recent
Prasanna Malaiyandi:
sandbox, copy of your instance.
Prasanna Malaiyandi:
It's recent.
Prasanna Malaiyandi:
This is the problem.
Prasanna Malaiyandi:
It needs to be recent enough to have the user's permissions to match your current
Prasanna Malaiyandi:
permissions, but it can't be too recent because if it was too recent, in other
Prasanna Malaiyandi:
words, if it was made in the last few hours, it's just a backup of our mistake.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
It's just, it is just a copy of our mistake.
Prasanna Malaiyandi:
So they were saying that what, what infuriates me persona is.
Prasanna Malaiyandi:
Not once did in, in, in, in in any of the external, uh,
Prasanna Malaiyandi:
stuff that Salesforce put out.
Prasanna Malaiyandi:
Not once did they say, by the way, if per chance you did what Curtis tells you to do
Prasanna Malaiyandi:
and actually backed up your data.
Prasanna Malaiyandi:
W you could just go and, and easily restore the, basically
Prasanna Malaiyandi:
the, the user's table is
Prasanna Malaiyandi:
what, you know, for those of you that don't know, you know, Salesforce has,
Prasanna Malaiyandi:
you know, all these different tables.
Prasanna Malaiyandi:
It's like any other database.
Prasanna Malaiyandi:
They call them objects.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
So it would've been the user's object, uh, is what I would assume was
Prasanna Malaiyandi:
that.
Prasanna Malaiyandi:
needed to be restored and you could restore the, just restore your user's
Prasanna Malaiyandi:
object to any time before, you know, 12:35 AM on May 17th, 2019, and you'll
Prasanna Malaiyandi:
be
Prasanna Malaiyandi:
you'd be fine.
Prasanna Malaiyandi:
W. Curtis Preston: But they never said that.
Prasanna Malaiyandi:
I, I just.
Prasanna Malaiyandi:
That I remember posting a blog at the time that basically said Salesforce
Prasanna Malaiyandi:
proves they know nothing about backup,
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: right?
Prasanna Malaiyandi:
Because it's like they never once suggested they, they sort of thought
Prasanna Malaiyandi:
of the, of the sandbox as a backup and never thought that anybody might
Prasanna Malaiyandi:
want to have backed up their, their
Prasanna Malaiyandi:
Of course not.
Prasanna Malaiyandi:
'cause who does backups of Salesforce?
Prasanna Malaiyandi:
You don't need no stinking backups.
Prasanna Malaiyandi:
W. Curtis Preston: Nice.
Prasanna Malaiyandi:
Nice.
Prasanna Malaiyandi:
Uh, all right.
Prasanna Malaiyandi:
Uh, um, brownie points or extra points, if you can tell me what
Prasanna Malaiyandi:
movie that is referring to.
Prasanna Malaiyandi:
And I mean, the original movie, not the second movie that,
Prasanna Malaiyandi:
Not the Rob Schneider one
Prasanna Malaiyandi:
W. Curtis Preston: oh, that would be a third movie.
Prasanna Malaiyandi:
Prasanna Malaiyandi: because of what that was.
Prasanna Malaiyandi:
Water Boy
Prasanna Malaiyandi:
W. Curtis Preston: What's it?
Prasanna Malaiyandi:
I, no.
Prasanna Malaiyandi:
Yeah, I don't know.
Prasanna Malaiyandi:
I'm referring to the original movie starring Humphrey Bogart.
Prasanna Malaiyandi:
It's called The Treasure of Sierra Madre.
Prasanna Malaiyandi:
Oh,
Prasanna Malaiyandi:
W. Curtis Preston: The, the badges.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
We, we don't,
Prasanna Malaiyandi:
we don't know nothing about no stinking badges.
Prasanna Malaiyandi:
Great greatvine.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Anyway, um, the.
Prasanna Malaiyandi:
Why don't you read this, uh, this part about the, the stack exchange part there,
Prasanna Malaiyandi:
there's a, there's an interesting comment on the stack exchange, uh, thread there.
Prasanna Malaiyandi:
Do you see that?
Prasanna Malaiyandi:
Is this is the, that's not even the worst that is going.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So, yeah, so on Stack Exchange, one of the users commented.
Prasanna Malaiyandi:
That's not even the worst that is going on.
Prasanna Malaiyandi:
Apparently in an attempt to fix this, they remove the modify access all data
Prasanna Malaiyandi:
from all admin profiles in some instances, including standard and custom profiles.
Prasanna Malaiyandi:
W. Curtis Preston: So.
Prasanna Malaiyandi:
so they removed the, they removed the permission even from the
Prasanna Malaiyandi:
people that needed the permission.
Prasanna Malaiyandi:
Which basically means do you end up with a read-only
Prasanna Malaiyandi:
copy of your data while they're trying to figure things out.
Prasanna Malaiyandi:
Well, because I could see that they don't want you to change anything
Prasanna Malaiyandi:
because it might not let them restore things back to a good state later on.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
You know, in that, in that Stack Exchange thread, uh, which we'll put a link to
Prasanna Malaiyandi:
it in, in the show notes, in that stack Exchange thread, they were saying that as
Prasanna Malaiyandi:
this was going on, Salesforce was saying, please don't try to fix this yourself.
Prasanna Malaiyandi:
We, we got it.
Prasanna Malaiyandi:
Like we're gonna, we think, we think we can fix it.
Prasanna Malaiyandi:
Um, so let's talk about some of the things that they did.
Prasanna Malaiyandi:
Uh, you know, in the backend, and by the way, this is all news to me.
Prasanna Malaiyandi:
This was not covered in the original stories that covered this.
Prasanna Malaiyandi:
This was, you know, in classic, you know, news stuff.
Prasanna Malaiyandi:
They only covered that initial explosion.
Prasanna Malaiyandi:
No one
Prasanna Malaiyandi:
covers the, the remediation and everything afterwards,
Prasanna Malaiyandi:
especially given that this was,
Prasanna Malaiyandi:
It's a not sexy stuff, right?
Prasanna Malaiyandi:
W. Curtis Preston: It's not as sexy.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
If it
Prasanna Malaiyandi:
bleeds, it leads.
Prasanna Malaiyandi:
And this is, you know,
Prasanna Malaiyandi:
uh, so about seven months later, so this was May and in January
Prasanna Malaiyandi:
of, of 2020, um, oh, I just, I just realized like timeframe.
Prasanna Malaiyandi:
You, like this is a group of people that are writing, they're
Prasanna Malaiyandi:
just, they're just, they're just writing about this, this problem
Prasanna Malaiyandi:
that happened in, in, in last year.
Prasanna Malaiyandi:
Little did they know in two months that the world was gonna fall
Prasanna Malaiyandi:
Yeah, so the first thing that they.
Prasanna Malaiyandi:
Yeah, so the first thing they attempted to do was run a backout script.
Prasanna Malaiyandi:
They were like, Hey, we had a script that ran.
Prasanna Malaiyandi:
We should just go undo it.
Prasanna Malaiyandi:
Which in my mind makes total sense, right?
Prasanna Malaiyandi:
You're like, Hey, the script did something.
Prasanna Malaiyandi:
Let me just go undo everything that I just did.
Prasanna Malaiyandi:
W. Curtis Preston: and it looks like the, and it looks like the script had
Prasanna Malaiyandi:
automatically, it basically, it, it made a backup of the permissions that.
Prasanna Malaiyandi:
It was supposed to change.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Which is what a script should
Prasanna Malaiyandi:
do.
Prasanna Malaiyandi:
right before I go do a bunch of stuff.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So why
Prasanna Malaiyandi:
didn't that work?
Prasanna Malaiyandi:
the problem is, it did not, however, record
Prasanna Malaiyandi:
things that it had done that it wasn't supposed to have done.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
So the, so the backup, the backup line didn't have the wild, the, the
Prasanna Malaiyandi:
asterisk in it and the, and the execution line did,
Prasanna Malaiyandi:
uh, oh.
Prasanna Malaiyandi:
That's, I, you know, I'm sorry.
Prasanna Malaiyandi:
. I'm having, uh, shun Freud at the expense of this poor person
Prasanna Malaiyandi:
who, you know, according to the, to the aftermath and the report.
Prasanna Malaiyandi:
You know, they, they said, did we follow our process?
Prasanna Malaiyandi:
They did follow their processes.
Prasanna Malaiyandi:
They did, um, uh, or most of their processes.
Prasanna Malaiyandi:
What they didn't do when they did the initial script run that, that
Prasanna Malaiyandi:
did all of this, they tested it.
Prasanna Malaiyandi:
But what they didn't do was they didn't do a phased rollout.
Prasanna Malaiyandi:
Of the script.
Prasanna Malaiyandi:
They were like,
Prasanna Malaiyandi:
we got it You know, this guy wrote it, this person, uh, you know, uh,
Prasanna Malaiyandi:
you know, sanctioned the script.
Prasanna Malaiyandi:
We've tested the script, the script runs, run it everywhere.
Prasanna Malaiyandi:
Did they do it on a Friday evening
Prasanna Malaiyandi:
W. Curtis Preston: Let's see.
Prasanna Malaiyandi:
17th, 2019.
Prasanna Malaiyandi:
It was a Friday.
Prasanna Malaiyandi:
They did it on Friday.
Prasanna Malaiyandi:
Oh, those poor guys, you know, they didn't have a weekend.
Prasanna Malaiyandi:
So they did have a variety of technologies that they could possibly use.
Prasanna Malaiyandi:
To solve this problem.
Prasanna Malaiyandi:
And one of them was that they have a Dr instance.
Prasanna Malaiyandi:
We talk about this with 365 as well, because we know that 365
Prasanna Malaiyandi:
has a rolling, um, you know, uh, replicated copy of their system, right?
Prasanna Malaiyandi:
So, so this is, again, this is a quote from their report.
Prasanna Malaiyandi:
A site switch to a DR instance was not an option since the purpose of
Prasanna Malaiyandi:
the DR option is to replicate it near real time the state of the primary
Prasanna Malaiyandi:
site, which meant that the inadvertent
Prasanna Malaiyandi:
permission change would've been replicated in near real time to the redundant site.
Prasanna Malaiyandi:
We talk about this, don't we?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Dr.
Prasanna Malaiyandi:
Isn't intended to be a backup.
Prasanna Malaiyandi:
That is not its purpose.
Prasanna Malaiyandi:
W. Curtis Preston: yeah.
Prasanna Malaiyandi:
Well, I would say.
Prasanna Malaiyandi:
Replication, but like,
Prasanna Malaiyandi:
because we've talked about this in previous episode replication,
Prasanna Malaiyandi:
which is what they're using by itself is not, is not a backup.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Because it, you know, you know, as I, as I jokingly say, it
Prasanna Malaiyandi:
makes a mistakes more efficient.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And that's what they, that's
Prasanna Malaiyandi:
what they're saying here is, yeah, it would've, they're like, well, we knew.
Prasanna Malaiyandi:
We knew we couldn't use that.
Prasanna Malaiyandi:
, and then there was a, another thing that they talked about called flashback.
Prasanna Malaiyandi:
You wanna talk about that?
Prasanna Malaiyandi:
yeah.
Prasanna Malaiyandi:
So flashback.
Prasanna Malaiyandi:
Their database vendor has this technology which allows you to
Prasanna Malaiyandi:
sort of keep a point in time of the database automatically in the system.
Prasanna Malaiyandi:
And so you could use that to restore from in case something happens.
Prasanna Malaiyandi:
Now, the one challenge though is they did look to see can we go use our
Prasanna Malaiyandi:
flashback area to restore the database,
Prasanna Malaiyandi:
get everything up and running again.
Prasanna Malaiyandi:
Unfortunately, they only kept six hours.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
And so that's the furthest back they could run because that makes sense.
Prasanna Malaiyandi:
You have some sort of database corruption or you accidentally drop
Prasanna Malaiyandi:
a table, you just need to go back a couple seconds, you're good to go.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, they don't say it, but it looks like they're
Prasanna Malaiyandi:
referring to an Oracle feature.
Prasanna Malaiyandi:
Uh, and
Prasanna Malaiyandi:
there are a series of features there, but it's kind of like
Prasanna Malaiyandi:
the, the snapshot thing, right?
Prasanna Malaiyandi:
You can go back to when you took the snapshot, but if you, you know,
Prasanna Malaiyandi:
if you want to go longer than that.
Prasanna Malaiyandi:
You don't have, because there's a, there's a window that they, that
Prasanna Malaiyandi:
they specify and six hours must
Prasanna Malaiyandi:
have been the window.
Prasanna Malaiyandi:
And there were pa they were past the window by the time they, they optioned.
Prasanna Malaiyandi:
So it's interesting the, the option that they chose to.
Prasanna Malaiyandi:
To figure out what permissions were, what to be able to, you know, to restore them.
Prasanna Malaiyandi:
'cause the problem, once you've granted modify all, well, how
Prasanna Malaiyandi:
do you know what to go back to?
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
You, you can't just say read all right.
Prasanna Malaiyandi:
Uh, 'cause even that, right.
Prasanna Malaiyandi:
You know.
Prasanna Malaiyandi:
Um, so the, what they started doing is essentially log scraping, right?
Prasanna Malaiyandi:
They call it log mining.
Prasanna Malaiyandi:
To, to, to look at customers instances of.
Prasanna Malaiyandi:
To see what permissions in the logs that these things were set
Prasanna Malaiyandi:
to, and that's what they began.
Prasanna Malaiyandi:
And they started going through and in, in the story, in the, in the
Prasanna Malaiyandi:
postmortem, there are these series of.
Prasanna Malaiyandi:
We think we did this, we think we did that.
Prasanna Malaiyandi:
You know, we've, we've restored all these instances.
Prasanna Malaiyandi:
There were, there were dozens of instances that were affected and they're
Prasanna Malaiyandi:
like, we think this instance is good.
Prasanna Malaiyandi:
This instance is good.
Prasanna Malaiyandi:
So if you're on that instance, then you're good.
Prasanna Malaiyandi:
But even when all that was done, there were still customers
Prasanna Malaiyandi:
that were not restored.
Prasanna Malaiyandi:
and and they said, you know, we're working with you.
Prasanna Malaiyandi:
And then they gave instructions on how to basically manually fix this and
Prasanna Malaiyandi:
Which if you have thousands of salespeople
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
Um, you know what would've been really helpful to those customers
Prasanna Malaiyandi:
in this scenario, persona.
Prasanna Malaiyandi:
using another vendor.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, a backup, right?
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
If they, if they had backed up the data outta Salesforce into another vendor,
Prasanna Malaiyandi:
so, so companies you know, that use Salesforce and other CRM products and if
Prasanna Malaiyandi:
they used a backup, they would've been able to fix this literally like that.
Prasanna Malaiyandi:
Uh, well, a backup intended for Salesforce
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, so back well backup of Salesforce,
Prasanna Malaiyandi:
intended for Salesforce.
Prasanna Malaiyandi:
I'm not sure what
Prasanna Malaiyandi:
other, what other method you
Prasanna Malaiyandi:
No.
Prasanna Malaiyandi:
So I was thinking like someone could have done a backup by
Prasanna Malaiyandi:
just dumping out the objects.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And I don't know if that would've necessarily kept all the permissions
Prasanna Malaiyandi:
as well if they hadn't backed up the, or dumped the user table as well.
Prasanna Malaiyandi:
W. Curtis Preston: Well, if you do, if you can do a manual backup of Salesforce,
Prasanna Malaiyandi:
it basically gives you all the objects.
Prasanna Malaiyandi:
The only problem with every manual backup is you have to manually do it.
Prasanna Malaiyandi:
You have to do it every
Prasanna Malaiyandi:
once in a while, which means, I don't know how often you're gonna be doing it.
Prasanna Malaiyandi:
It might be once a week.
Prasanna Malaiyandi:
It still would be better than nothing,
Prasanna Malaiyandi:
Nothing.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: you basically, it just means your permissions
Prasanna Malaiyandi:
would be a week old, which in this case would be a good thing.
Prasanna Malaiyandi:
Um, but it just drives home again to me that there are, you know, you, you've
Prasanna Malaiyandi:
heard me say this, I think I said it in the last episode of like, there, there
Prasanna Malaiyandi:
is more in heaven and earth, Horatio than dreamt of in your philosophy, there are
Prasanna Malaiyandi:
things that can happen to you in the cloud.
Prasanna Malaiyandi:
The cloud isn't magic.
Prasanna Malaiyandi:
There are things that, that you're not gonna
Prasanna Malaiyandi:
anticipate.
Prasanna Malaiyandi:
be magical.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, it is just as magical as actual magic, which
Prasanna Malaiyandi:
is an illusion.
Prasanna Malaiyandi:
So I love magic by the way.
Prasanna Malaiyandi:
I'm a big fan.
Prasanna Malaiyandi:
Like, I went and saw, like, I saw David Copperfield in Vegas, and I was amazed.
Prasanna Malaiyandi:
I, I loved it, but inside I knew it was all just an illusion.
Prasanna Malaiyandi:
Illusion.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: Um, so, you know, my advice isn't so much
Prasanna Malaiyandi:
to Salesforce, Salesforce.
Prasanna Malaiyandi:
Did as much as they could do in this scenario.
Prasanna Malaiyandi:
It seems like they were communicating with their users.
Prasanna Malaiyandi:
They had a status page like we, like we tell them to do.
Prasanna Malaiyandi:
Um, maybe go.
Prasanna Malaiyandi:
I, I think the only thing I would fault them for is
Prasanna Malaiyandi:
maybe a bit more communication about what they're doing internally, right.
Prasanna Malaiyandi:
W. Curtis Preston: again, I think they may have been doing that
Prasanna Malaiyandi:
just not publicly the way we were
Prasanna Malaiyandi:
looking.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
So maybe they were
Prasanna Malaiyandi:
communicating
Prasanna Malaiyandi:
privately
Prasanna Malaiyandi:
'cause ' cause they said, they said in their.
Prasanna Malaiyandi:
In their postmortem, they're like, what?
Prasanna Malaiyandi:
What should we have done better?
Prasanna Malaiyandi:
And, um, they listed a whole bunch of things they were
Prasanna Malaiyandi:
doing to communicate, uh, what
Prasanna Malaiyandi:
was going on.
Prasanna Malaiyandi:
We just outside.
Prasanna Malaiyandi:
I was really angry at the time because all I saw was that one page,
Prasanna Malaiyandi:
because I saw the one page where they said, Hey, uh, sorry, um, we
Prasanna Malaiyandi:
just messed up all your permissions.
Prasanna Malaiyandi:
And so.
Prasanna Malaiyandi:
Can you fix it?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: And, and by the way, we're not gonna mention backup.
Prasanna Malaiyandi:
I was furious at the time.
Prasanna Malaiyandi:
Uh, so I'm, I'm a little less furious.
Prasanna Malaiyandi:
But, uh, this is just another big example of why we back up, you know,
Prasanna Malaiyandi:
everything and why we recommend backing up cloud vendors and es
Prasanna Malaiyandi:
esp and especially SaaS vendors.
Prasanna Malaiyandi:
Yeah,
Prasanna Malaiyandi:
W. Curtis Preston: Any, any
Prasanna Malaiyandi:
final thoughts from you from Peanut Gallery?
Prasanna Malaiyandi:
I think that's the right thing.
Prasanna Malaiyandi:
They did everything they could and users should have backed up their data,
Prasanna Malaiyandi:
W. Curtis Preston: again, we're blaming the victims.
Prasanna Malaiyandi:
Uh oh goodness.
Prasanna Malaiyandi:
could have easily been avoided though.
Prasanna Malaiyandi:
W. Curtis Preston: It could have, it could have back it up or give it up people.
Prasanna Malaiyandi:
Uh, thanks for listening.
Prasanna Malaiyandi:
You know, you are why we do this.
Prasanna Malaiyandi:
We want to turn you into Cyber recovery Heroes.
Prasanna Malaiyandi:
That's a wrap
