Building a Cybersecurity Culture in Your Company (Encore)

You probably already own the cybersecurity tools that you need.

That's not the issue.

The issue is that nobody's using them, or certainly not using them correctly.

Today's guest built an entire cybersecurity culture out of that

one realization, one week at a time.

This is the final episode in our Encore series, where we picked the

episodes over the last few years that our listeners really engaged with.

We found this guest schooling people on Reddit with a series of posts, uh,

on what he called a security cadence.

His pitch was to skip the new six-figure product and instead commit

to one small, deliberate security change, uh, and improvement per week.

We get into how that habit turned into a real culture shift at his company.

Plus, of course, we talk about things like MFA, SIM hijacking,

and why he thinks that chasing the latest zero day misses the point.

He goes by his Reddit username of Snorkel42, and he clearly knows his stuff.

If this is your first time watching or listening to me, hi, I'm W.

Curtis Preston, AKA Mr. Backup.

I've been obsessing about backup, recovery, and now cyber

recovery for over 30 years.

If that's your bag, then I'm your guy.

You're not gonna find anyone that cares about that topic more than me.

Ever since 1993 when I had to tell my boss that there were no backups of

the database that we had just lost.

Now I've written five O'Reilly books, a blog, and a podcast.

Here we turn unappreciated admins into cyber recovery heroes.

This is the Backup Wrap Up

Hi, and welcome to Backup Central's podcast.

I'm your host, W. Curtis Preston, AKA Mr. Backup.

I have with me, my ghee deployment consultant, Prasanna Malaiyandi.

uh, Curtis, how's your ghee going?

You know, you may recall it a month or so ago.

I got ghee for the first time.

And, you know, for those that don't know what ghee is, it's

clarified butter specifically.

It's apparently an Indian thing, right?

we did learn that ghee is from Sanskrit.

That means sprinkled, which is interesting.

But the, um, and the thing about it is that it can sit on the counter.

Um, like it's shelf stable so it can sit on the counter and

you made a comment by the way.

I know this because I'm literally editing this episode right now.

And you made a comment that, you know, it can sit there probably for a

couple of months or until you run out.

So I'll just say this, the, uh, the jar, I thought that I was going to

be the only weirdo using the ghee.

Yeah, I am not the only weirdo using ghee.

In fact, if anything, the rest of the house is used the

ghee much more than I have.

And that jar that we bought is close to gone.

So

it's convenient.

It's super easy.

You just keep it out.

You warm it up a little.

It becomes really liquidy.

You put it on warm bread.

You toast that.

super good.

You can also put it in rice.

Warm rice.

It's really good.

Yeah, well, I mean, it's like it's butter, right?

you know, all the places you can put butter, you can put ghee.

Um, it's just, it, it, it is interesting for those that have never had it.

It has a slightly different flavor than butter, but it, you know, so there

is a, there is a, I didn't know what my mouth was going, what was going

to happen when I put it in there,

That's funny because I never think about that.

Like to me, like butter and ghee it's like, ah, yeah.

Yeah.

The first time I had, I had ghee, I remember going, huh,

this tastes different, you know?

Um, but what it was, but it was, but it was yummy, you know, so we continue to,

but yeah, I think that jar is almost gone.

So we're going to have to, we're going to have to find the ghee at Costco, which is

like 64 ounce.

like one pound.

Yeah, one pound jars or something.

Well we have a, a back by popular demand guest here.

Uh, he was on the podcast before and.

Is the author of the security cadence series on Reddit, been in IT for about

25 years and in InfoSec about 20 years.

And he is quite the celebrity over there on Reddit because I, you know,

his posts have been incredibly popular with, uh, uh, he's got a, uh, a, uh,

what a, what do they call it over there?

The karma of 35,000, which, you know, if you don't know anything

about Reddit, that's a, BFD, I'll just say that right now.

And we had him previously on the podcast so if you haven't heard that podcast,

you should totally listen to that.

Welcome back to the podcast.

snorkel42.

Good to be back.

And I tell you, so last time I learned what karma meant on reddit.

time I learned what ghee is so good for me.

So, um,

I want to hear about this thing.

You mentioned about security cadence.

What, what started, because that was, that was the, um, that was

the, what, what do you call that?

Uh, the thing before the thing, the, the precursor, the preamble,

the, in the title, um, uh, to the post that we saw that.

So where did that term come from?

So the term came from a previous employer.

I worked at where I was a network engineer, um, and it was a large company

that did not have an InfoSec presence.

There was no InfoSec team.

It was just kind of considered, Hey, all engineers are responsible for security.

Um, and you know, we would, we'd have our occasional shots off the bow in terms of

security, you know, problems or issues.

You know, I, I hesitate to say breaches, but, you know, incidents and when

they would occur and we would all pile into a conference room and we would

talk about what happened and what we should have done to prevent it.

And it would always come down to, well, if we just bought six figure dollar

product X, this would not have occurred.

Meanwhile, in the data center, there were piles of six-figure products

that were completely ignored because those products never run themselves.

And if you're complaining of not having resources to run those products, what

makes you think you're going to have resources to buy and run new products?

So I finally got fed up one day and just challenged the team to, you know what?

We've got plenty of tooling.

That's not the issue.

We just don't have the oomph.

We don't have the motivation to actually use it.

So why don't we set ourselves a goal for six months, we're going to

implement a security change every week.

It could be a big change, could be a really minor change, but

there's going to be something.

Move the ball forward.

And also during that six months, we are going to just put the

kibosh on talking to any vendors.

So no products are allowed, no solutions are allowed that we don't already own.

Um, and you know, it took some, took some talking and wrangling

people, but eventually folks fell in line and we sat down.

We'd wrote out a list of, you know, here's some things just off the top of our head

that we know we could implement with, you know, very little roadblocks, no impact.

And we just started and we called it our security cadence.

Once a week, we have a security cadence of releasing a security update and

it, you know, it ended around six months in our CIO came to us and said,

listen to what you guys are doing is fantastic, but please don't restrict

yourself to only free solutions.

Like we have money.

If there's stuff you need to keep this ball moving, please ask.

Um, so, you know, it kind of turned the whole thing on its head of,

you know, securing the company.

But also we were no longer begging for resources.

The, you know, the executive leadership was asking us, was begging us to start

spending resources because they saw what we were doing and saw the value.

So, could you talk a little bit, I'm sure at the very beginning

as you're starting this right.

With any new process or new, any new endeavor, it's a little difficult, right.

Sort of getting into what does it mean?

And trying to figure things out.

So what were some of the challenges you guys went through and how did you address.

Yeah.

So I would say the biggest challenge with any sort of security changes,

especially in a large company is just the unknown of what will this break.

Um, because quite often, especially in those early days, what you're changing

are out of the box configurations.

So there's this kind of mentality of, well, it's probably an out of the box

configuration for some reason, or, you know, we don't know what legacy, I

mean, this company that I was working at at the time was started in the 1930s.

Now we don't know what kind of legacy applications are

relying on this technology.

Um, so I would say the first, the biggest thing was to just start

easy, take the really easy ones to get as much buy-in as you can, you

know, sit down to the engineers.

Can anyone think of anything that would break by doing this?

And you will get some feedback.

Yeah.

But who cares?

It's not going to fix anything.

Okay.

That's fine.

Let's just do it anyways.

Um, and then start, you know, slowly ramping it up and

taking little bite-size chunks.

And if you look at the security CA cadence, Reddit posts, that's exactly

how I've been approaching them.

You know, I've started off with just really easy things to do and things

I would not expect, um, to break many enterprises, you know, I tried to

make it very clear in those posts of, you know, everyone's environment is

different and be careful, but you know, I've called out certain items of this.

Isn't going to break anything, just do it, you know, please just, just do it.

I do remember in your, the trio of posts that you did that were around

ransomware, you, you had a, there was a phrase that came up a lot.

It's escaping me at the moment, but it was like, turn this on

and then customize as necessary.

Right.

That, that, that, you know, that you can't, that no one solution does, uh, you,

you can make a general rule for example, and then you're going to find somebody

that needs, that thing turned on the thing you just turned off, you're going

to find somebody that needs, that turned on and then you can turn it on for them.

Right.

Um, and, and that's okay.

Yeah.

So one of the catchphrases are one of my guiding lights in InfoSec is to never

let perfect get in the way of being good.

I call it out a lot.

And part of the reason why I lean on it so heavily is it's often a. Uh, voice of

dissension that you get from folks when you're trying to talk them into things

like, oh, well that won't solve this one edge case, so let's not do it at all.

Um, and you know, when it comes to security, security, it's all about

layers and it's all about catching the attacker and yeah, there, this may

not solve all of your problems, but it might be the alert that gets generated

that tells you that they're there.

Um, you know, and so it is definitely a strong, um, demand I make a people of,

you know, if you can only do this for one system, great, it's better than none.

Um, and yeah, so there there's something to be said about going

slow and implementing slowly, but there's also something that I said

about implementing broad and then backing off where you need to.

Hmm.

Yeah.

Yeah, exactly.

I would say that.

When you, when you try the latter, when it, when I was thinking about your,

your initial, this, the six months program that you had, the farther

you got into that six months, and the more complicated things that you were

doing that were potentially riskier, if you will, to the environment that

you could potentially impact someone's ability to do their job, the more

you're going to need support from above.

Right?

Like, I, I, I told, you know, I told them to do this.

We're sorry that it broke, you know, we'll

Yeah.

we turned it off for now.

We didn't realize that by pushing this one button was going to make everyone in

the company not be able to log in ever.

Uh, we've turned it off until we figured that out.

Right.

Don't don't go, don't go beat, snorkel out.

And maybe that's also where you get some of those early

wins before you take on those.

So you get sort of the buy-in from upper management that, Hey,

they are doing the right things.

They are making improvements.

Yeah, absolutely.

Yeah,

One of the, the biggest allies of InfoSec people that they forget about is the CFO.

The CFO is the person when you're doing these sorts of things that you

want to have in your back pocket to be able to go have that conversation of,

Hey, where is our money actually made?

Because I want to know, Hey, what, what divisions of this company aren't really

contributing that much to the bottom line.

Cause those just became my test case the things that I'm really

not sure about.

Let's take them down because that's not going to, you know, that's not going

to ruin our end of quarter numbers.

So they're going to start this off as a retailer.

So that was obvious.

Don't take down the stores under no circumstances do you take down the stores.

Right.

Right.

but legal?

Go for for it,

Yep.

I don't know how legal would feel about that.

But yeah, no, I understand what you're saying basically.

So w every change that you made, you don't have to roll it out.

Company-wide you, you put it into places where you felt that it would do, you know,

hopefully the change would have a minimal impact, but if it did have an impact,

it would have a minimal impact to the company, because it only made legal, not

be able to do something for a day or two,

Absolutely.

which is a very different thing than no one can log into the

cash registers for a day or two.

Absolutely.

In retail, taking down the chain is the worst thing you could possibly do.

Yeah, exactly.

So for the record, I actually started in retail.

I, I worked a hundred years ago.

I was a shoe salesman at a, a chain called Kenny shoes, which no one.

Under 25 even know exists, but you know, it used to, it was the parent

company that created Footlocker.

So Footlocker is still around, but Kenny shoes was its own store.

And I worked in, uh, retail.

So I know I also worked at some, some what we now call big box stores.

So I know what it's like to be at the receiving end of that.

And when, uh, when corporate, when corporation changes things

and then poof, you know, you, you suddenly can't do your job.

That's unacceptable.

So you went through this exercise, you had this process of, um,

going for six months, doing a security update or roll out a week.

And then you started writing about this small things that people can do

to improve their security posture.

And where did you go from there?

Like, did you think you would keep writing this long because

how long have you been posting on Reddit for your security cadence?

So I, I only started it in January.

I did it as a new year's resolution.

Um, and it, the idea came to me.

I was on a, on a different podcast.

Um, and we were talking about InfoSec and we were talking about a term that I

believe Wendy Nader from duo security, uh, coined, which is InfoSec poverty.

Um, and it's basically in reference to companies that just don't have the

resources to have dedicated InfoSec people or InfoSec tooling, and how,

you know, it's not really fair to expect these companies that just

don't have those resources to really be able to stand up against you know,

the modern era of security threats.

Um, so on this podcast we were discussing, you know, what do, what can we as InfoSec

professionals do to help those companies?

Um, and it's been kind of living rent free in the back of my brain

since I was on that podcast.

Um, so as I was approaching the new year, I was like, you know what, I'm just going

to hop on Reddit, starting in January and make that weekly post and see if

I cant' help um, you know, some of the folks in the sysadmin sub Reddit, which,

you know, the sysadmin subreddit, they have the, um, the flares for everyone.

And there's a lot of them that list themselves as Jack of all trades.

And those are those sysadmins that are working in smaller companies.

And they're, you know, if it plugs into the wall, that's their job.

Um, and you know, those companies are exactly what InfoSec

poverty is calling out of.

You know, you have these brilliant sysadmins who are heavily

overburdened, and they just don't have the time to focus on this.

Um, and they just kind of need someone to say, Hey, you know, this week, why don't

you disable this one thing that comes out of the box in windows and you do not need,

and it creates a massive security risk.

Um, yeah, so I started in January and I have a nice long list,

uh, in one note of post to make.

And you know, every every week around Wednesday night, I just pull one

up and I write a quick blog post.

Yeah.

Cause you can't, you can't schedule Reddit posts, Right.

I don't

Right.

You can, uh, you can put them in drafts.

So I write them

Wednesday night and Monday morning I remove it from draft,

got it.

Yeah.

that was Paul's InfoSec weekly, I believe.

Was it the podcast where you were, right, right.

Yeah, Shout out to them.

Um, so, uh, so you said you started in January, So, you're what, uh,

like eight or nine posts in on that.

And did this, this, uh, this, what do you call it?

Um, or maybe like 10.

I don't know.

I can't do math.

Anyway.

This is, um, the ransomware posts were, where did that fall into that?

You know, the

yeah, it's a fun question because literally, since I started this since

post one, I've had people messaging me on Reddit saying, Hey, could

you do something about ransomware?

Um, cause it's you know, it's a top of mind topic, especially for the

smaller orgs, that's the big boogeyman.

Um, and I've been honestly kind of Mr. Miyagi'ing it in terms

of, well, everything I'm posting really has to do with ransomware.

You just don't realize it.

Um, but when.

reference by the way not sure if everybody listening will understand that

reference, But, very nice reference.

But, yeah, so my expectation was I haven't, you know, a list of posts that

eventually I was going to say, Hey, you know, if you've been messaging me

about ransomware, go read these posts.

This is what I was driving at.

Um, but then, uh, when Russia invaded Ukraine and the Conti ransomware

groups, uh, came out and said that anyone that takes up arms or,

you know, it goes against Russia.

We're going to come after I got flooded with people saying, no, really, please.

We need something.

So hence the title, the, okay, fine.

Let's talk about

Yeah,

Um, so I decided, yeah, it was time to just at least take

a truncated approach to it.

How did you approach because ransomware is such a huge topic, right?

I know Curtis, you and I, we talked about it, but just sort of your

thought process behind like the series that you wrote and how do you

get such a dense topic out there?

Because there are so many different ways that ransomware can attack you

and so many different, uh, crews out there with different methods.

So how do you sort of generalize it, especially, like you said,

for those people who don't have the time to research and follow up

everything related to InfoSec, right?

So I giggle when you say dense.

Cause one of the other pieces of feedback I get quite frequently

is that my posts are too long.

Um, but yeah, so my take on ransomware is that companies tend

to focus on the exact wrong spot.

Um, and I apologize for coming on to a backup, um, related podcast

and say that most companies focus on backup and that's, that's

No, no.

effort.

That's the, that's the thing that

hopefully saves the company when everything else has

just gone poorly for you.

And don't Don't worry, snorkel.

We have the same opinion as well, or at least I do, right.

That

it's just a last resort, but you should really be protecting yourself upfront.

Um, and so the thing that comes, that happens every time, there's a major

ransomware breach, um, is, you know, it hits the media and everyone starts talking

about the indicators of compromise.

It loves talking about indicators of compromise because it's easy to deal

with, you know, how did they get in?

I was an email.

Well, where did the email come from?

What was the subject?

Did it link to something, where did the link go to?

What did it download?

What was the hash of that downloaded and on and on and on and on.

And because it's easy then to go into your controls and, oh, we're going to put in

our spam filters to block that address.

We're going to block that domain.

We're going to put in our, uh, endpoint security tools to block that

hash, but it's all pointless, right?

Because that breach is done.

That entire infrastructure has been burned.

There was nothing left of it.

So, you know, you're, you're reacting to something that's no longer exist.

But when no one ever asks is, wait a minute, how did you know Susie

in accounting downloading this attachment lead to their entire VMware

infrastructure getting encrypted.

And that's, that's the real takeaway from every single ransomware breach of, you

know, it's one thing to come in and, you know, the accounting system, one person

in the accounting system is encrypted.

It's another thing entirely to come in.

And yeah, the entire network has gone now and we don't have any data.

Um, and that's really where the security cadence posts come in.

And w what I try to focus on, especially in the first post of this

is what I would be doing right now.

If you are waking up to a world where Russia has invaded Ukraine, and you're

all of a sudden, greatly concerned that ransomware group's going to

come after you, these are the things to start off with, and it isn't

necessarily preventing ransomware.

It's preventing ransomware from being able to do anything significant.

Um, and the nice thing about those controls is it translates

to way more than just ransomware.

Which is another issue that I think smaller companies particularly have

when they're dealing with InfoSec, as they put their blinders on and

very specific attack types, like how do we protect against ransomware?

Oh, we get good backups.

Well, how about, how do we protect against any sort of extortion attempt?

You know, we, we had the lupus group or excuse me, Lapsis

Yeah, I think it was locked system.

Yep.

I had been talking about all week and I just blanked on there anyways, you

know, going after Nvidia and Okta and Microsoft and LG, really their playbook

is the exact same as a ransomware group.

You know, ransomware only exists, not because they care

about encrypting your data.

They've went to extort you for money.

Ransomware shifted, shifted recently to exfiltrating data because people had

good backups or had restoration methods.

We'll find let's steal the data.

Cause we never really cared about encrypting the data.

We just needed that incentive to get you to pay.

Um, so when you take a step back and look at how attacks function

from the ground up and started going at the common denominators.

You, you really don't care about what the actual end objective is any longer

because the controls are there to make sure that they never made it past sending

that initial email, um, or, you know, tacking this particular vulnerability you

had exposed to the perimeter for a week.

Because those vulnerabilities will constantly be evolving.

Right.

And so you kind of need a generic.

Protection scheme, if you will, rather than something tailored for a particular

ransom group, but that comes after you.

Yeah, the rent

A ransomware attack is it's the conclusion of, you know, what, like you

got infected, but the ranch, I don't know if I'm saying, I'm not saying this

right, but It's I want to say it's the symptom, but it is actually the infection.

Right.

But that

not the cause.

The problem is what allowed them to get there in the first place.

You're absolutely right.

And it's, it's interesting that it is a multi-tiered product at this point.

There are, there are groups out there that sell you the initial breach.

Um, so if you and Conti is one of the groups that are suspected of doing this,

that they don't do the initial breach, they buy the breach, they buy someone

who already has the foothold and then use that foothold to do the actual encryption.

Hmm.

And so you have this entire life cycle of, you know, third-party vendors

that lead up to the final breach.

Right.

Um, you know, you know, Conti may be purchasing someone else's exploit kits,

um, someone else's encryption kits.

So you're exactly right.

That there's the initial breach.

That's really your first opportunity.

And when you get to the point where things are encrypting, so many other

things have been missed by that.

That at least if you get to that point, you know, you have

a lot of great opportunities to prevent it from happening again.

Cause you should have learned so much up to that point of, oh my gosh, they

got the phishing email through my end user is able to download this thing.

They were able to click this link to this weird domain that was stood up yesterday.

Um, they're able to execute a program after they downloaded it off the internet.

I mean all these different controls that had to go poorly just to

get to that point of encryption.

Uh, speaking of phishing, I did see something like this is just a couple

of days ago and they were, and, and again, I, I don't remember exactly

where I saw it, but it was like, it was saying that phishing had surpassed,

uh,, that it now become the number one method of attacking companies versus

I guess, uh, a standard exploit, I guess, would be number two, right?

A standard sort of direct hacking attempt the phishing had become the number one.

I don't know if you,

No,

sounds like you saw that as the number one.

Yeah.

Actually someone that, one of the things I really liked with the

security cadence post is when people get in and correct me, or, you know,

point out other things, because I'm certainly not an expert in all things.

Uh, but in the first ransomware post, I made a person who works

for a cyber insurance policy holder actually called me out and said, yo,

phishing is the number one for sure.

But right close on its heels is the proxy shell exchange vulnerability, uh, which

is a vulnerability from last year and, you know, impacting on-prem Exchange,

uh, deployments and, you know, still plenty of unpatched boxes out there.

But yeah, you know, you get these massive blips, right.

You know, a log4j S sort of thing.

That is a crazy large vulnerability.

That attackers jump on quickly.

But the internal one is always fishing.

There's always social engineering is the quickest path to get past your perimeter.

Yeah.

And especially with some of these large spikes, you also

have the long tails, right.

In terms of how long it takes to get every single system out there patched.

And you'll always have systems out there which don't go patched for so long and

still continues to be an attack vector.

Right?

Right.

And it's that InfoSec debt that, that again, of, you know, a company that

hired someone else that comes stand up their IT infrastructure one time

and it's been neglected ever since.

And there's no one patching those systems that are running

their exchange 2003 deployments.

I mean, they're out there.

the way.

Here's what I want to say.

Who the hell is still running on prem Exchange.

That's all I want to say about that.

And why aren't you using 365?

That's all I'm saying Microsoft.

You're welcome.

I'm just saying I don't, it's just, it's just

Wait, wait, you forgot to add one thing to that, Curtis,

what's that?

What's that.

if you are using Microsoft 365, make sure to back it up.

Yeah, absolutely.

Yes.

Thank you.

Because Microsoft isn't doing it for you.

Yeah.

Yeah.

It's a standard thing.

We have to mention here on, on the podcast,

So a dropper is typically the initial thing that gets downloaded.

So if you look through a normal, any sort of malware campaign, we'll keep

it as ransomware that, you know, I sent an email, uh, as a, as an

attacker, that's a phishing email.

And then the whole point is to try to trick someone into clicking a

link and downloading the program.

Um, or maybe it's attached, uh, maybe it's a word document or

something like that it's attached, but it's something small and.

Typically, you're going to see it as a document macro.

Um, and the whole point of it is that's the simple, easy thing that's going

to slip through, you know, your, your various defenses, because it's just

a word document, but you enable the macro in the macros, what reaches out

and downloads the current malware.

And there there's a few reasons for that.

A big one is that malware could potentially being, be being generated

on the fly, meaning that the definition that's behind that, the hash for it, or,

you know, the, the detection mechanisms that more traditional antivirus is

looking at won't have those definitions.

Cause it was generated at the moment of downloads.

Um, you know, we've just minor changes, but just to throw off that hash, um,

but then that's the thing that actually gets downloaded and executed and, um,

you know, causes you all your problems.

And, you know, from there it could be any number of things.

So as we were saying that there are people who would just tell

you that footprint, right.

Or that foothold.

Right.

That dropper could download just seed, too.

Just something that's calling back saying.

Yep.

I got something running on this computer and that's it.

Hmm.

All it could literally.

Oh, okay.

So he could just literally sit there and wait for the second group.

That's going to purchase that.

And then they download the malware that they want to download.

Right.

So.

That's what you were referring to earlier.

And so it looked like the, and again, this is common sense to you,

but not necessarily to everybody, it looked like, you know, your

best advice was to, to stop.

Ransomware is to just think about how ransomware works when it gets

in, when that dropper gets in.

You're not going to, I mean, yes, you should do user training and

yes, you should do, you know, you should do all those things.

And, but you should just assume that at least one of

them is going to get it wrong.

I mean, I remember back when I was, uh, you know, 25 years ago when I

was at a bank, we did regular InfoSec training with every new employee.

And one of the things we constantly said, well, Uh, no one in it

will ever call and ask you for your password ever, ever, ever.

And then we would, and then immediately after the training, we would call them

and ask them for their password and still a percentage of them would give it to us.

Right.

Um, So.

you, you do the training, but then you just sort of assume that that's going to,

um, you know, um, that th that somebody is going to click on the wrong link.

And so then you just think about stopping that malware at that point,

you know, stopping them from accessing a command and control server, looking

for, you know, this, this weird, you know, domains that stood up yesterday,

domains that were stood up a long time ago, but just suddenly when active,

um, you know, the limiting lateral movement inside the company, all of

these things, uh, what, what did I miss.

A big thing that a lot of ransomware particularly will do.

First thing is start deleting, shadow copies as a quick restoration point.

So that is a pretty dead giveaway of, you know, you get the event ID

that shadow copy was just deleted.

That's

And you're referring to VSS there, right?

The windows shadow copy.

Yeah.

So I had a question for you snorkel about that one.

Is, does that prevent backup apps from actually running that might

leverage VSS and shadow copies?

Maybe it is the short answer, but depending how you attack this,

if you're, if you're attacked for this is just, I want an alert on

anything that delete shadow copies.

Well, you know, if you have a backup solution that makes use of shadow copies

and deletes shadow copies, then you know, that's something that you tune out, right?

So you need to know the source of what deleted then that should be your event

ID and you just tune that one out.

Gotcha.

So then you should only look for anomalous events that happen.

Typically backup apps are going to, um, create a, create a shadow copy just

to have a stable frame of reference and then delete it when they're done.

Your backup app is probably running as a service.

So that's going to run a system or whatever your backup, um, username

is, or a user account is, whereas your ransomware is likely going

to be running as that end user.

Hm.

So when you ask yourself, does an accountant have reason to

be deleting, shadow copies?

Probably not.

So you can look for all these patterns and determine what's real versus

what's not because I guess that's the other hard part in InfoSec is like

tuning out the noise or the normal behavior versus what's anomalous.

Right.

Um, and you know, it all starts with really, really good logs.

you need to have that log information and then what's going on in your systems.

But honestly, going back to deleting shadow copies, of the other call-outs

I made from a higher level, it's just looking at what would, you

would expect an end user to run, especially from the command prompt.

Right.

You know, do you expect someone in legal to ever open a command

prompt, let alone, you know, run whoami or run nets, you know, and

start looking around your network.

Probably not.

So if someone in legal opens up a command prompt, that right there, it might be

enough for you to go well, that's weird.

Start running, you know, typical attack commands, or, you know,

living off the land commands.

Now it's real weird.

And what would you use to watch for.

Th there was a tool.

I forgot its name that you mentioned about that.

Uh, so the tool I mentioned in one of my blog posts was raccine,

which is so vaccine with an R, um, which is a tool that just monitors

for shadow copy deletion, and just kills any process that does it.

Um, the problem is it doesn't discriminate.

So again, if you do have a backup tool that does make use

of deleting shadow copies, it's going to kill that process for you.

Um, but if you don't have that limitation, it's a really handy,

little quick, simple solution.

but can you tune that or do you need another tool that's tuneable.

Uh, well, it's, it's open source, so you can certainly modify the code, but no,

uh, the current version that exists does not have any sort of options for that.

It is, uh, a one and done sort of thing.

Gotcha.

Okay.

yeah.

So that would be, that'd be a perfect example of, like you said, when we

were talking earlier, let's try this.

Right.

Hopefully it doesn't kill the backups, but if it does kill the backups, it

would be pretty obvious because all the backups will fail because they're

unable to create, uh, the shadow copies.

I think one of the ones, and I don't know which article number was from that I

thought was very unique that you brought up was a different way to sort of trick

the ransomware, um, into sort of not destroying your entire infrastructure.

I think one of the examples you brought up is sort of creating

hidden drives and book-ending normal drives available on that system.

So ransomware kind of get stuck, or you can monitor for that.

Yeah.

So, I mean, it's part two, in case you're wondering, um, so the, the actions

on objectives posts, so, you know, we have the, the initial infection,

you know, they mapped your network.

They're starting to spread out.

Now they're actually going to start trying to attack, you know, at this point.

1, you you have to call out that in 2022, one hopes that your endpoint

security software, you know, whatever anti-virus, anti-malware, you're

running sees process X is encrypting word document Y. I'm going to kill it.

If it doesn't, you really need to have a come to Jesus moment with your

endpoint protection vendor at this point.

But you know, if you have something that's running, that's actively doing that.

Um, at that point, I think one of the best controls you can possibly have.

Is feeding it data that you don't care about and putting alerts on it.

So, you know, as you were saying, Prasanna.

One of the things I do is I create, um, deceptive file shares on my network.

So just file servers that, you know, on their own separate windows box, doesn't

don't have any useful data on them.

I actually just clone my actual production file names and structures

and just put random data in them.

Uh, but then I make drive mappings to my end points.

Um, hidden drive mapping said, you know, from the windows GUI,

you can't see them, but, you know, from command prompt, you can.

Um, and I just book in my valid drives.

So, you know, you have a home drive at H. So put something before

that and put something after that.

And, you know, hopefully the ransomware will go after those first.

Um, and then I put just, you know, files on those servers that I monitor.

If anything gets changed for them, no reason for anyone

to ever touch these servers.

No reason for anyone to touch these files.

So, if anything gets modified, then it sets off alerts and I

know something weird is going on.

Um, and hopefully it buys you enough time to, you know, to remote in at three in

the morning and down whatever's going on.

Um, and yeah, the other thing I, I offered up in that it was what

I coined as a ransomware tar pit of an actual service that's just

running or monitoring that server.

And it just starts seeing files getting modified.

It starts generating more.

Um, so, you know, Hey, the ransomware, it hit my fake file share.

It file one.

Well, here's four more files for you and it'll just keep going and just keep going.

And you know, it may not be foolproof, but it might just depending on

how the ransomware is written, it might just put it in the loop

that it will never escape from.

Yeah.

I mean, uh, the concept of honeypots is not new, but I like to sort

of modifying it to, you know, the world of, of, uh, ransomware.

I agree with everything you said about stopping it in the first place.

What about, um, detecting data exfiltration?

What, what do you think, um, companies can do there?

Yeah, so it's a definitely a trickier process.

Um, mainly from a tooling standpoint, at this point, you're probably

going to have to open up the wallet.

Um, but you know, there there's one, there's just a basic security controls

of content filtering and making sure that your end users don't have a path out to

the internet for mass file transfers.

So for most enterprises, you probably don't need more than HTTP

and HTTPS from your workstations.

Um, so you know, a lot of those transfers are trying to transfer out via FTP

or, you know, a more traditional file transfer method that shouldn't be allowed.

Um, but going further, you need to look at, you know, what sites are out there

for allowing mass transfers, you know, to, to keep it simple, to do all your users

need to be able to reach Dropbox, um, or box or Google drive or any of that stuff.

If the answer's no, prevent it because that's an exfiltration method.

Um,

Let me ask you about that.

I guess I had this, this apparently misconception that they would be sending

these exfiltrated files to something that they owned and controlled.

Right.

So, and that's, so I'm trying to build this out from, what's easier to, harder

to implement, um, going to that point.

Yes.

So when you get to the point of we're going to just

transfer to something we own.

And honestly, at that point you're probably hitting up AWS

or Azure or something like that.

And that's where it gets really messy.

Unfortunately, in the cloud world, we live in, you can't exactly block Azure.

Right.

Um, but that's where I started looking at DNS security.

Um, and one of my absolute favorite controls is just blocking newly

registered domains or domains that have been parked for years that

have all of a sudden gone live.

Um, cause a lot of the attack infrastructure, and this is honestly a

great way of also stopping the initial drop or download because there's a

good chance that that's going to go somewhere that was just newly stood up.

Um, but yeah, so that, that is another control where you might be able to

just stop them from being able to get to whatever destination there

they stood up to accept these files.

Another point is always just a basic security, um, control of proper ACL's.

Um, you know, when it gets to data exfiltration again, you know, keep

picking on poor Susie in accounting, but Susie and in accounting shouldn't

be able to get to your HR documents.

She shouldn't be able to get to your operations documents.

Yeah.

They might be able to export out your payroll, which that's terrible, but

you know, your payroll showing up on pay spend tomorrow is a bad day.

It's not an end of the company sort of day though.

Right?

Like that's not trade secrets going out.

Um, you know, so, so that's another control.

Another thing that I would absolutely call out though is still honey documents.

You know, having documents for them to interact and transfer out that as soon

as you see somebody interact with that, you're, you're figuring out what process

it was and figuring out where system that came from, just stopping that information.

Um, and then going into the, probably the more logical solution, but

definitely at a cost it's just the behavioral controls because in that

exfiltration attempt, there's going to be an end point that's all of a sudden

transferring a lot of data out to a new source that has never been seen before.

And if you really know what normal looks like in your network, that should

stick out like a big, big red flag.

Um, and it's a very hard thing to do with free solutions, but there

are plenty of security products out there that are all about mapping,

how your end points interact with each other on the network and what

looks like normal, what isn't normal.

Um, I think it's money well spent for those types of controls.

Can I ask a question about an earlier topic you brought up around EDR.

So if most, or if EDRs are useful for detecting when encryption is happening and

killing processes, et cetera, if that's the case, would a lot of these ransomware

attacks be prevented to start with.

Or, and is it that companies who've been hit with ransomware have not deployed

EDR solutions in their environments?

If EDRs can detect when encryption is happening on endpoint devices, if a

company has deployed EDRs, does that mean they'd be able to stop ransomware?

And so a lot of companies who got hit with ransomware.

Didn't have EDRs deployed.

Endpoint detection response is what we're talking about here.

So first information security is all about layered defenses, and

you never rely on a single defense.

Um, in my mind when your antivirus, your EDR, WM, whatever your endpoint security

tool is, if that's the thing that stops the malware, thank God it was there,

but that's still a, oh my goodness.

How many different things failed before it got to the point

where that had to step up?

Like, I never want to see anything out of that system.

Um, that's not false positives.

Cause that's the fun with EDR is they are a pile of false positive.

Um, you know, to your question, did they not have it maybe, um, you know, EDRs

are expensive, they are expensive tools.

They are great tools, but they're expensive.

And like so many other expensive security tools that are rarely

set it and forget it tools.

They are tools that require a lot of tuning and a lot of

the finding of what's right.

Excuse me, within your enterprise.

Um, but you certainly have a lot of companies out there that are still,

you know, with prop with old definition based antivirus, that's just scanning

files and doing your nightly full scans.

You know, like we did back in the nineties, um, and companies that have

been very happy to embrace Microsoft defender as their only antivirus.

And, you know, there there's some logic to it.

It's a great solution and it's free depending on what your

office 365 licensing looks like.

Um, but I think the number of customers out there that have these large EDR

solutions are few and far between.

Uh, definitely the companies I'm targeting with my security cadence

posts are the companies that probably don't have that kind of assessment.

Yeah.

Or even if they did sort of managing it on a daily basis, becomes

difficult, especially with everything else they have to do, because it's

not a one and done sort of a deal.

Right.

And you know, honestly, if you think about it as a, the evolution of antivirus,

antivirus was a one and done for the most part, you know, you deployed Symantec

back in the day, next next finish.

And you never touched it again.

Right.

EDRs aren't that.

EDR is, are constant tuning and definitions and breaking

things in your environment and having to tune them back out.

And I think that's also in the recall that I know of a number of

companies have thrown their EDRs out because, oh, it just broke everything.

That product was terrible.

It wasn't, you just needed the resources to handle it properly.

Yeah.

I want to sort of round out things here.

There's some things that we haven't talked about that were obvious ones that were.

You mentioned in your first post that, you know, you talked about, you know, you

password security, you talked about MFA.

Um, these are things that just everybody should be doing.

Uh, my, my personal opinion at this point, you know, th th you know,

if you're not doing MFA on anything that matters, uh, you know, you're

not doing your job and, uh, and you know, that's my opinion for what it's

worth, but MFA stops so many things.

Yeah, I If you were waiting for me to disagree.

I wasn't going to.

I have a question for you though.

So I dunno if we're going to talk about it now or later, but I've read

recently with a lot of the Lapsis attacks as well as other gangs, right.

There is a notion of SIM swapping attacks, right.

Which sort of hurt some of the MFA approaches taken.

So Prasanna, don't let perfect get in the way being good,

I I was gonna I was going to say that that goes right to that, right?

Yeah.

Just because it won't fix everything doesn't mean you shouldn't do it, right.

but no.

not, there is no silver bullet.

Right.

Um, and you know, a good backup person would never say don't do InfoSec and

a good InfoSec person would never say, do backup or not do backup.

Um, but I think that MFA is just so, and by the way, I, I finally

ate my own dog food maybe about two years ago where I just realized that

there were a lot of vendors that I personally interacted with, banks and

things, that offered MFA as an option.

And I finally said, look, I know it's going to make it harder for me to access

my bank account and my, you know, my PayPal and, you know, I, there, there's

only like, I dunno, there's only like 20 accounts that I felt had that level

of information that I needed MFA on.

Um, and then, and then, and then I became like this like MFA Nazi, where I was like,

I'm mad at them if they don't offer MFA.

I remember what happened when you traded in your phone and

you lost access to your MFA.

Yeah.

So I was using a Google authenticator, not realizing that when I traded in my

phone that I lost all of my MFA tokens.

And so I switched actually to authy, so that I can, I don't have that problem.

But, um, and now, and now I'm actually looking at a password manager.

I think it's one password that manages both your passwords and your MFA stuff.

That sounds nice.

So I'm already a big password manager, uh, fan, I just, um,

didn't, you know, I currently have to use two solutions, but Yeah.

Yeah.

And to your point, Prasanna, about, you know, SIM jacking you know, and not being

the silver bullet, the one thing I'd say is not all MFA's are created equal.

Any MFA is better than no MFA, um, but you know, it doesn't have to be a

roadblock in your organization, you know, Fido keys, Titan keys, things like that,

that are literally you get the prompt and you tap a, the thing hanging out of

your USB port or taps onto your phone is a really, really nice MFA solution.

That's really easy for your users.

And they require something that you have that's truly physical.

Um, and rather than, you know, replying to a text message, you know, there's,

uh, one of the big debates going on in the InfoSec world right now is the, the

push notifications of yes, that was me.

Um, and it's right out of, uh, the playbook of lapsis of just that they

actually had a picture on Twitter, um, yesterday of one of their chat things.

Yeah.

Just spam them a hundred times.

Eventually they'll get mad and hit yes.

Yeah, absolutely.

And it's funny, cause I actually had, um, our MFA at work today went a little

bit sideways and they started pinging me repeatedly for something I had just

tried to sign into it had in the back of my mind, like, Hmm, check the logs to

make sure like, this is kind of weird.

Um, but still it's, it's better than nothing and yeah.

Your end user may fail you and hit.

Yes, that was me.

Cause I got tired of getting this prompt a hundred times.

at three in the morning, but still, um, but as InfoSec practitioners,

that's where we need to step in and go don't we think it was abnormal that

they got a hundred prompts, like did that not set off an alarm right there?

Why is it if we're getting pinged over and over and over and over again?

Um, cause I guarantee you that created a log somewhere.

Yeah.

All right.

All right.

Well, the summary statement of your, of your blog series or your post series,

whatever you going to call this, and by the way, your posts are long.

I, uh, after you made the comment I went and while you were talking,

I copied and pasted the three posts into a, uh, Google docs.

Uh, one of them is 4,500 words long my friend.

I mean, that's long, even for me, I'm just saying

I got to tell you someone the other day commented about how

much they liked my writing style.

And it was the first time ever in my 42 years of life that

anyone has ever said such a thing

I actually liked your writing style.

I thought it was good

This is a reason, this is a reason you're here is you.

Well, it's a complicated issue.

So, you know, I, I jab, but you know, 4,500 words is not that much

for, for an issue of this magnitude.

Right.

But, uh, the first was 2,500.

The second one was 2000, but the last one was 4,500.

I was like, yeah, the boy could talk.

Um, I would just reiterate what I said of so many companies

focus on the recovery side.

Um, and that focus comes from focusing on what the actual objective was of

the ransomware group to begin with.

What was it?

They were trying to do, whether they were going to encrypt your stuff.

So you start there and it's the wrong place to start.

Start at the beginning, start with how they're going to

compromise your first endpoint.

And if you start looking at InfoSec from that perspective, what you'll end up

finding, and it's a really gratifying feeling is you would turn on the news one

day and there will be the latest, massive vulnerability or exploit being discussed.

And you'll look at it and go, oh, my controls account for that.

Not because I built controls around that particular exploit or vulnerability,

but because I just built my controls around, how do I walk an attacker through

initial foothold, moving laterally throughout my environment, and then acting

on their objectives and then how do I stop them at each one of those steps?

Um, so I guess the, the, the too long, didn't read it to use Reddit terminology.

Don't focus on ransomware, just focus on how attacks function and you'll get

ransomware taken care of by default.

Right in the, the only major one that we didn't discuss, which we really

should have is the whole patching thing.

Right.

I go back to, I'm pretty sure if my, if my memory serves correctly, Wannacry.

Was, you know, it was one of the.

first big ones that really went, you know, it went haywire and everywhere.

If I recall correctly, it was an exploit that had been patched

a year prior in a windows.

And if you had just been anywhere near up to date, then you'd have been fine.

Yeah.

And Microsoft is, they are both great in how they maintain with a

lion's grip to a, or with an iron grip to backwards compatibility.

And they are also just ridiculous.

in they're lions grip, iron grip on backwards compatibility,

but I mean, Wannacry.

The other side of Wannacry is exposure to the internet, um, which, and this kind

of goes back to the proxy shell thing.

You know, Wannacry has went gangbusters because of all the companies that

have Samba, SMB exposed to the internet, which again is you take

a step back and go good god, why?

But then you go jump on something like showdown and you look

like, oh yeah, there's tons.

There's tons of vCenters exposed to the internet and you go good god, why?

Um, and again, it just kind of comes down to, well, I had this one sysadmin who

was overworked and doing what they could, and we had this use case and he stood

it up, but he wasn't InfoSec focused.

He didn't know what he was doing.

And from that regards and didn't see the problem with it.

Um, so yeah, I mean, to your point of patching.

Absolutely.

Uh, but I guess my takeaway is focusing on the big things.

And don't worry about the latest zero day, quite as much.

Um, cause I tell you attackers rarely are focused on that.

Cause there's so much low-hanging fruit of the stuff that has

been patched since 2018.

Anyways.

Yeah.

you, you mentioned SMB.

My other big one is RDP RDP to the internet is just, I just want to slap you.

Well, listen, we could talk all day.

Uh, I just, I want to say thank you again, and, you know, for

coming on the podcast and talking to these really important things,

No, it was a pleasure to be here.

I really, I have to say when I got, when I got your message,

like, wait, is that Mr. Backup?

That is Mr. Backup!

I was really excited.

this is me.

I, well, and I'm honored that you, that you knew who I was, so, Hey,

you know, we're, we're members of the mutual admiration society, Prasanna?

Yeah, no, it's been great snorkel having you on and yeah, great

articles I'm will continue to read.

I hope you keep doing your weekly posts on security cadence, because I'm sure a lot

of people learn a lot of things from that.

So

I'll try to make a cliff notes version.

yeah, I keep it long.

I

Yeah.

Good, good luck with that.

Uh, yeah, he's easy to find on Reddit is snorkel 42, make sure to check them.

out and make sure to subscribe.

The Backup Wrap Up is written, recorded, and produced by me, W. Curtis Preston.

If you need backup or DR consulting, content generation, or expert witness

work, check out backupcentral.com.

You can also find links for my O'Reilly books on the same website.

Remember, this is an independent podcast, and any opinions that

you hear are those of the speaker and not necessarily an employer.

Thanks for listening