Stopping Ransomware Lateral Movement Before It Spreads

According to CrowdStrike's most recent State of Ransomware

survey, 78% of respondents were attacked by ransomware last year.

With odds like that, we figured it was time to bring on one of our most

popular episodes, and this one's all about stopping ransomware from moving

around once it's already in your network.

I brought on Tom Hollingsworth, who knows networking way better than I do.

Uh, we get into VLANs, zero trust, firewalls, and a few other

things, including some, at least I think, really good war stories.

If this is your first time watching or listening to me, I'm

W. Curtis Preston, AKA Mr. Backup.

I've been obsessing over backup, recovery, and now cyber recovery for over 30 years.

If that's your bag, then I'm your guy.

You're not gonna find anyone that's cares about backups more than me.

Ever since 1993 when I had to tell my boss that there were no backups of

the database that we had just lost.

Now I've written five O'Reilly books, a blog, and a podcast.

Here, we turn unappreciated admins into cyber recovery heroes.

This is the Backup Wrap-Up

Hi and welcome to Backup Central's podcast.

I'm your host, W. Curtis Preston, aka a Mr. Backup and have with me possibly

my Pex consultant Prasanna Malaiyandi.

it going?

Prasanna,

am.

I'm good Curtis.

And just for people that's p e x, not P E C K S.

Yeah, this is the piping, the, the modern piping alternative to copper,

which I think is far superior.

And, You know what?

just for those that are watching this on on video, which is only a handful of

you, but I'm gonna tilt my camera up and this is what my office looks like right

now because I got yet another pinhole leak in my, second story water supply,

which happens to be right above my office.

And yesterday I was just sitting here at my desk and I get this

drip drip on my face and I'm like,

you're like, am I sweating profusely?

Yeah.

And the pipe is actually over there.

the joint that's leaking, it's actually over there, the water finds its way,

down cracks.

Yeah,

it just drips

down onto my face.

Yeah, we wanna bring on our guest.

he is both, I would say, a friend of the pod.

He's also been an enemy of the pod at once.

You may recall that we had an episode where basically we just argued

with Tom without his per, without him being here to defend himself.

that was over a blog post that he said, something about, backup

people reporting to security people.

And, I

Yep.

had an issue with that or something.

Tom has been in the industry about 20 years and he is an

event lead over at Gestalt.

It the, what would you call it?

The makers of the Tech Field Day series,

and, we're glad to have him on the podcast.

Welcome, Tom Hollingsworth.

thank you for having me on Curtis.

It was, it was fascinating to listen to an episode where I was arguing

with somebody and it wasn't even here.

But, I love listening to you guys, and I've learned quite a bit.

In fact, the very first time that Curtis and I ever met at Tech Field Day back

in 2011, he was teaching me about data de-duplication, and I was trying to

convince him that IP V6 was important.

And I can tell you which one of those things panned out a lot better than the.

is it, that the thing where you do the nat behind the thing?

That's what I recall really learning from you was that you gotta do

NAT for I P V C

like the Kool-Aid man, just keep

Yeah.

Yeah.

Yeah.

I wanted to bring on somebody that actually understood networking

far better than me, right?

Which, which is basically many people in the world.

With ransomware attacks.

One of the things that we talk about is once you've, figured out that you

actually have a ransomware attack, you want to isolate the network.

And there's a discussion, I've been talking with CISOs lately and what

appears to be the reality is that few environments do the actual full.

Like we just we're just shutting everything off.

Go grab the cable, pull it out

actually, I know.

Tom, did you ever watch, alias when it was on

with Jennifer Garner and.

Okay, there's an episode in there when they were having a cyber

attack and the, what's his name?

Marshall Flank man comes running into the data center and he just literally

starts flipping, flipping power switches.

He's they're downloading all the files off the server and he down.

He just flips all the power switches off.

on one end there is the

like networking, shutdown, like literally both internal and external, right?

because, once the ransomware is inside, it's gonna try to crawl

around and make things worse.

So that's one way.

And then there are, and then there's the, those that go, I'm just going to turn

it off, I'm gonna unplug the cable at the one server or the three servers that

appear to be infected and I'm not gonna worry about the rest of the network.

And somewhere in the, between those two extremes is what everybody else does.

And maybe we should also talk about basics of networking before we jump

into this to talk about the detail.

Because just

Go ahead.

I,

Prasanna,

no, I,

you think we should be talking about first?

no, I think it's because what you just mentioned, Curtis, like everyone

might think, oh, all computers are plugged into the same network.

I think it's important to talk about some of the best practices from networking,

Tom, if you could, about sort of network isolation, BLANs, other things like that.

Before we get into sort of the other side of things,

Yeah, so please explain all networking technology, period before

we get started.

, you've already talked a little bit about it because it's just

a series of tubes, pipes, if you will, that we send things through.

now the important thing to realize when you're trying to think about how

ransomware propagates through a network is to realize that, the way that networks

have traditionally been built is we have this perimeter on the outside, it's

probably bounded by and a bunch of other stuff, and it looks really imposing on the

castle walls, but inside of the network, it's a whole lot easier to get around.

And that's just due to the nature of the way that networks operate.

ethernet is effectively like trying to shout out somebody's order

number at a fast food restaurant and hoping that you get the right one.

Everybody's gonna hear the message, but if it's not meant for you,

we're just gonna ignore it.

But the problem is that allows you to propagate a lot of information very

quickly, and that's what ransomware is trying to take, advantage of whenever

it's trying to, do almost reconnaissance lateral movement in the network.

So I'm looking for a whole bunch of, , potentially vulnerable servers going

all the way back, to the beginning of my professional IT career, I was

actually working on a help desk, when the S SQL slammer worm came out.

And boy, you'd be surprised how many people had that port open to the

internet, because everything shut down.

And it was really weird to see that.

And you're like, at the time I'm freshly minted in my career.

And I'm like, how could that happen?

and now all these years later, I look at it and go, oh my God,

these people were stupid because you're not supposed to do that.

But that's one of the things that people want to take advantage of because the

systems want to talk to each other.

They want to be able to exchange information.

That's the purpose of a network.

Right.

to do extra work to prevent them from talking to each other.

Right.

Yeah.

I think that's, I, and the number of times I went in and out of data centers, over

the years, I remember only one, where they had very solid firewalls, basically.

That, that it was very difficult to do, to traverse laterally

within the organization.

And that was actually Intuit, and it's because of what they felt they had.

They had all of this very sensitive personal data, thanks to their,

they had QuickBooks, they have TurboTax, they have all of that stuff.

And so they had to basically firewall off systems between each other to

prevent that lateral movement that you're right by design in most networks,

you buy a switch, you buy, a bunch of switches, you plug everything in.

And everything talk, everything can talk to everything.

and unless you do something to prevent it, lot of those ports that you talked

about, just like the SQL, issue, a lot of those ports are visible to the internet.

I think a, another one would be a vCenter Right.

And Hyper V, the, that, those ports being visible to the internet, I suppose

you hear about that a lot as well.

Yeah.

I usually do.

Whenever there's some kind of, a vulnerability that comes out and

everyone's I hope you don't have these exposed to the internet, and

you can literally hear the scrabbling as people run into their keyboards

to figure out if that's the case.

But,

as Prasanna mentioned, we have ways to like segment

networks away from each other.

And it's funny that you bring up that, that Intuit had a rigorous internal

firewall structure because in my experience, companies or organizations

that are very, heavily regulat.

Have much more strict structure.

And the reason for that is because they need the ability to say

for a fact, Curtis cannot see anything on this network because he

hasn't been authorized to see it.

Now, you can do that through software constructs.

VLANs, virtual local area networks are the most common way to do it, where we

effectively divide some, partition on the switch and we say, this port belongs

to this vlan, so it can only talk to other ports that are on that vlan.

but that's not even good enough for some organizations.

and the one that everybody always thinks of is Mission Impossible, the Tom Cruise

movie with the machine that's in a vault that's not connected to anything else.

We would call that an air gap system.

Or you can have an air gap network a lot of times things like, HVAC or

management systems are air gap from the rest of the network because they

have different controls and different needs, but I also don't trust those

people to, secure their stuff.

So I'm gonna build a wall in front of that air gap or just completely

isolate it, itself so that I don't have to worry about securing it.

And if, you say hvac, you say things like, environmental control systems

and any security people listening to this podcast are immediately

thinking, man, those are back doors that I can use to get into the system.

Because no matter what, they're still gonna have to be connected

to the network somehow.

And that just increases your, your threat profile.

Yeah, it's interesting because I think most people who think

about home networks, right?

Everything's typically flat in a home, right?

Everything can talk to everything, every single iot device out there, right?

And they're not always thinking about, Hey, I got this smart light bulb.

Isn't it great?

Isn't it awesome?

And then realizing that's on my network, everything is now exposed and could

be potentially exposed if there's a security issue with that single device,

Yeah, Those devices are, they obviously have an IP address, they

have some kind of a control system.

You would hope that most of them have some kind of a security function that

allows them to securely communicate back to whatever controls them.

But multiply that by a factor of 10 for all of the devices that could be

on your average enterprise network.

And when you start saying things like, access controls for those

devices, or port security like network engineering and operations folks, they

just start breaking out into hives.

the, just the amount of work that it takes to create that level

of security is its own monster.

anyone who's ever deployed a technology like 8 0 2 0.1 x, which is effectively,

I am only gonna allow authorized devices to be plugged into this port, knows that

there's this whole enrollment process and are you on the authorized users list?

And what happens if you're using a different device today?

And it's just, it's maddening and it drives people to insane to the

point where, and that's the normal people who know what they're doing.

Could you imagine an executive plugging their laptop into a network port one

day and going, this doesn't work.

And you tell 'em, oh, it's doing that on purpose because we

want to keep everything secure.

What do you think is gonna happen?

The executive's probably gonna look at you and go, I don't care.

Turn it off.

Exactly.

. We don't need that.

It's getting

Yeah.

Yeah,

Yeah.

I know that when we had, we had a security person on and they had a list of things

that they wanted people to do that they felt were common sense, that were, ways

to prevent basically, I think the proper thing today when we talk about ransomware

is to just assume something in your world is going to get ransomware, right?

It's just, it is, I think it's just impossible to stop it 100% of the time.

So just assume that's going to happen.

So then there's all about.

How to prevent it from activating itself, from talking to the command and control

servers and also the lateral movement.

So he

reducing the black

raised,

right?

So what's that?

Limiting the blast

radius.

so Tom, what kinds of things besides VLANs?

Because even VLANs, we have the VLAN for this and the VLAN for that.

Still all the servers within that VLAN can talk to each other.

What else can companies do, with modern networking equipment to prevent

lateral movement or to basically prevent it from everything and then,

selectively allow it for certain servers.

the first thing you have to do is you have to realize that a completely flat network.

not a stable network.

there is a limit to the amount of chatter that a network can tolerate

before it starts running into problems.

ethernet is not a, a medium that allows for a large number of hosts because

eventually they're gonna, it, it's like recording a podcast eventually

with too many guests on the podcast.

You're all gonna wanna talk over the top of each other,

and ethernet doesn't like that.

So once you had a certain boundary, you have to divide it

up into these little domains.

collision domains are what we call them, and that's one

of the things that a VLAN is.

But as we've learned over the years about what we really should be doing,

we've built a super set of that.

And anyone out there who has been reading any kind of the tech press recently, or

been to any trade show in the last couple of years, probably heard of something

like Zero Trust Network Architecture or, just Zero Trust in general.

It's a buzzword.

I'm, I'll be the first to admit it, but the principles behind it are fairly sound.

what you do is you take the tools that you've already been given, those ones

that I told you, make your network team break out in hives, and you try to

implement them in such a way as to reduce the complexity of the implementation.

And think about think about a teenager and they want a list of, things that

they can do when they get a car.

Are you gonna tell them you can do anything you want, but

you can't do this and you can't do that and you can't do this?

Or are you gonna be more explicit?

You can only do these things and if it's not on that list, you can't do it.

most people would say, I'm only go, I'm gonna do the second thing

because I want to make sure that they're only going to school and to

work into this one friend's house.

But we don't build networks that way.

we typically allow as much as possible because of the situations

we find ourselves in where something doesn't work right.

And we don't know why.

So we will put a little catchall at the bottom of the access list

and go permit everything else.

and then we leave it.

And that's the worst thing that you can do.

And what Zero Trust Network architectures try to do is they try

to say, okay, that server over there is running our backup software.

What should it, what should communicate with it?

And how should it be communicated with, maybe it only needs to accept

connections on these three or four ports.

Maybe it only accepts connections from these authorized users.

And you're effectively creating an isolation for that unit.

And if something needs to access it and you're having problems with it, the

software usually allows you to dig into that a little bit and go, oh, it looks

like that this program did an update and it now needs to communicate over this

port, and I need to allow that port.

But you're doing it in a way that allows you to control that access.

But more importantly, what happens is that when something tries to operate outside

of that access control, slams it shut and hopefully will send you some kind of a

warning, Hey, we just noticed that this server over here is trying to communicate

with the rest of the network on Port 4 45.

know it shouldn't be doing that.

You need to take a look at it.

And so limiting that blast radius, that broadcast capability tends

to prevent lateral movement.

And like you said, people who are going to attack you are going

to be dedicated in doing it.

Either they're gonna be dedicated to looking for a very specific exploit

and just hauling in whatever they can do, or they're gonna be looking to

attack you, you specifically, however they can get to you that second kind

of attacker, very difficult to block.

It's like a door lock, a dedicated burglar is gonna get into your house.

You're looking to prevent more of the first one where it's oh, we were able

to get in through your HVAC system and boy, we're gonna turn this thing loose

and see what open file shares you've got out there and what we can do with them.

You, you need to create in the organization that does not allow

people to move laterally that prevents them from accessing things.

Or worse yet, alerts you when things start doing a lot of across your

network, looking for those kinds of things because the rest of the group

that's trying to get into your network doesn't know that stuff's there either.

They're gonna have to go looking and just like the burglars that are casing the

joint, you need to look for those people.

So multiple things popped up in my head, Tom, as you were talking.

So the first is, as you're talking about the burglar example, I'm gonna bring this

up again for the second week, but Curtis had recommended reading The Cuckoo's Egg.

I don't know if you've read that book.

Tom.

Highly recommend you read it.

It's basically, 1980s, a hacker gets into a mainframe and starts moving

laterally across all these like military networks and science networks

because everything was connected.

And

Yeah.

you said, that example was go and try all the door locks and he would

try default passwords and some of these systems, like the mainframes,

people would not change the defaults.

And so he got in and it was just that lateral movement across

everything in the environment.

So that's like the first thing that came to mind as you were talking.

the other thing that also came to mind is I totally get the reason to have that

zero trust and only enables services that, and patterns that are known to

be valid and disable everything else.

my question.

As a network engineer or operations person, how do

you manage that at the scale?

Because there's so many applications, so many servers, it's hard to predict what's

going to talk with what, and coming up with, because everything's all connected.

Like in my mind I think about Facebook and graphs, right?

Everything is connected in the world, right?

And so everything in your network to some extent is probably

connected in some form or fashion.

So how do you go about even coming up with, okay, these things are the

things that should be talking to the backup server in your example.

So it takes a lot of teamwork because as a network person, I don't care

what's running over my network, I just need to make sure that these

two things can talk to each other.

And so in a way, like if you've ever deployed a server, you have a list,

okay, it needs to communicate, using this protocol over these ports or,

think about, opening something like, I need to open HTTPS to the server,

but not http because I don't want it to ever communicate over http.

And that's actually one of the things that we've noticed a lot recently is

that a lot of protocols that used to have their own dedicated ports have

now just started writing over, HTTP and https s. Because it's just easier.

bit Torrent was actually one of the first ones to start doing this because

they're like, eighty's gonna be open anyway, which is the port for http.

So we'll just ride on that because most people fire, most people's firewalling

systems just allow that by default, because that's what the web uses.

And so it gets insidious and you almost have to think at a higher level.

So what.

it crack open any networking textbook in the world, and they're gonna

give you this seven layer model.

It's like a seven layer dip from Taco Bell, but there's no refried

beans in the seven layer OSI model.

But we play a lot in the bottom of that, where the physical connections

happen, where the IP addresses allow systems to talk to each other.

Once we get above a certain level, that's where the applications take over.

And as networking people, we're not as concerned about that.

But boy, the server people are because, oh, I need to be able to have these

two devices talking to each other.

I need to make sure this is all un impeded.

And the first thing that happens when two servers can't talk to each other

is you gotta find the network people.

And you're like, you need to tell me what's going on here.

And then invariably, like the security team gets drawn in because oh no, we

told him that he had to block that because nobody should ever be using that.

and you really do have to pull those people together.

think of, think of a book like, gene Kim's Phoenix project.

Like you can't work in isolation anymore.

As much as we might like to.

Because so many things are so inter interdependent now.

It's the old joke is what does the server do?

I don't know.

Unplug the cable and we'll see who screams the loudest.

You wanna figure out what people, what port is being used.

Let's block it and see who comes to yell at us.

that's the way you have to do some of these things.

Cuz

the other thing, and we all know that nobody ever skips documentation, right?

You brought up an old memory of mine.

Literally like my first months in being a cis admin, we were trying to

decommission, the, the first computer to run Unix was the three BK and the

at and t had a three BK I think it was like a three B And it was their

attempt at a multiprocessor architecture.

And we had this beast and we were trying to decommission it.

And, we had gotten down to, we had fi and we had gotten down to

that phase where it's we're just gonna turn it off and whoever yells

will be the one that we missed.

But I remember the, We had, stripped it, all of its regular networking cable.

I don't exactly remember exactly why, but I remember that there was one cable left

and it was running across the floor and we were doing the last like download of was

off of this server onto something else.

And the manager for that cost center was in there and he

kept stepping on the cable.

we told him that he was slowing down the download whenever

he would step on the cable.

we actually caught him, we left him into data center.

We actually caught him like watching the monitor and like the

throughput speed and stepping on and stepping up and off on the cable.

Bad Curtis.

Yeah.

good stories.

I, so question I want to ask you about, all of the things you just

talked about, this something built into modern networking equipment or

is this, are these extra applications that I'm buying that then configure

that networking equipment for me?

So it can be both.

the basics of being able to isolate hosts and configure systems

has been built in for years.

anyone can write an a c L, right?

The thing is that scaling that across a large organization is

where it typically falls down.

Eventually, your security team can't keep up with all the changes.

They throw their hands up in the air and it lies fallow for as long as

it takes for you to get infected.

So the additional tools that are basically being brought to market and

are popular now, organize that system.

They put a shiny.

UI on it, if you will, to go in and say, okay, I want to enable port

security on these ports because back when I started this port security was,

if it isn't being used, shut it off.

Just shut down the port.

And then if somebody plugs into it and it doesn't work, then now we know

we need to enable that port and we need to know who's trying to use it.

But now you have the ability to have somebody plug in a device, whether

it's an IOT system or what have

you, and this, the device will register with the system.

It'll say, Hey, I need access.

And then the system can come back and say, Hey, it looks like somebody

plugged in an S thermostat over here.

that's actually a bad example cause they don't use wires, a laptop

or some other kind of device, you need to go, check it out.

Or you can even set a policy that says, I'm going to allow you for now, I have the

ability to just cut it off if I need to.

Or if it's one of these recognized device classes or something like that.

So for smaller systems, for smaller organizations, if your IT department

isn't already completely overworked, you can't implement some of this by hand.

It's just a matter of if it works really well, that means you're gonna

be spending a lot of time tuning that system to keep working effectively.

And once you get past a certain point, the, the solutions that do

this are reassuringly expensive because they're worth it.

Oh, I understood cuz that they would help you save the labor.

And is there a category of these types of tools that, that a

category name that we give to them?

there's a bunch of different ones.

access management is typically one that honestly, tools like Aruba

ClearPass or Cisco ice, ise, integrated services engine, or integrated security

engine, I forget which one it is.

But they're not identity and access management, although

they can be integrated that.

There are some smaller ones that have these capabilities.

A lot of it is mostly figuring out what you need because there's different,

some systems are configured so that you're controlling access to devices.

I only wanna authorize people to be able to log into this

device and make changes to it.

that's different than I want to change the way that people in my network are

accessing data like that is a different kind of identity and access management.

So you need to do a little bit of investigative work to make sure that

you are, properly using the right tool.

Cause if you spend a lot of money on one that doesn't give you what you want or

does a terrible job of it, then not only are you gonna be upset, but the people

that are or authorizing your budget are not gonna be very happy with you.

Now a lot of these changes, if I think about an enterprise

environment, things are easier to a fair extent to control, right?

If you're looking at servers or virtualization, other things like that.

But then I go to think about other environments like a school, where you

have students coming and going, right?

Or a stadium or a conference center, right?

Does it get significantly more difficult to do what you talked

about, Tom, in those environments?

Or can the same tools apply there as well?

Yes and no.

I'm the typical IT nerd.

The answer is, it depends for whatever question you ask, but I'll tell

you that in some ways, schools and other places where your user base

is not employed directly by you.

have a slightly easier time if you're willing to, a little bit.

So I know that there are a lot of colleges out there that treat their

student dorm networks like the wild west.

We don't care what goes on out there, but we're not gonna keep an eye on it either.

So if

there's a, a piece of ransomware or something that's running rampant through

the system, all we did is tell you that you had to have your antivirus up to

date to be able to join our network.

Yeah.

the stadiums are actually a really interesting, problem too, because not

only do you have a, a group of users that are outside of your control, they're

very transient, in a lot of those places.

Like they, they actually have, wireless networks that are set

up so that, they can only talk.

, like they block all device to device communication, which

is something that you can do.

It's a little bit more complicated, but it effectively treats, the

stadium itself like a demilitarized zone in a, in a security structure.

So for most people that are familiar with it, you've got the outside

internet, which is big and scary.

You've got your inside network, which is soft and you don't want it to get hurt.

And then in the middle you have the dmz, which is basically the moat where

you're like, I'm gonna put everything that I don't care if it gets attacked

out there so that if it breaks, it can't get back into my network.

And but the otherwise, the other thing there is I only allow certain

traffic to come back through.

So if something bad were to happen, can just basically cut it off and

sink it into the moat and I'm done.

Yeah, I think, hotels have a similar model, right?

Where the base, I know having plugged in multiple devices that needed to

talk to each other in hotel networks, they don't like that very much.

and you end up having to bring basically your own router if that's

something that you want to do.

the, so it sounded like, if I understood you correctly, the access management

part is this sort of basic security thing, that there are tools that do

just that, and then there's also this identity access, which is a bigger pain.

I would imagine.

But those that want that, and it sounds like when we put those two together,

that's what we call a SEIM tool, right?

Is identity and access management.

But it sounds like there's just an access management.

That, for those that need just that there, there's smaller and less

expensive than a full SEIM tool.

Yeah.

not inexpensive, but just less expensive.

and it also matters as to what you're spending your resources on, because there

are tools that will do this for free.

But they are not supported at all by anybody other than people on a forum.

And they'll be glad to tell you that you misconfigured something

and go figure it out yourself.

Like we, we've dealt with that.

And I'm not really crapping on the open source community because

they do an amazing job of this.

I'm crapping on the fact that open source communities are not as well supported

as the bigger players in these markets.

And that's the expensive part comes from.

You're not paying for the software, although you are in some ways.

You're paying for somebody to answer the phone when somebody is like breathing

down your neck because something won't work or something won't come online.

And so a and you're also trying to get to that point where not automated

as much as it is as low possible.

Because what you want in situations is people to just

be able to get on the network.

that's the thing.

If you've ever tried to log into a wifi network that has a captive portal

that requires you to accept a whole bunch of licensing agreements and

type your room number in and all the other stuff, you know that it's not

the most frustrating thing, but it's definitely not what you want to hear.

As opposed to oh, this device has already been pre-authorized cuz you logged in

with your active directory username.

we'll just let it on the network.

That's completely frictionless.

But the amount of effort that it takes to make it frictionless is where your time

and resource invests gonna come from.

Tom, I know we started this all out with Curtis asking, how do

you prevent lateral movement in networks right from ransomware?

Just given the fact that ransomware does move laterally in a lot of networks?

Does this mean people are not using these tools or have not

configured the networks correctly?

Because it seems if you did all the things that we just talked about, it

should have prevented a lot of the lateral movement that we see in ransomware today.

Prasanna, I'm gonna tell you something that my dad always tell

me, and you have to understand.

My dad grew up in the country.

If a frog had wings, he wouldn't bump his ass every time he hopped.

yes, if you turn on all of these tools, you will cut down on a lot of this stuff.

But does that mean your network's not working correctly?

No.

It just means that we didn't enable all these extra features that we have

to keep track of because I can get four, four ports on a. and plug four

devices in there and they're gonna work is the best way for them to work.

Absolutely not, but I also don't have to do a whole lot of extra configuration.

A lot of people are looking at this from the perspective of, I need to

make sure that everything is able to communicate with everything else.

They're not looking at it like you, like the example you had earlier, Curtis, when

you log into the hotel wifi and I can't talk to anything else on the hotel wifi.

They're not thinking in a, in an isolation mode.

And we're that ship's turning because a lot of people are now realizing that

traditional idea of having a very stiff, crunchy perimeter with a very soft

internal network doesn't work so well.

Right,

ends up happening is that once people get through the perimeter, they have

free reign to do whatever they want.

You, you do have to build these controls in place to effectively

slow them down or to herd them to places that you want them to go.

And that's what a lot of people have spent time developing and working on.

And there's varying degrees of success to make that work.

It has to shift the mindset though.

application people are just turn on all the ports and I'll turn them off later.

When I tell you which ones I don't need, you won't, because you'll

right.

something else.

It's like developers, they're like, I'm gonna load everything I can possibly

think of in the memory so that I know the library that I need is there.

And then you wonder why your, application is consuming like three terabytes of ram.

It's maybe you need to pa pair back a little bit on that.

Yeah.

So it, it sounds like these tools are there.

I think a lot of people do use them, but you talked about, like in the

very beginning, you said that people's heads are gonna start spinning or

whatever, because there is a lot of work involved in implementing these things.

And the moment you flip that switch from, per, from everything is

permitted to only the things that are permitted or permitted, you're

gonna get 5,000 tickets, right?

I can't do this and I can't do that.

And they see that.

They see that very real worry.

and I think it stops many people from implementing this because they just see

it as the amount of work they're gonna have to do to initially implement it.

they're, and they're not seeing the risk of what's gonna happen when

they get a ransomware infection, and then it just goes crazy.

Most tools that are set up like this.

they have a learning mode where they will, you could put 'em in place and

they just sit there and they watch for at

least the first, week or two.

And they're mapping out all of these application dependencies.

the backup system needs to receive traffic on this port for this

application from this subnet.

And then it allows you to carefully craft that rule so that only devices

from this subnet can talk to that server on these ports and nothing else.

And if you let the tool go long enough, you'll be able to like, suss

out exactly what you need to know.

But yeah, that first day you click the switch to from, allow list to deny

a list is just like you're staring at the ticket queue because you're

like, oh, what happens if I, if this machine hadn't been turned on for a

Yeah.

And this

yeah.

Yeah.

It just, it is, it's maddening because you're always gonna wonder if you didn't

get the right stuff, but like you said, would you rather be worried about one

machine that can't talk to another?

Or would you be worrying about the fact that you're getting a phone call from

the CIO saying, yeah, the database has just got encrypted by this new flavor

of malware that we haven't seen yet.

why did that?

Yeah.

Another thing I want to ask you, I wanna move forward

into the ransomware part here.

Although Prasanna, I'm so glad you basically told us to go backwards.

You always, you're really good at that, you're really good at

I try.

anyway, I wanted, so one of the things, so we talked about

trying to limit lateral movement.

Another thing that was suggested was to not

new either new domains, like domains that just recently were created, or

domains that w got recently active, From a DNS perspective, is that still

fall under the networking purview?

or is that is that another world?

it tend, anything that involves names and not numbers tends to float up towards

the application team or the security

Okay.

and the reason for that is because, like you said, like one of the things

that, that we see a lot in security now is it's this idea that you wanna black

hole things that are relatively new.

Like why is this machine suddenly starting to communicate over a d n

s name that I've never seen before?

But it also

requires that your devices have the intelligence to be able to resolve that

because, application layer firewalls will see, oh, you are trying to access

this service that I don't recognize on a domain that I've never seen before.

Whereas a lower level, almost a packet filtering firewall will say,

oh, that's an IP address connection on this port from here to there.

I don't see a reason why I shouldn't be using that.

Gotcha.

You, have to integrate those two things together because like you said,

something doesn't look right here because why would it be contacting a

brand new DNS name that it should, it has no reason to contact or worse yet?

You can ask the people over at SolarWinds.

Why is this DLL suddenly talking to .ru addresses?

right?

Yeah.

when he says new domain names, he actually means domain names that were

like recently registered, not just domain names that are new to your network.

And then also ones that, that were, they were registered but they hadn't

been active or something like that.

So that sounds like that's a d n s there's a d I world, right?

we had somebody on from that.

I think we need to have some, because this is, I think that's, , if you

can reasonably do that, where you could basically push a button, just

like the deny the allowed deny thing.

If you can reasonably say, I, I don't want, want anybody talking to domain

names that were registered 24 hours ago.

I if you could do something like that, it will of course also

create some trouble, tickets.

But I'm thinking far less.

And if you could do that, it stops to command and control, the ransomware from

reaching out at command and control,

the

down.

But the one thing I will say there though, is that you need to make sure

that your users are expecting that change.

Because if it requires you to go out and check a list or, get some kind of

una authorization to go to this domain name, even if it adds one second to the

resolution time, that's one extra second that people are going to complain about

and you know who they're gonna complain.

mm-hmm.

team, because the network isn't working.

Not the d n s block list checker or the application that has this built into it.

Oh, no.

It's the network's fault because the packets aren't

going where they're supposed to.

we used to say back when I was, when I first said that we would

say the problem's under the floor.

meaning, meaning it was a networking problem.

go ahead, Prasanna.

So moving on.

So we talked about how to prevent lateral movement, how to detect these,

rogue, servers that are coming up.

One thing I wanted to ask is, so say you do get hit by ransomware, right?

They're able to move laterally.

What happens next from a networking perspective?

I guess two questions.

One is how do you, how would you go about bringing down your network or sort

of isolating what needs to be isolated?

Like how do you actually figure out what's going on in your network?

And then the second question is, okay, now that you've identified that, how do

you slowly recover from those situations?

Incident response is never fun because it's a whole lot of cleanup.

And, and the first thing you have to do is you have to get people out

of your network because there's, there's obviously, there's the

tools that kind of run on their own.

And there are tools that kind of have to be piloted by people.

So you have to create, limits on the system to be able to stop that.

And fingers crossed that you're not in a situation where your entire

network has been taken down by whatever is causing the problem.

Because I've seen that before too, where not only does it try to laterally move to

infect systems, it also throws up enough extra garbage that you are, it's Inca,

you're capable of logging into any of your

Oh,

So we're lesson number one.

Make sure all your management networks are isolated so that you

always have the ability to use those.

But the first thing that I would.

As I would cut off outside access immediately, I would

lock the firewall in place.

you don't have to run through the data center screaming with your

hair on fire and start yanking cables out like the alias episode.

But you need to be able to lock all of those connections down.

And specifically you need to look for ones that, could be like, from

really weird external addresses, or worse yet ones that are coming in.

Once you've blocked that external access in and out, you gotta do it

in both directions because obviously you don't want anything getting out

because the two things that I can think of are command and control traffic.

If some kind of tool that's being, orchestrated or data exfiltration

Yep.

and you're like, oh, I can stop those file transfers.

Yeah, look up oil rig.

It was, it was able to exfiltrate data through DNS queries.

that's the kind of crap you have to worry about.

So you've gotta lock it down.

Then you have to isolate because that's

And

too.

before you move on,

yeah.

you there?

so how do you do that, right?

is this something where you have to create.

A button to press up, because this sounds like a lot of little steps you

probably need to do this manually, or is there something I can do

upfront that says, in the event of a ransomware attack, push this button.

Hey, gum.

Shut up.

Anyway, in the event of a ransomware attack, press this button and it

does the 10 things I need to do.

what do you think

Some of them do have a big red button press here to terminate

all firewall connections.

But most of the time you're gonna have to create like a checklist or have

a system of okay, I'm gonna go into these rules and I'm gonna uncheck these

five boxes and then I'm gonna hit the terminate connections button to make

sure that no new connections can be made.

Also, if you have a rule at the bottom of your firewall list that

says Permit ip, any, take it out

now because it's not doing you any good.

but more importantly, you have to, all kill switches have to be wired.

, such thing as a magical switch that you can just hit, even if it's one that the

provider has given you what it does.

Does it dump the rules completely?

Does it just suspend the rules until you go in and manually add them?

Remember that could also cut off your connection to the firewall, so

you need to have another way to get into it just in case that happens.

Another reason for an isolated management network, but the idea is that you need to

investigate what your options are because God help you if you really do have to

run down to the data center and yank the cables out, and if that is a case and

hey, it's just as valid as anything else.

Can you make sure that you have the right keys, that you know which

firewall you're yanking out of?

Are there any other exits off of your network?

Because that's another problem that you may run into.

What happens if someone has created another exit off of your network,

either accidentally or on purpose?

And what happens then?

Because you know it's just as easy for me to plug something into your network.

And if there's another way off of it, I'm gonna find it.

Yeah.

The one other thing though, I know you talked about, and it totally makes

sense to kill all incoming and outcoming traffic, but just thinking a step forward,

like when you're dealing with incident response, doesn't that also take out like

your chat channels, your slack channels, your video conferencing, everything

else, like what do you do at that point?

Is it just hope you have everyone's cell phone numbers?

you need to have a plan for out of band incident response because y

it's, it's just like any crime scene.

I need to figure out what's been hit and I need to figure out how

much of it is going to spread.

And you're thinking to yourself like, I can't shut my network down

permanently because you know it's gonna cost me X amount of dollars.

Yes, but it's also gonna cost you x plus whatever amount of

dollars when the next system gets

If you don't

a device that no, nobody's patched it in years.

I'm not gonna lie.

Incident response can work over iMessage text threads for a good couple of

hours while you try to figure that out.

Or, buy your incident response team like those little, hotspots or enable the

data plans on their phone so that they can join their laptop there and join a

Slack instance outside of your network.

Yep.

way nothing is working internal to your network.

Because that's the other thing too.

If you, if this is something that's particularly insidious on a window

system and your incident responders are using Windows systems and they

join the network to be able to do incident response and their laptops get

compromised because they join the network again, you're gonna feel really dumb.

It's the professional, when they blew up the bomb squad truck, it's come

on guys, what were you expecting?

You just reminded me of the, there's a series of commercials and there's

one where the commercial is it's like a horror movie and the, there's a

bunch of kid, it's like the, I got the guy with the ax murderers looking

for the group of kids, and they're like, why don't we go hang out?

Why don't we go hide in that shed over there with all the, with all

the, machetes or something like

that, so we talked about blocking external traffic.

What about blocking internal traffic?

basically the lateral traffic, be due to the, we know we have ransomware

and we know it's gonna try to crawl.

What about blocking that, access?

So that's where you hope that your management networks are, isolated

because the first thing I would do going into a router is shut down the route.

Tables prevent, traffic from being passed across network boundaries.

what you're effectively doing in there is you are containing the damage to one area.

Now, yeah, you're gonna take things down, but if you can isolate that network as

the location for wherever the problem is, you can then bring other networks

back online be relatively certain that they're not gonna be infected.

I really hope that you're not using just regular routing, that you have

some kind of a security boundary there, because that makes it a whole lot.

But you've got to think in, in phases.

Obviously, using the kill switch is gonna take everything down, but then you have

to start, can I bring this back online?

Is this going to be infected?

What would I be looking for?

so I actually have a story about this, this happened last year to my children.

one of 'em goes to the public high school here, and I got a rocket text

message from their IT department saying, please turn off all public school

issue devices until further notice.

And I'm like, uhoh, somebody got hit with something fun.

And this was like the last day before Christmas break or something.

So we went in and we turned off my kid's MacBook, right?

So now, immediately I, because I know what the thing was, I don't want anybody to

like phone home and get infected and then infect the parents networks or whatever.

Okay, no problem.

We just shut it off.

But then I'm like, I wonder what it could.

like I, I'm curious and they've, to this day, they've never disclosed what

it was, but you would get an email like the next week, oh, if you're using like

a corporate phone or if you're using a MacBook, you can turn it back on.

that automatically lowers the horizon of, it has to be something that's focused

on Windows or something like that.

So then you start running through your head of what it could possibly be.

an incident response, you have to do the same thing.

What server got hit?

Oh, it was the database server and it was running this version

of, windows or SQL server.

Okay.

Does that mean that Max can get on the network?

Do I want them on the network?

Is it a situation where even though they can't be infected, they could

propagate something to another location?

there's a lot that you have to go into because obviously the executives are

gonna be like, when can we do back up and.

and if you're a publicly traded company, oh God, the stockholders are like outdoors

with pitchforks and torches and they wanna know when they can get their dividends.

And you're like, when I figure out how much of this data got encrypted

or stolen, and you're always gonna be fighting that tension and you can't

just shut everything off forever.

So that's part of incident response is you've got one team working on figuring

out how to stop whatever infected you, but you've got another team figuring

out how to bring things back online.

That's why we call it business continuity now.

It is interesting about the incident response.

How have you seen cases?

Like how do you actually, two questions I have.

How do you figure out like that, this segment, going back to what

you said, you kill all the routes.

How do you figure out that this segment is safe or not?

And then I guess that, yeah, that's actually only one question.

so typically what, and you're effectively, when you create these

boundaries, it's like looking for the hot potato effectively, because

unless you, like in the alias episode, just go click all the switches off.

Those devices can still communicate to each other at layer two.

Now, where you don't wanna have a problem is that it's in the data.

because if you isolate the layer two data center, now you've got a real problem.

Because if those servers, if it's looking for servers, those

servers can still get infected.

That's why it's actually better to have a, a host route or something

like that, or something that, that kind of isolates that per unit thing.

honestly, like a V switch is perfect for this because if it's not bound for that

host, I'm not gonna let it go any further.

But effectively what you have to do is you have to look for chatter

that's still going on in the network.

Like you, I've shut all this down.

I told my users to disable their machines or turn them off or

whatever, what's still trying to talk.

And then you go take that on a case by case basis.

Oh, this device is still sending traffic that it's, but

it's looking for this server.

Okay, I'm, I can shut it off because I know that it's probably safe.

But then you run into something like, oh, this thing is chattering

an awful lot and it's chattering on a way that it shouldn't be chattering.

that's how I've gone and found hosts that have been infected, but not

by ransomware, but by early malware because they just kept hammering the

firewall with these outbound requests.

And I'm like, you shouldn't

be doing that.

So it's almost like a little bit of detective work.

The good news is that even though the network devices are like dumb from

the perspective of I don't care what application is trying to talk, where

they're really good at telling you that things are still generating traffic.

It's oh, this port is still sending a ton of packets bound

for this address on this location.

And so then you're like, oh, I think something might be up here.

Do you ever see cases where people.

, almost do a, create a black hole on the device itself sync the packets there so

it doesn't go out, rather than having to necessarily do it on the switch.

you can, that's actually a really great way to determine what it's

trying to contact is to create like a null route on the system.

going all the way back three or four years.

Like Mark Marcus Hutchins, that's how he actually stopped a major outbreak

of malware, for all the good it did, and he got arrested by the FBI later.

But he basically black hole the dns.

yeah.

He bought the domain black hole it because if that domain name was

active, then it would stop propagating.

And so he figured that out by saying, oh, I wonder where this is

going and I wonder what it's doing.

You can do that.

And it's actually the next step in incident response, which you've isolated

the system, is I wanna see how it behaves and what it's trying to do.

Cuz that could give me a clue as to what I got hit with and

what they could be looking for.

And that gives you, a little bit of opportunity, but that's

a little bit more of an advanced tool that you would want to use.

just because black holding traffic on a device takes a little bit of setup,

especially if you're fighting against people who don't want you to do that.

Yeah.

Yeah, so it sounds a lot of the things that you talked about in the last

couple of minutes, would be a lot easier to do again, if we segmented

the network in the first place,

Mm-hmm.

We put people with Windows laptops on one network.

We put people with Mac laptops on a network, another network.

We put the phones right?

That are doing the wifi.

We put them on another network.

and we put servers on a different network.

We put, maybe we put servers of a different type on a different network.

So that way you could basically say you don't have to tell

the users to not do anything.

You can just say shut off the laptop, network.

and you shut off the laptop network and so on.

and all the networks that where we don't currently, what we're not looking at.

And then, okay, who's trying to talk?

Who's trying to talk?

Why is this server surfing?

The web

Yeah.

There's nobody over there.

Why is this server going over report 80?

a lot of places already have this by default, even if they didn't realize

they were doing it because you have different classes of devices that

you wanna treat them differently.

Like for example, the the server network, we want to have a little

bit more security in there.

Maybe a little less host to host East to west traffic kind of thing.

The wireless network where all the laptops and the devices connect.

I'm a little less careful about that because I actually have identity

management in place that validates the users when they try to log in.

Maybe I have a guest wireless network for my, for people that come into the lobby.

That one's wide open to the internet outbound only.

So I don't need to worry about that quite as much.

And then, like phones and printers and things like that, that have very specific

things like, I wouldn't enable Bonura in my internal network, but maybe for

the printer vlan I would, because I want people to be able to find a printer.

Open up their laptop.

So they've already created these segments.

You just have to know where the buttons are to shut them off.

So maybe the example is I wanna isolate the servers from the rest

of the network, cuz I think there's something in there, but I can still

leave the wireless network up.

Maybe have everybody join the guest access network and force them all out

to the internet to do, incident response or chat channels or something like that

where I'm, but I'm creating these bounds so that traffic flows one direction

only, or it prevents certain things inside of other areas because, there's

nothing to say like the, the, s IDs that are on printers that are like, set up,

Yeah,

up or something like that can't be compromised.

And then if they can get into your printer network, it's oh

crap, where can they go from?

Yeah.

and Bonjour of course would be the, I don't know how would

apple file sharing.

it is, it's almost like an auto configuration announcement, setting where,

it, and you can thank Steve Jobs for this.

He's I hate setting up printers.

And so basically what he did is he set up a system so that the printers

can announce that they exist.

And your laptop is constantly listening for these.

Bonura is another one of those protocols that is extra chatty and you kinda

wanna put bounds on it so that like you don't have the Apple TV four hallways

down announcing itself to the people in accounting because one, it's annoying.

And two, you never know when you're gonna do something you're not supposed to.

Interesting.

So yeah, I guess a lot of these are really around setting up

that initial network properly.

So then when you do have these issues, you can recover quickly and

identify and then recover quickly.

But if you don't have that initial setup done, then you're in for a world of hurt,

and not just initial setup.

You actually do have to treat the network like a living, breathing organism.

I can't think of a single server admin out there that installs,

windows What are we up now?

20 20, 20 23 Windows, server X, I don't know, installs it

and then never patches it.

Never

Yeah.

it again.

like you people are probably just shaking, even thinking.

, you cannot configure a network and then just leave it alone.

You do have to go in and tweak things and move things and change things.

And, not just when you're trying to fix a broken thing,

Yeah.

have to like, okay, is this subnet big enough for the

number of hosts that are in it?

Should I create routes over here?

It looks like there's a lot of extra traffic going on over this direction.

Maybe I need to disallow that because it looks like it's something

that shouldn't be happening.

if you're not pruning back what you are working on then, and that's the

problem that a lot of the, ransomware writers have figured out, like a lot of

their secrets, if you wanna call them, that are just inadequate it support.

we're gonna hope that you had left this on by default and we're gonna

take advantage of it and use it.

And if you did, sorry, but if best practices guide out there says, shut

that off, and you didn't shut it off, are you in that big of a hurry?

Yeah, we're living in a world where, people don't even

change their default password.

listen, here's the thing, Tom, my plumber's here, I, I got a tradesman that

actually showed up at two o'clock when he said he was gonna be here at two o'clock.

So I gotta , we gotta shut this baby down.

Tom, this has been a great conversation.

so thanks a lot.

thanks for having me.

it's been fun to talk about networking with, with some folks that coming at it

from a slightly different perspective and understanding, what are we trying

to accomplish with it, and in some cases, what are we trying to disallow?

Absolutely.

Thanks again, Prasanna, once again, making me go backwards,

I, you know me, I try, you take one step back, two steps forward

or something like that, right?

something like that.

I

like that.

All right.

And thanks again to our listeners