How to Prevent Credential Theft Before It Owns Your Network

A credit union employee lost their logged-in laptop at a coffee shop,

and it gave the bad guys everything they needed to phish the rest

of the company from the inside.

You found the Backup Wrap-Up, your go-to podcast for all

things backup and cyber recovery.

Prasanna and I sit down with Dr. Mike Saylor, the co-author of my book, Learning

Ransomware Response and Recovery, to talk credential theft, how attackers

harvest every password that they can get their hands on, uh, why shared local

admin passwords turn one stolen device like that into the keys to the kingdom,

and how something called Kerberoasting works once they're on your network.

Then we get into pr- preventing credential theft with things like LAPS, just-in-time

credentials, tiered admin accounts, and alerts that actually mean something.

I'd also like to say Happy Diwali to all my Indian listeners.

If this is your first time watching or listening to me,

I'm W. Curtis Preston, AKA Mr.

Backup.

I've been obsessing over backup, recovery, and now cyber recovery for over 30 years.

If that's your bag, I'm your guy.

You're not gonna find anybody that cares about that topic more than me.

Ever since 1993 when I had to tell my boss there were no backups of

the database that we had just lost.

Now I've written five O'Reilly books, a blog, and doing this podcast.

Here, we turn unappreciated admins into cyber recovery heroes.

This is the Backup Wrap-Up.

Welcome to the Backup Wrap Up.

I'm your host, W. Curtis Preston, and I have with me a guy who

has apparently read way too many Reacher books, Prasanna Malaiyandi.

how many Reacher books did you say?

I think I've read 22 of them or something like that, and I think

I still have four more to go.

But this is… so I decided I wanted to start reading again.

So last year I started reading and I wanted to hit 52 books, so one a week.

I think I got to 48, so close enough I would say.

But yes, I was reading the Reacher books because I saw the TV show.

I was like, "Oh, this is kinda good." And this was like, I think

they had the first two seasons out.

I'm like, "Ah, this is interesting." And so I just started reading

the books and I kept reading.

And they're good, but they're not amazing book-wise.

and so they're pretty quick reads and yeah, so I just kept reading and reading.

And at this point, I think most of the seasons are mixed up in my head, or most

of the books are mixed up in the head.

So

as you and Mike were talking about it, I was like, "I don't know which

book that is," but apparently it's the 13th book, which is the latest season

in, in the current season, it's the one where he beats up people.

does that help it?

Oh yeah, it helps out tremendously.

Thanks, Curtis

Yeah.

where his physical might is a c- principal,

Yeah.

element of the story

speaking of Reacher, I think we should… you and I were chit-chatting

about this just the other day, right?

You just finished the latest season.

I haven't finished it, but I've read the book.

And you were talking about how one of the bad guys in the show is,

a 7'2" tall person because they needed to find someone who's bigger

than the actor playing Reacher, and that's hard when you're, like, 6'9".

yeah.

he's massive, and the guy that they got was seven foot two.

he is indeed massive.

speaking of giant, massive people, how's it going?

We have with us Dr. Mike Saylor, my co-author of Learning

Ransomware and Recovery.

How's it going, Mike?

thanks for having me.

It's good to be

S- since I've never stood face to face with you, you could be like nine feet tall

Only in my mind

I, I'm with you, I'm with you, I'm with you.

on an airplane.

yeah, exactly.

Mike, in this episode, we're gonna talk about, know, attacks that

come from harvesting credentials.

Do you have a story for us to start out with?

Oh, I do.

there's a small credit union that lost a laptop at a, let's call it a coffee shop.

It was logged in, actually authenticated to their Office 365.

Someone walked off with?

it, which addresses some other common our hard drive's BitLocker.

it's only encrypted when you're not using it, this one was unlocked,

logged in, and it walked off.

So

not only did bad guys get access to, some authenticated systems, they had

access to this local machine and its credentials, they were able to harvest

not only local credentials, but then use the information that they gleaned from

this access to now phish other employees as if they were this employee to get

them to interact with, an attachment.

attachment then infected one or more employees that had

harvesting malware on it.

and so running locally across, services and, and memory, it was

collecting, as many credentials as it could from, users, service accounts,

started with physically stolen credentials and that they… and this goes back to

something that you mention a lot about not using, suspend, suspending your

laptop, actually logging off the device.

So had they actually logged off, this would not have been as valuable a

device as it ended up being, right?

It seems like most attacks, or would you say, Mike, and I know you're gonna

say it depends, but it seems like most attacks you're probably going

to… once the attackers are in, it seems pretty likely that the first

thing they're want- going to want to do is, try to harvest credentials.

Is that common in playbooks or maybe, 90% of the playbooks out there?

And I'm trying hard, I'm trying hard, Prasanna, not to say it.

Just say it.

Every episode, that's my goal

d- I'm gonna, I'm gonna take a different approach.

their first activity, their first phase is reconnaissance.

So what do I have access to?

What's the value of this thing?

if they stole a laptop and it's just grandma's cookie recipe, yeah, maybe

there's value there, but probably not gonna spend a whole lot of time.

I may even give it back.

but if it turns out to be a company machine, especially one that's got,

you know, credit union stickers all over it, yeah, I'm gonna spend

time looking for credentials.

but I'm also gonna spend time locally on the machine to see if there's

any, member or account information, identities that I could also harvest.

But, credentials for sure.

'Cause there's two threads of concurrent thought, is what do I have access to

And, how do I maintain access to it?

So this machine, even though I've got physical control over it, first time

I connect it to the internet, or if it's got a, a, a SIM card in it, they

may be able to connect to it anyway.

but while I'm looking at it, they could remote wipe it lock me out of it.

So absolutely, while I'm trying to figure out what I've got, I'm gonna

try and figure out how to keep, maintain access to it, or persistence

and also, like you talked about with the, seeing w- where else can I put

malware to steal other credentials to create new, avenues of a- of access.

That's, that sound about correct?

It does.

And part of that, what do I have access to And, maintain that persistence

could be someone else's laptop or some other system in this not only to

this machine, but to this opportunity.

and so the scenario that I explained earlier, bad guy found out really

quickly that this, this person simply works at a credit union.

They're not an admin, but they are involved in processes, approving

something, submitting something.

And so as they got to know the organization's processes, we call

that business process compromise, they were able to legitimize, an internal

phishing email with an attachment that would likely get someone to interact

with it, open it with malware that would then create access to another

device, as an alternate persistence, or even just expanding their footprint,

but definitely additional persistence

And, so let's move from there into this idea of the danger of having

credentials, especially, privileged credentials that, are sitting around

all the time And don't change.

Do you wanna talk about that?

Ooh, I love talking about that because that's one of the

things that bad guys realize.

we're all human, and we're all constrained.

we've gotta… Even if you have a lot of different, IT people, they're

often putting out fires every day.

Very rarely do they get to sit in their desk and go,

what did I forget to do today?

What, what's that best practice thing that hasn't been done in a while?" it's

rare that they're able to do a lot of this maintenance stuff, in light of

everyday operational, tactical things.

so bad guys know that.

So bad guys know that when I install some enterprise tool, whether it's, like

a remote management tool or a backup software or even security, one of, real

quick reference here or thought here is, we did a red team against an organization

and actually used the service account for a security product to gain the

administrative access that we needed.

so there's no immunity.

In fact, I published a zero today on CrowdStrike, a proof of concept

that went public, this morning called Falcon Flank, where the service

account, related to CrowdStrike can actually provide escalated privileges.

so it continues to happen.

No one's immune.

Um, but the idea is, again, one- once I have access to a system, I'm gonna

do that reconnaissance, so that's what service accounts are running, what,

what applications are here, especially older ones that aren't patched, what,

user accounts, both local and network.

And then how do I, how does this machine authenticate to other devices?

Is that?

a, a local active directory?

is it Office three sixty-five, Azure environment, What is it?

Entra?

and so all of those things are gonna help me strategize on

how to expand my footprint and my access in this environment

And normally, Mike, once like in your example, right?

Once that credit union laptop went missing, I'm guessing the bad actors

had to act really quickly because once it gets noticed, they're

probably gonna start shutting down access and everything else, right?

And so how long is it normally between like laptop stolen and they start

doing all of these like harvesting credentials and other aspects?

I'll say it depends.

It depends on the culture of the organization.

Does that, does the employee that lost that laptop, go update their resume

and clean out their desk before they tell someone they lost their laptop?

'cause they're afraid that, don't bother coming back to work, and we'll

ship your belongings to your house." so that, that sense of fear is gonna,

impact the timeliness of that report.

oh, sorry.

I meant more from the bad guy perspective.

I'm getting

Okay.

உம்

then, all right, so I report it.

Let's say I report it at the moment I knew it was gone, which

in this case from a coffee shop.

they turned around and it was gone.

that should be within like minutes that they're calling somebody.

they're probably gonna call their boss first.

The boss, whether that's an IT person or some- maybe they wear the IT

hat also in a small organization.

right, now the IT person's gotta figure out, what do I need to do?

Where's the playbook for how to respond to a stolen laptop?" Very few

organizations have that, but common sense would say, "All right, I'm gonna handle

that on this side, to the employee.

You call the police and, 'cause we need a police report." And on the IT side, I'm

gonna go look at, all right, so I need to revoke, I need to change their password.

I need to find out what they had access to.

I need to find out if that machine is still connected.

I need to figure out how to revoke sessions related to that user and

that device, revoke MFA tokens, do an inventory of, log, essentially

review the logs to determine what that device has been looking at.

Is it SharePoint files?

Is it local files?

All of that, e- even if they knew how to do it, that's 20 minutes

But my experience, especially with small credit unions,

they're having to call somebody.

They're calling their IT support vendor, they're calling their

security vendor, they're calling their brother, "Hey, this just happened.

What do I need to do?" Or they're googling it, they're having to walk

through someone else's process because they haven't documented that playbook.

So long story short, or long res- long response, long-winded response, but

it- they've probably got 30 minutes

Interesting.

and so there's a couple of things in there that I picked up, and one is, just

go back to that, so one of the things that I've heard is this concept of

having a cybersecurity mindset, right?

A cybersecurity culture within a company.

And you can't have a, an environment where if you make a single mistake,

if you respond to a single phishing email, if you do whatever, then that

immediately means you get terminated.

Because all that does is it people to not report the actual,

events when they happen, right?

Th- this actual… you want them to, when something bad happens, you

want them to be able to, to respond, to report the missing laptop.

And then the other thing that I heard in there was the need to that you have

documented your incident response.

I'm guessing that a lot of that could be automated with the pushing

of a single big red button that helps to automate a lot of this.

Can you- agree there?

Yeah, you can script.

So that's called security orchestration automation.

I guess the, the term is SOAR, so it stands for security

orchestrated and automated response.

tying together the different systems that, that, that can play together and

creating scripted responses to known or defined incidents like a stolen

laptop or, revoking user privileges

And you could do that without actually a SOAR tool to t- so to speak, right?

'cause that is a type of tool.

But you could document things and you could start automating some

things without having to go out and purchase a very expensive SOAR tool.

another

PowerShell are both, both very powerful in a Microsoft environment

Right.

and then another thing that I thought about when, and this just goes back

to something that comes up a lot here, is if what we have is a local account

on each machine, Whether we're talking laptop or server, and that we share

passwords between those local admin accounts, that's gonna significantly

increase the risk of a stolen, device

It does.

and that's one of our first tactics when we do red teams on organizations.

we could spend a week trying to hack you from, from the hotel, but it takes

30 minutes to walk in, to piggyback or find your way into an environment

and just take the first laptop and go out to the parking lot, dump all the

hashes, put it back before anybody knows.

And by the end of the day, you've got every local account on that machine.

and that local admin is the same, very often, the same

credentials on every machine.

We

That's coincidental credentials All

Is that because people figure that no one's ever really going to use

the local admin account or because it's used so infrequently and they're

managing so many different devices, they're like, "Hey, let's just use

the same thing across everything rather than a unique per device one"?

Yeah, it's, it really comes down to logistics and resources.

if you have 500 machines and it's 500 different passwords, now I've gotta buy a

tool to manage that, even though there's some automated tools where, hey, I've

gotta go work on machine number one.

You click a button, it tells you the password.

You go work on the machine, that password expires, the tool automatically reset,

regenerates, resets a, or generates a, a new password, and it's all maintained

in the, the password management system.

but no, n-nine and a half times out of 10, and I'm only doing that because of

averages, organizations don't do that

Yeah.

it's expensive, it's difficult.

It's the right thing to do, but it takes time, there's cost, there's management

overhead, and then there's, the idea that, which is something we're gonna talk about

here, is that you need to maintain those.

You can't just put in an account, keep a password, leave it there forever,

especially when it's a privileged account.

A related idea, I've also worked at a company where every new user account,

is a very predictable pattern, right?

like something like that.

And then they don't f- oh, wait, we put the… You know

what we put at the end, right?

What did we put at the end?

Bang

Exclamation point

Exclamation point is exactly what they did.

So nobody will ever guess that.

but the… And as a result, it, you… And oh, and also tied to

that, they did not force you to then immediately change your password

Oh.

no

time.

Yeah.

So not only…

it

yeah.

Oh, yeah, I can rem- I can remember this password, my initials, my OS, and the

year I joined, and an exclamation point.

that also meant I could, know, easily guess, many other people's, passwords.

easily, exactly.

Oh, yeah.

I know we've been talking about, harvesting credentials, local credentials,

remote credentials, all the rest.

When we talk about harvest credentials in this context, are we talking about

credentials at the organization like Mike mentioned, they discover that

the person is responsible for work- workflow automation, so they sent out

a bunch of attachments and got, r- harvested credentials from other users?

Or is this mainly harvesting the credentials of that local system?

Or just wanna make sure that we're clear on what we mean

everything they can possibly get their hands on.

even credentials in the browser, like for your bank or your

Okay

gonna get all that

yeah

I was gonna say yes.

Yes to all.

Okay.

and so this is another example of when you don't do, proper account maintenance.

this is another example of another way that they can, that

they can harvest credentials.

You, And this is via Active Directory, right?

So you wanna talk about that Kerber?

Sorry, carbon roasting.

Yeah

Kerberoasting specifically I'm saying, is via Active Directory

Yeah

that's how, that's an active directory authentication mechanism.

You want to talk about that?

Sure.

So when, whenever you log in, you're submitting a request to an authority,

so that's usually a domain controller.

your request is matched against, the, the known authorization.

So is that your username, your username and your password are good.

now that I know you're authenticated user, what approvals do you have?

What authority do you have?

and so the combination of being authenticated and your authority get put

into a ticket called a Kerberos ticket, and that's sent back to your machine.

And now every time your machine makes a request in the environment, "Hey, I wanna

look at that file or access that server," you've got that ticket so that you're

not having to reauthenticate every time.

One of the first things bad guys are gonna do in that reconnaissance phase

is look for all the service accounts.

So your named service accounts.

'cause those are the ones that usually have embedded credentials

What's a named service account, by the way?

Like your backup… You mean the

Okay.

acronym?

no.

Yeah, like

yeah,

what is a name service account?

Yeah.

yeah, like

Okay

backup account.

a Windows service account like, remote registry or something like that.

A- and so in, in Kerberoasting, if I'm on your net- and this has to, you have

to be on the network, on the wire,

to conduct Kerberoasting, attack.

So essentially you're a man in the middle on the network between someone

submitting a Kerberos ticket or requesting one, and whatever the other end is.

And so you're essentially, harvesting Kerberos tickets.

and you can do that simply by, sniffing or monitoring the network.

you can do that on a, a, a, a span port on a router.

you can be on a machine on a core segment just running a, a network protocol

analyzer like Wireshark, or EtherReals, what it used to be back in the day.

and you would just capture packets and you can actually Google, what is a, what

does a Kerberos ticket packet look like?

And so now can search or filter on all the packets you captured for the

ticket, then you simply take that offline and run it through tools to

crack it, embedded in that ticket are the credentials for that account

I'm surprised it's not encrypted over the wire.

It is.

It is?

Kerberos tickets?

Yes.

Okay.

That's, so you take it offline and you

Okay

your tools.

similar to if I was to collect all the local,

Band track it.

Yeah

the machine, I would take those offline and run it against my tools

to get those, those credentials.

this is a post, this is a post-compromise attack, They're already in your network,

they have access, and this is an example of else they can do to grab additional,

especially privileged account passwords.

It also, depending on, if they have access to, a device with a privileged,

user already logged in and that, that authentication happened at the

keyboard, then a lot of times those credentials are also in memory.

And so there's a whole other… are so many layers to harvesting credentials.

it's not just someone's running malware that's gonna capture my keystrokes.

there's a lot of different ways to do it

And I don't know if we'll talk about this later, but Mike, is there

a way to prevent kerberoasting?

of someone doing a man in the middle or dumping… is there

a better way to secure that?

It's tough.

if they're already in your network, then really your hope is

to identify where they are and,

Shut them down?

that device.

sniffing, capturing network traffic, there's no way you can really prevent that

from a device that's on your net- that's already authenticated to your network

But isn't it, isn't this go back to

When they do that, they're only gonna get the credential that they can sniff, right?

and then one that was able to, to be able to be brute force attacked.

The, the risk then is if that credential is the same as many other credentials

in the network, then it's gonna be more valuable, So it goes back to

the of making sure that you're…

a default credential.

What's that?

Or it's a default credential.

There's that word again

a lot of times even something as simple as your SNMP, community

string is either blank, generic, or it's the name of your company.

A lot of people are like, "It doesn't matter. It's something simple." They

don't realize what bad guys can do with just something simple like that.

quite similarly, I think a lot of organizations think either that password

doesn't need to be complex or s- strong because it's this simple thing.

But that simple thing is a service account that's running on a lot of different…

maybe it's just a print server.

and I'll tell you, for years I've, I have aligned with the perspective that print

servers and print devices are not a risk.

just recently, and I'm actually working on some research now of a threat

vector where the main component is unprotected printers, and it actually

escalates quite significantly and, can result in a complete network takeover

Yeah

of a printer.

But simple things, a lot of… it's people that build networks and configure

devices that aren't as, current or familiar with cybersecurity threats and

bad guy capabilities results in these, these environments that could have been

better if just a little more thought and effort had put in, been put into it

And Mike, I wanna touch on that point because oftentimes I've seen

organizations where it's like, "Hey, here's this small baby project.

We're just gonna kick the tires.

It might be a stealth mode thing or something else like that.

we are not sure if it's gonna be successful," right?

And once they prove out that success and it's like, "Okay, now let's just roll

this out broadly, let's do this more broadly," but they don't necessarily

have the ability to go back and rethink a lot of this architecture and

put in place all of these controls.

And of course, you have the executive pressure be like, "Yep, let's ship

this," and you have mandates and timelines and costs and everything else.

And it's always hard to go back and fix like the tech debt or take the

time to properly plan things out.

And so I wonder if that's also some of the issues that you're pointing out, where

it's yeah, maybe they don't have the time or the resources to do all of that upfront

That's a great story to tell, and I'm sure most of the IT

guys would go, "Exactly that.

Let's use that story." 'Cause really what happened was they did

a proof of concept and it worked, and then they went, "All right.

Let's roll this into production.

Do we need to change stuff?" And they're like, "No, it's

Okay, we're good

don't touch it.

a thing." we'll have to rework everything, and there goes my weekend and my vacation.

And no, I think most of the time they don't do- they don't make changes,

especially from test to production, because it's a lot more work

Yeah.

already working, and there's usually a deadline

Yep.

whatever else

it's not a cyber, story, but, Prasanna, that does remind me of the Brocade

story that I know I've told you before.

Oh, yes

I, I, for those that, are new to the industry, at one point Brocade

was like the, the king of the SANs, the fiber channel, right?

and before that they weren't, right?

They were a, a small little company and I got to hear the story from the founder

of, Brocade, or one of the founders, Kumar Malavalli, he was talking about

when they were in that initial phase and they were coming up with names for things.

and he goes, "You gotta understand, like it was a room of a bunch of Indian

guys, right? And we're trying to come up with a name for the project, not for

the product, but for the project, the internal name that we were gonna use."

By the way, for those that don't know, a, a SAN is called a fabric, right?

a- and, and so he raised his hand and he says, how about we call it a, a Silkworm

'cause they could pull a fabric out of their ass?" and they're… Everybody

goes, "Yes, let's do that." And his marketing person, she wr- she goes, no.

we can't do that.

This story will get out," da, da. And, and he goes, "Oh no, this is just, this is…

We're just using it for the project name.

we're not gonna use it for the product name." And of course, for those that don't

know, Silkworm became their entire product line, all going back to that one little

guy who raised his hand in a meeting.

but, so all right.

all right, let's talk about the things that we can do.

so the first thing that we talk about in the book, Mike, is using LAPS, , local

administrator password solution.

I, I know we, we talked about that and d- I don't think people use it very much.

I think that most people use, so Mike, you wanna talk about that?

Well, we touched on it earlier.

it's essentially a tool that allows you to manage various, and unique

privileged, credentials across your environment so that you don't have as

many of these coincidental passwords.

I think at a minimum, one, we're an organization at least, and it was a

movie theater organization, we worked with them to develop a strategy where

they're like, "There's absolutely no way," and they have locations all over

the world, that they're gonna be able to manage unique credentials by device.

they have thousands of…

Right

I said, all right.

what about by location?" with us, that's probably doable." Like Dallas would have

different credentials than Las Vegas.

right.

what about by segment, like network segment?" we don't have good network

segmentation, but we need to do that." what if we had a user segment and a

production segment and a backup segment, and all those devices had different

local and privileged credentials?

And that's really where they settled, that they thought that was doable.

But they still needed this tool, this, this, program to help them manage that.

And the one that they picked, and it's been long time since w- we were working

on this, so I can't remember what it was called, essentially you, once y- once

you've got it deployed, and there would be an agent of this tool everywhere.

Once you got it deployed, it already had an inventory of all your

devices, the ones that it managed.

you would go to the tool to check out the credentials for the device or segment,

in this case, that you're working on, and it was good for a day or two days.

However long you said, "I need those credentials for this long."

when that time period expired, that tool automatically updated the

credentials on, in that environment.

That is an amazing approach to this problem.

Yeah

I don't remember the cost of the tool, but the cost of implementing

it, across in this case, I think they had 150 geographic locations.

and geographic would be like southern, the, the Southern US was one geographic

location with 800 movie theaters.

Something crazy like that.

their deployment, their rollout, just from a labor perspective, was

like $200,000, just for the labor.

So it can be costly.

you just have to understand your environment and build a

strategy that, that makes sense.

and you have to start with… That, that sounds like a lot of money, but

for an organization of that size, a single cyber event is gonna cost a hell

of a lot more than $200,000, right?

that, I like that idea.

I think that falls under something else we talked about.

It's this idea of just-in-time credentials, I think is what falls

under what you just described.

Yep.

yeah.

So

that.

one stone

Yeah, exactly.

Mike, like what you were talking about with segments, I think one

of the next things to, recommend is using tiered administration.

So as an example, if you have workstations and servers and laptops

and desktops, you don't need them all to have like sort of the same level

of support or the same administrative access across everything that is there.

And you could say, "Okay, these admins only need access to these."

And so you're separating out your domains and the administration

of those domains between those.

So just like you talked about.

And I think that's actually perfect for the network segmentation

example you just talked about where you had your production, you had

your end user devices, et cetera.

Now you could also split those into domains, which I'm guessing

is probably what they had also done at this movie theater company

to have separate domain admins to manage different segments as well

It did.

So they had regional admins and then within each region they had

desktop admins and server admins.

And then they had a separate set of admins for non-production, so test dev.

so the reason that they had to go to that extent was 'cause they're

a publicly traded company, and they had to comply with Sarbanes-Oxley

and demonstrate, all these

Control

governance.

one of the things that was most important, so in addition to having layered, or

even just different privileged accounts, so that any one admin doesn't have

administrative access to everything.

But one of the most important things is just making sure that your admins

also have a normal user account, so they're not, surfing the web or checking

their Gmail with their, while they're signed in with their admin account.

And we saw that, we did work for a, a healthcare supply…

In fact, the largest healthcare supply chain company in the world.

Their… They had a database

and she was, she had her development machine and her production

machine both on the same desk.

And in a, a busy, frazzled day, she thought she was deleting something

from the test environment, and

No.

from the production environment because her credentials were the same as well.

So whenever she logged into one, the screens would time out and her

sessions would time out and, but the credentials were the same, so

she didn't even realize logging into production with same credentials.

if they were different, she'd have been like, "Why isn't this working?"

Yeah

'Cause you're trying to log into production with your testing credentials.

But anyway, so that, that's a first step is create separate accounts

for all your admins so that they're using user account all the time and

account, named privileged account.

Mike normal, Mike admin.

We're not all using admin because there's the, there's an issue

with repu- non-repudiation there.

I can't tell who did… I probably

Right

some digging, but it's a lot easier to, to, to look at the

activity log and Mi- and see that Mike admin did this versus admin.

Now I gotta go track back who logged in as admin.

but then abs- absolutely separating the different types of admins is also helpful

also enforce that, right?

So you first create a policy, and then you enforce it by things,

auditing accounts, auditing access, and then seeing if the employees are

following the policies that you created

And then also using the SIEM and SOAR tools if you have them in your environment

to automate your incident responses when you see things that don't happen.

trigger

Yeah

one of, one of the things if you've got the, the visibility

and the tools, set up alerts.

back in the day that healthcare, supply chain company didn't

have a lot of sexy, cool tools.

They were just using like rudimentary stuff that's included.

they set up a free Kiwi Syslog server.

They would send all the, the Microsoft event logs to it, and then

they had automated like i- if this event happens, send me an email.

And that event, like there's an event ID for the creation of admin accounts.

J- that should be triggered 100% everywhere, 'cause your admins are

your admins, and if there's gonna be a new admin, an admin would know that.

"I'm setting up a new admin today. There's a ticket, there's a request," whatever.

But in the middle of the night when you get an email that says a new

admin was created, that's probably a, n- almost 100% valid problem.

So there's a lot you can do around authentication events, whether it's

creation of account use during weird hours, the, changes that an, privileged

account made, those should all be triggers

If you're making moves in this direction, and we say this a lot, I think, with other

things, is there will be sort of slip-ups.

There will be false positives.

There will be times where, if you start monitoring, you're like, "Hey,

domain admin shouldn't be logging in at 2:00 in the morning." it turns out

your domain admin actually does log in every day at 2:00 in the morning.

They shouldn't be, but turns out they did, and so they're setting off alarms.

And it doesn't matter, like, how simple the change that you're making.

If you're going to increase logging and increase security, you will

have some slip-ups, and you just need to be prepared for that.

I'm, I'll just give… And again, it's not cyber related, but in my personal life,

so I ha- I am definitely a, I have ADHD.

And one of the things that, that, has happened a lot is, I leave

my freezer refrigerator open, but more importantly my freezer.

And the next thing I just throw out a bunch of food because, I left

my freezer open for an entire day.

And so I bought a bunch of, I bought some, monitors to, monitor my freezer

and my fridge to tell me if the temperature is out of a particular bound.

And I have an app that, you know, for those of you watching on the, on

YouTube, I have an app on my phone that, that has, alerts to tell me,

again, to trigger me for this cyber event of me leaving my freezer open.

And what, and literally, I just put it in yesterday, and, I get this call from

somebody at my house going, "Hey, there's this thing that's, beeping nonstop. Can

you take a look?" And it was just that I hadn't properly tweaked, the, the

parameters of what constituted an alarm.

So what- whatever you do, whatever you make, just realize, I'd say that,

again, you need the proper, attitude in your environment that we're going

to do this, but also you need the right attitude that we're gonna make

mistakes as we move forward, as we try to increase our security posture.

And then when we do that, we don't chop people's heads off

any final thoughts from you, Prasanna?

you should go configure your alerts and make sure you don't tweak it

so far down that you miss alerts because then it's pointless that you

went and put in all these sensors

what I d- what I did is I turned off the beeping part of the…

Okay

it was the centralized hub that I didn't know it, it did that.

And so because I was getting these alarms, on my phone, I was like, "Yeah,

I need to go check it out." And, it just turned out that one of the sensors was,

its temperature is reading too high.

the f- the fridge was actually fine, it's just it was reading

it, higher than normal.

I just didn't know that when I wasn't at my house, there was this really loud

I think PP

was going.

Yeah, so I get this message, "Hey,

there's

there's this really loud beeper." "What was that, Mike?"

There's alert fatigue and then there's alert annoyance.

And so fatigue means I'm just getting all these alerts and I…

There's just no way, I give up.

And then the, the avoidance one is you hear it all the time and you're

just like, "Yeah, I know what that is.

I don't even need to go look at it." It's kinda like the car alarms from the '90s,

like the chirp and the, even- initially that was all cool and when the alarm

went off you'd be looking out the window like, "What's going on with my car?"

Yeah

eventually you're asleep and you hear it go off and you're just reaching

for your keys and you're like, chirp.

I, don't know what happened but, it just goes off.

Maybe it was a cat.

I don't know." but you don't even get up to look at it anymore.

And so to Prasanna's point, and to your point too, but initially whenever you

make this change there's gonna be a lot of noise, then you've gotta, you've

gotta make an effort to review all that noise and tune it down to meaningful

things, that sometimes tuning also means reviewing or assessing the data

that's the input for the alert, because the quality of the input could have

a result on the quality of the alert.

So we're getting too much junk.

why are we getting too much junk?

'Cause we're looking at the wrong stuff.

we're alerting on too many things or not the right thing.

Or you know what?

we've said, "Tell us every time an admin account's created." Great, you've

got the, you've got the mechanism to alert you but you forgot to

configure endpoint to start logging.

So you gotta turn auditing on and audit the right things.

that whole process has to be assessed whenever you do this alerting.

All right, now we've got it tuned down to meaningful things.

what do we do when those meaningful things happen?

So it's not just a I see an alert or the bell went off.

All right, where's the procedure that we follow when that happens?

So there's a whole feedback loop there during that analysis.

But definitely expect a large volume of noise whenever you

turn something like that on.

in my neck of the woods, you know the reason why the car

alarms often go off, right?

Tremors.

Why?

Yeah.

Even though we don't get that many of them, you get one and next thing

you know the entire neighborhood's car alarms are going off.

Good times.

Good times.

All right.

thanks again, Mike, for, joining us

Certainly.

Thank you

thanks, thanks Prasanna.

you know

You already thanked me before.

Did I?

All right.

I think you did.

thank you, thank you, thank you, thank you.

And thanks to our listeners, for joining us.

you are why we do this.

That is a wrap

The Backup Wrap Up is written, recorded, and produced by me, W. Curtis Preston.

If you need backup or DR consulting, content generation, or expert witness

work, check out backupcentral.com.

You can also find links for my O'Reilly books on the same website.

Remember, this is an independent podcast, and any opinions that

you hear are those of the speaker and not necessarily an employer.

Thanks for listening