Emergency Episode: The PyPI Software Supply Chain Attack You Need to Know About

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

This episode, we're doing something actually very different.

We're calling this an emergency episode because what just

happened is that serious.

There's an attack on the PYPI repository targeting light LLM.

A library that's pulled into developer environments three and

a half million times every day.

They took stolen credentials to publish malicious code as the real thing.

This malware is grabbing SSH keys, cloud credentials,

Kubernetes tokens, everything, and encrypting it and sending it home.

We're breaking down exactly what happened, how they pulled it off,

and what you need to do right now.

To, uh, find out if you were hit.

We also cover what to do to protect yourself from something

like this in the future.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Backup, and I've been passionate about backup and recovery and now

cyber recovery for over 30 years.

Ever since.

I had to tell my boss that we had no backups.

Of the production database that we had just lost.

I don't want that to happen to you and that's why I do this.

On that podcast, we turn unappreciated admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the backup wrap up.

I'm your host, w Curtis Preston, and I have with me a, I don't

know, a, a, a cadre of joy.

Uh, first we will start with Dr. Mike Sailor.

How's it going, Mike?

Doing well.

Thanks for having me guys.

We're glad to have you.

I mean, it is gonna, it's a, it's a big, this is an important, I'm, I'm

gonna call it like emergency episode.

We're recording this.

Unlike, normally recording this very before we, uh, publish

District.

course, my trustee.

I, I was almost going, going my trustee

stead.

stead.

You know, I don't think I've ever met a Prasanna or heard of a Prasanna

racehorse, so that would be a first.

That would be a first.

Well, thanks for, thanks for being here early in the morning, um, for both of

us, uh, because this is when we were available because we wanted to cover this.

Uh, and I'm gonna start with a story, uh, a story that happened

when Prasanna was four months old.

It's September, 1982 in Chicago.

got a headache.

You walk over to your bathroom, you reach into what should be

the safest place in your home.

You open a bottle of extra strength Tylenol, take a capsule,

and within hours you're dead.

This happened to seven people, including 12-year-old Mary Kellerman.

They weren't killed by a manufacturing error.

They were killed because someone had tampered with the bottles

on the store shelves lacing them with potassium cyanide.

There was a nationwide panic.

Johnson and Johnson did the, the best, I think a, a gold standard of response.

They recalled 31 million bottles worth over a hundred million dollars.

Uh, I remember this very much.

You, you do too, Mike.

Right?

And this, I, I, I thought of this story because when, uh, when I was putting

together the outline for this Mike, I came in, I came across your, uh, it was

a term that you used in our book that we wrote together, learning ransomware

response and recovery, uh, available.

Uh, for everyone right now.

Uh, and you use the term getting poisoned by your own medicine cabinet

because you go to a trusted source.

Uh, you know, in this case it was an actual medicine cabinet.

In the case of the story that we're gonna talk about today, you use a

library, uh, in this case called, uh, p pronounce it people, P-Y-P-Y-P-L.

Uh, you ingested into your network and next thing you know.

You have a catastrophe.

So, um, what are we talking about, Mike?

I don't know that I've heard anybody actually pronounce it, but I, if.

If they did, I think it would be pipe

Ple,

since it's Python.

Yeah.

Oh, oh, that's right.

It's pi, yeah, yeah.

Ple, yeah.

so, uh, in this case everybody relies, well, millions of, uh, organizations

rely on what they felt was a trusted.

Uh, resource, a trusted medicine, if you will, um, for their daily

updates and, and grabs from a open source, uh, environments.

and on the 24th, uh, they did what they always do and they,

they go to this, uh, this library.

They use this tool.

They download what they thought was legitimate.

Uh.

Uh, data software, uh, or I, I guess a library in this case.

And, uh, it, it, it didn't turn out to be, uh, uh, malware infected.

So, you know, complacency and security are, are not congruent.

Uh, you, you've, your security diminishes the, the more complacent you become.

Uh, and in this case, you know, just going to the same place you always go

to get the same stuff you always get.

Uh, and this time they got a little extra.

What, what did they get, by the way?

I know, I know that it was very, I know it was bad and I, I, I, I read that, that,

that this tool that was infected, uh, down is downloaded 97 million times a month.

But what, What exactly happened to them if, if they put this

tool in their environment?

So what they downloaded was a, an infected version of the light, L-M-L-L-M, uh,

which is a open source Python library.

Um, in, in this case.

There's a legitimate version of that, you know, with the, and we'll get into

hash values and fingerprints of things.

Uh, so, you know that it, it came from a trusted source versus a

modified version of, of that,

Which on the surface looks the same, you know, maybe even the same file size,

uh, looks the same, smells the same, uh, did not taste, did not taste the same.

so when they downloaded it, it came with malware.

The malware was intended to, uh, well, primarily intended to

harvest credentials and secrets.

You know, API, keys and SSH, uh, tokens and, um.

Uh, there were a couple of other things that it did.

It did look for, um, you know, it did try to call home to see if there, if,

if, if having identified a particular target host, if the, the threat actors

wanted it to do something different.

I think there were some, uh, geographic, um.

Implications.

So if, if the malware knew it was in Iran, for example, uh, it did something

different than if it, if it infected something, uh, you know, a machine

or a, an environment in, in Europe.

Um, but yeah, it was, it was designed to, to harvest secrets in credentials.

So.

Thanks for that great summary.

As you were talking, the first thing that popped into my head was in the very

beginning was have you seen the picture or the meme where it's like, Hey, here's all

these cool things built on top, and then at the very bottom it's like this very

small stick holding up everything else.

That's literally what I was thinking about the, as you were talking through

this, how there's all this open source tooling out there, or libraries out

there that people leverage heavily.

Right.

But it's like holding up everything right and.

That's where, like you said, if you sort of attack the common source, then you

now have access to this wide spread of people who are using that common library.

And a lot of times like these open source developers, right, they aren't

paid well, right, they or at all.

And so it's sort of in their good terms and wills that they're doing this.

Curtis, as Mike was walking through this and kind of this sort of attack,

another one that came to mind that we talked about on the podcast before,

do you remember the developer who.

Was wondering why SSH was performing slightly slower and realized

that someone had in, had sort of taken over the open SSH and had

sort of infected it and just luckily he happened to notice because it was

slightly different performance wise.

Right.

But this sort of seems like another one of those like supply chain attacks.

Yeah.

And, and again, I'm, I'm, you know, the, the summary I'm reading

here, uh, there were two versions.

It was like the initial version that sort of pushed out the second version

and the second version, which is 1.8.

2.8 is even worse than the first version.

Uh, but yeah, the, it's that.

You're, you're, you're, you're, you're, you're subject to this

because of, you know, for those of you that aren't developers, the problem

here is what we call dependencies Prasanna . Do you want to talk to that?

So if every single person out there had to write every single line of

code from scratch, right, there'd be no software really out there, right?

And so what a lot of people do is they'll say, Hey, this, uh.

Package, or this library does a whole bunch of things I need to do,

instead of me writing it myself, let me leverage that library.

And so you sort of import it into your code base, and that's what sort of

gets built around, but you don't have access necessarily to everything in it.

You're just sort of treating it like a black box almost and being like,

yep, I trust whatever's in there.

They've done their things and this is the functionality I get.

Yeah.

Any comments on that, Mike?

No, I completely agree and, and.

One of the things that, that these bad guys did, either intentionally

or incidentally, is when they, when they pushed out the first version,

that was kind of a test run to see how likely people were gonna

download that illegitimate version.

In other words, they didn't check, to see if it was legitimate, you

know, hash values and other things.

Uh, and then how much time would they have?

to push a second update.

And, and in this case it went pretty quick.

Uh, but if you, if you don't identify the first one, uh, and it didn't get

shut down, then pushing out a second one gives you, direct access, uh,

through updates potentially, uh, to those that downloaded the first version.

We saw this a lot in the, in the app stores on mobile phones.

You know, bad guys would publish a, a seemingly legitimate and either

entertaining or useful app, uh, to get people to download it and use it.

And then the malware and the other nefarious things would come in

the, in the update to the app.

It's like a, it's like a tracer round.

Yeah.

Right.

Uh, for those of you that don't know what a trace round is, it's a, it's

a separate round that, that's not, it's not, doesn't have a payload.

We're talking like weapons here.

Right.

Um, and it's a separate round that just you, you can see it, you can see

what it does, uh, and you can use that for your, you know, the actual round.

Uh, and in this case, the actual round was very, very, um, um.

Dangerous.

Um, so

oh, one thing about the.

Go ahead.

Go ahead.

One thing about the dependencies that my kid touched upon too, and I

think is important is as someone who is leveraging this library, right in

your code base, you have control about sort of how often you want to update.

So say you are only developing once a month, right?

You may not want to pull, constantly be pulling a new version of that

library into your code base, right?

Or if you're doing it once a year.

you have the ability to control how often, and so like Mike was saying,

right, the fact that they pushed out a version sort of implies that hey,

people are sort of constantly updating to get the latest and greatest, because

sometimes that's what people want, right?

New functionality's always coming out.

You want the latest and greatest.

Mike, uh, if you could jump into how someone would know that, um.

That they, that they had this, uh, specifically, or maybe in general,

how would you know that you have some sort of malware in your environment?

Do you wanna walk people through what we're talking about there?

I.

Sure.

So.

If, if you are, if you're concerned that, that, you know, you might be, uh,

affected by this supply chain attack.

In other words, you're, you're, uh, you often download

these pipe Python libraries.

Or actually, know, th this attack group, uh, team, um, it team CPC?

Um,

PCPI thought.

PCP.

Yeah.

I was close.

Uh, thi this attack group, uh, team, uh, PCP is, is really

hitting every open source, uh, project, uh, repository out there.

They, you know, they started with, uh, you know, the trivia

scanner, check marks, GitHub.

Uh, so, you know, the Python libraries were a, an a good

next, uh, pivot for them.

So if, if your organization often, uh, interacts, downloads, uploads, even, um.

And definitely any that have, authenticated access to these

repositories, uh, you might be concerned that you've, you've been

affected by this, this attack.

So common things to look for.

And one of the reasons that this attack was identified so quickly is that there

was a developer that went, Hey, why is my machine, uh, running so slow?

We're doing this weird thing.

And he dug into the services and the, the processes running.

And that's when, um, that developer identified.

Something nefarious is going on.

and that's usually the, you know, the, the symptoms much like getting

sick, the symptoms, you know, tell us that there's something off.

so anything obvious, uh, would be a good indication that something's weird.

You know, don't get paranoid right away.

But definitely look into, uh, anything out of the ordinary

running on, on your systems.

the easiest thing to do.

'cause you're paranoid doesn't mean nobody's out to get you.

Mike, just, but anyway, but go ahead.

Just because you're not paranoid.

Uh, I don't, yeah.

yeah, yeah.

Okay.

Go ahead.

It, it goes either way.

But yeah.

So the easiest thing to do is go and look at, uh, in this case, if you downloaded

the, the light LLM uh, uh, version, uh, go look at the hash value for that.

Compare it to the.

The light, LLLM, uh, trusted hash value.

If they're the same, you're good.

Uh, if they're not the same, then you know, either assume you've been

compromised and just change everything.

I mean, that's the least effort.

Um, you know, fastest, fastest path to, um.

To confidence.

but if you wanna take the next, you know, few steps is, you know,

start looking through your logs.

Uh, if, if you're working in a cloud environment like AWS or

Azure, uh, there are tools there.

Um, so the AWS, uh, cloud uh, and log analysis tools, uh,

one, they have to be turned on.

Uh, and that's just good security, best practice.

Anyway, turn your logging on, log as much as you can.

Uh, and keep it as long as you, uh, as long as you're able to, because those logs

help tell the story when we need to figure out if something weird has happened.

go look at your AWS and Azure logs.

Um, there are some specific IOCs like calls to, uh, third party libraries,

uh, um, ex uh, data xFi, uh, volumes.

So, you know, if, if your environment doesn't send data.

To other places, you know, test dev environment is, uh, encapsulated and, and

you're just working with it in that, and, and all of a sudden something has changed.

You know, that deviation from normal behavior.

Uh, those are other symptoms and, and things to consider.

realize too that sometimes malware doesn't behave.

uh, it sometimes lies dormant.

and a lot of malware, and I think in this case too, uh, install some back doors.

So if, you know, you simply delete the package downloaded, if you simply, you

know, change credentials, uh, sometimes that back door can, uh, can open up

and, and regardless of all those other things you did, uh, bad guys could

still have access to your data and, and, uh, and create some havoc that way.

Um,

talked about network,

Oh, oh.

Go ahead, finish.

I talked about network indicators.

So, uh, unusual traffic to unusual destinations, IP addresses, VPN

tunnels, uh, tour exit points.

Um, you know, if, if you're a, a development shop in the, in Texas as

an example and, uh, you know, you're, you're fairly isolated or, or, um.

Domestic and you, and you're seeing all this international traffic,

especially around the dates, uh, you know, the last couple of days.

Uh, those are also things to be considerate of.

Um, if you, if you're a Kubernetes shop, uh, there are some specific IOCs, uh,

in this, in this attack for Kubernetes.

Um,

You wanna define IOC, although people should know, but what is an

so.

IOCs are indicators of compromise.

So when we talk about attacks, there's, there's two primary acronyms we use

IOCs indicator indicators of Compromise.

So those are the things that we would go look for in our environment that

are specifically or generically.

Related to an attack.

So known bad guy does things a certain way with certain tools.

Those tools leave fingerprints, uh, in either logs or active behavior.

And those are IOCs.

so hash values, uh, certain file names, uh,

I IP addresses, uh, URLs, things like that.

And then we have TTPs, which are, uh, tactics, uh, techniques, and.

Procedures.

So what are the bad guys?

How, what's their mo like, how do they, how do they conduct the attack?

What they do this first and this second?

And they use these things.

Uh, so those are TTPs.

Um, so reviewing those, uh, IOCs and, and they're, they've already been published.

If you just Google.

You know, the, the light LLM Supply Chain attack, IOCs and

TTPs, you'll, you'll get those.

But we're also happy to put a, a short one pager together and put, push that

out to our stop ransomware website.

Um, so hash value comparison of the, the known, trusted version

or versions, um, of the light LLM.

Um, review your logs.

For any known deviations from expected behavior, review your network

traffic, if you have that capability, looking for behavioral anomalies.

Um, and if at all you suspect that you've been compromised.

The best practice, I know it's a, it's a, it's a, it's a pain, but the best

practice is restore from a trusted backup.

Don't try to just onesie, twosie, you know, delete it from the

registry, delete the image, delete the the app, turn off a service.

Those are all I. One-offs, you know, kind of stomping out the fire

with your, your leather moccasins.

And I only say that 'cause it, it actually happened to me and

that's a whole other story.

Uh, but the fire will flame back up.

Uh, if, if you don't take the right approach to, to truly,

uh, starting from scratch.

That's one of the only ways to get,

You,

what are you saying?

you put out a fire wearing moccasins or you

I did not.

I did not.

My former, my, my former mother-in-law started a fire on my porch by putting

a cigarette out in a plastic, uh, pot.

The wind kicked it up, caught the pot on fire, melted it to the house.

She came out, saw it was on fire.

She kicked it off the porch, out into the dead grass, caught the grass on fire.

Uh, all the while.

Passing fire extinguishers and a water hose and all these things.

And she decides the best course of action is to jump out into the grass

and try to stomp it out with her fake leather moccasins, which did not work.

Uh, and she made several trips, uh, from the grass out into the

back, into the kitchen with a a, a simple watering pae again.

Bypassing fire extinguishers and watering hoses, uh, to try and put this water,

this, this fire out with a watering can.

Um, and she failed.

The, the fire simply burned itself out.

Um, but it did ruin her moccasins, uh, her, um, her Mickey Mouse t-shirt.

Oo.

Um, and a lot of my yard.

No,

Um, go.

As you're talking through that, right?

It's sort of helpful to know, okay, here's all the things you should be looking for

to understand, okay, were you compromised?

But given a lot of companies these days, right, you kind of have developers

who go off and do things, is there a mechanism to actually figure out like,

am I actually impacted by this breach?

Right?

Like, do I even, am I even using, uh, the light LLM.

Library in my environment or across, say, all of these developers or whatever it is.

Sure.

Uh, so if, if you're a security team and, and the development team is

not a, you know, they're not open and, or, or, or you're, you're,

you're concerned about their.

Transparency.

Uh, you can absolutely, again, go review all of your logs, review the systems,

uh, depending on your, your security capabilities, uh, whether it's endpoint

like a, a good, uh, EDR in malware solution like huntress, uh, or uh.

A complete environmental, you know, technology environment monitoring solution

like we provide with stellar cyber, uh, where we can collect data points from

tons of data sources and correlate that in one dashboard and go, you know, and

we can input these IOCs and it'll look through all of that data and go, yes,

no, if yes, it maps, you know, what endpoint, what firewall did it go through?

When did it happen?

What was the user involved?

All these things.

And then if it did trigger something, even if it, even if the system at the

time thought it was, uh, you know, uh, uh, you know, safe, I can, I can

tell it, I can tell the system, uh, if light element LLM was downloaded,

uh, you know, that library and, and it did these things, map that out for me.

Uh, and so even if it was.

Perceived safe.

I can now go, I can go through my, my dashboard and look at

all of the potentially impacted systems and, uh, networks and

users involved and all that stuff.

Um, absent all of those things, start the firewall.

well start with talking to your developers, uh, and

then start at the firewall.

'cause all your, all your traffic needs to go out through the firewall.

Um.

Hopefully you've restricted, you know, developer bringing his own hotspot to

download stuff so that it doesn't get blocked by your security policies.

But, you know, there's, there's so many different depend, it depends situations,

but start your firewall if you're, if you feel like you're, you can't get a straight

answer out of the, the people that might have been involved in downloading this.

Uh, another thing that I saw in suggestions was immediately revoke

and rotate every secret that was stored as an environment variable.

Um, you wanna talk about that?

So this was, this was actually a pretty, uh, uh, a, a potentially

pretty, uh, impactful incident.

Uh, so when, when, when the malware came down, uh, it, it

immediately started stealing secrets.

Uh, and so that could be a secret, within an application, uh, you know, trusted.

Uh, uh, trust between applications, trust between servers, trust

between network segments, keys, SSH, Kubernetes, uh, golden ticket theft.

I mean, there's, there's so many things that this malware could have done because

it doesn't know what it has access to.

It was just gonna try and steal everything.

Um, and so absolutely every machine.

That was potentially compromised.

You need to do an inventory of everything it had access to and everything it,

all those secrets it could have stored.

Absolutely.

All those things should be changed.

Revoke it, change it.

And I say revoke it because open sessions are not impacted by changing secrets.

So you've gotta revoke those open, se those open sessions first,

and then change, uh, change all your credentials and reissue

tickets and all that good stuff.

It's a lot of work.

It, or it sounds like a lot of work, you know.

It is, it is a lot of work.

is a huge breach.

It's why, it's why I wanted to jump in on this and we're, I'm actually gonna.

Publish this early.

Uh, normally we wait till Monday to publish our episodes.

I'm gonna publish this one early because this is, this is huge.

I mean, when I heard that it was 97 million monthly downloads, and

then I heard just how bad the, you know, their stealing secrets.

Um, you know, I mean, I, I do go back to that Tylenol scare, right?

Tylenol was such a trusted source.

Um, and, uh, then, you know, it was literally killing people.

Uh, and um.

So it was, you know, everybody immediately went and got the Tylenol

and ripped it outta their shelves.

Right.

Um, and, and again, the, vendor in this case did the right thing, right?

They, they didn't hem and haw, they just said, give us back a hundred million

dollars, um, um, you know, of, of Tylenol.

Mike, we're gonna talk about like, action items for the future.

Anything else that specifically regarding this attack for the moment?

Well add, I'll add, uh, with, with regard to this attack, you know, it

was, it was found or identified fairly quickly just, you know, within a few

hours, uh, that that developer, you know, was, was concerned about his,

his system not running, uh, And that's when he figured this, these things

out, and it got communicated and they, they took, they took that, um.

That infected version down.

So what I'd be interested to hear is what, what is the, the true impact of this?

You know, if it was only available for a couple hours, um.

know, how many organizations were impacted.

Uh, were, are there any follow on, uh, attacks based on this level

of effort it's gonna take to fix?

So if I was an infected or an impacted organization, much like Prasanna

mentioned, all those things, like I talked about, is things you need to do.

That's a lot of work, but that's not something you're gonna be able to achieve.

In, in an hour or a couple hours, maybe not even a couple of days,

because changing some of that stuff may impact operations, right?

So things stop working when you change trusts and credentials, especially,

uh, from a, an operations perspective.

So, I'll be interested to see.

Uh, we're here, uh, in the coming months.

What the fallout from this, even if it was only a couple of hours.

And if you take the 95 plus million downloads a month and you divide

that into the hours, so I think there's 720 hours a month on average.

I just did that.

I, I did great Mind signal light, that's 134,722 downloads per hour.

So.

could be significant.

Um, and usually, you know, it's not multiple de developers in an organization.

It's usually like one person is in charge for updating libraries

and do doing stuff like that.

so that could be 130,000 environments.

Um, so that could be huge.

And again, the level of effort for.

to this episode.

I hope so, and I hope they take it seriously because once we steal those

credentials, a lot of times those credentials now on a, on a production

side, like with service accounts and things like that, those are

often randomly generated some weird.

You know, hodgepodge, you know, random, alphanumeric, upper,

lower, all that good stuff.

And they're usually pretty long.

in some cases, especially from, uh, you know, developers and, and non-security

focused people, those passwords are coincidental with other things.

And so if bad guys stole credentials in this case, there's a good

chance that developer uses that password or those credentials or

those tokens for other things.

When will we learn?

So, um, especially if you're a, a, a man, you, you know, you're a, you're

a development shop where you're doing development for multiple clients, very

often, you know, your credentials are the same across multiple organizations,

which is also bad practice.

But, um, yeah, if, if they don't take this seriously and, and.

You know, gonna kind of go scorched earth on rebuilding and remediating this.

Uh, it could be, it could continuous, uh, continue to be bad

for a lot of these organizations.

And, and by the way, uh, even though the, this attack did attempt to, uh,

appears to attempt to exfiltrate some data again, the fir, the primary focus

of this was credential harvesting, which would be a type of attack that

an initial access broker would take.

I just want the credentials.

I'm gonna sell those to other people that knew, uh, that know how to

do or, or are interested in, in various other types of attacks.

you don't know what an in initial access broker, uh, is, listen to our episode.

What is an initial access?

Initial access broker?

I'll put a link to it in the, in the, uh, episode description.

so let's talk about some action, some action items.

And the first thing I'm gonna put on there.

gonna say that, you know, for the future, you need an inventory of your environment.

You need an inventory of what, what software you're using and

what dependencies you have.

Uh, you know, so I, I, I live in, I live in, you know, California and

one of the things we have here, if you sell food, you have to keep a

list of all of the suppliers, uh, of where you get the food so that when.

Um, you know, there, there's a, you know, a what, what's that?

What?

An coli on spinach, uh, outbreak.

And they say it was these suppliers you can immediately know, uh, you know.

And so I'm gonna say that the first item that that should be

on your list is, is an inventory.

Um, and, um, and of course you're gonna audit the, your current versions.

Um, for, for light, LLM, let's talk about,

Wait,

de.

so, so you talked about an inventory.

Could I also say an inventory with processes to make sure people don't do

things outside of sort of like what's approved or like Mike had said, sort of

like a security team who's kind of vetting the libraries before they sort of are

allowed to be used within an organization.

Yeah, absolutely.

That makes a, that, that makes a, a, a or that makes a lot of sense.

Um, and then we're gonna talk about, um.

The dependency pending and, and hash, um, the use of hash values

Prasanna , do you wanna talk about that?

Yeah.

So what happened in this attack, like Mike said, right?

The attackers updated the version of light LLM.

People started downloading it, and one of the reasons that happened is sometime.

When people set up to do these pulls of these dependencies in libraries,

they'll just say, give me the latest.

Right.

Rather than saying, okay, I want this particular version such that you

know that, okay, that's the only one.

So if they had done version pinning, which is to specify a particular version

for my build, then they wouldn't have been able to download the latest version

because it's only gonna pick one version.

Right.

And so that would've prevented that issue.

Another thing I know, Mike, you alluded to this earlier, is

also sort of the hashing, right?

So when you are downloading a version, confirm that yes, this is the LA latest,

or this is a version that I care about and here's the hash that goes with it.

So I know that it is a valid, uh, version of that library.

It's something that I expected.

You wanna prevent sort of the supply chain.

Attacks from immediately impacting you because if you can sort of delay when

you take the latest version, it gives other people time to react and sort of

uncover these issues before you get hit.

It's kinda like when you download the latest, uh, software updates

on your phone or your car, right?

Some people are like, oh, I wanna be day one, like right as soon as it's available.

Versus others are like, Hey, let's wait till it's baked out.

And they sort of worked out all the bugs.

And I'll take like the dot two or the dot three of that initial major version.

You know, as a person who's who, you know.

I, I am not a developer, right?

Uh, never have been a developer.

I have written some Pearl, I've written some pretty mean pearl in my day.

But, uh, but I am not a developer with dependencies and such.

That doesn't mean I don't know what app get, you know, and, uh, you know, and, uh,

so I'm gonna be thinking about that every time I. You know, have, I'm downloading

a tool and it says the first thing you need to do is update all your libraries.

Right.

Um, but the, I'm, I'm dependent on the people that wrote that tool to

do the things you're talking about, because this is something that the

person writing the tool has to do.

Right?

They have to, they have to.

know, you said specifically call out particular versions and also, uh, uh, so

again, because I'm not a developer, is that hash, is that going to be provided

by that tool, or is this something you're going to create when you download the

trusted version that you're familiar with?

The vendor, the developer,

Okay.

the trusted source would provide the hash value.

Okay.

Okay.

So, uh.

speaking, speaking, as a non-developer, that sounds backwards to me.

Like, so like some point we have to trust this, this vendor, right?

So this is, this is, so we have a trusted version and they're going

to, how do we get that trusted version in the first place?

How do we determine which version is a trusted version?

Yeah, and we, so we've gotta, we've gotta establish it as a trusted source.

And so whether it's directly from the vendor and, you know, they go

through any number of certifications as a trusted source, you know,

their processes, their controls.

So that could be an ISO certification or a, a SOC two, type two, uh,

audit certification, or, you know.

Something like that, that helps us as consumers, uh, feel confident

that they're doing business in a secure and, um, you know, good way.

But it's just a piece of paper, right?

And it's, it's some third party that it happened at some point in time.

You know, I could have gotten an ISO 27,001 security certification

over everything I do yesterday.

Well, today's a new day, and I could have changed things.

And so there's always a level of diligence, regardless of how

much trust you put in something.

Um, and there are several organizations I'm I know of that whenever they

download something new from a trusted source or not, they run it in a

sandbox environment for a period of time, to determine operational impact.

Is this gonna change?

Or, you know, is it going to.

Kill a process or is it even gonna work with our systems?

we see this a lot with Microsoft patches.

You know, those, those are all well known for creating issues.

Uh, well this could, this could very well follow that same methodology.

Whatever you download, you need to sandbox it, for a period of time

before you implement it in your, even in your test dev environment.

Um, but yes.

a lot of, go ahead.

Yes, you're the, the people that created something and they want to be, you know,

they wanna maintain their reputation and the, the integrity around their product.

Uh, they will often publish the hash value of that file or that object.

and it's very difficult to, uh,

it's very difficult to falsify a hash value.

Right, right.

I've done, I. a bit of work with, uh, you know, living where I live.

I've done quite a bit of work with, uh, the um.

Uh, biotech folks and they definitely have this concept of, you know, verified

systems that have, uh, it's not the term verified, it's been a while.

They have another term for the systems that have been verified, uh, and they

very much, you change a single thing and the environment and they have

to reverify the entire, uh, system.

So.

That's something, a lesson that we could take from them.

I am of course, going to suggest that if you haven't hardened your backups,

now's the time to harden your backups.

We talk about this a lot in the book and the, the, and, and all of the usual things

of, of MFA and password management, and hopefully pass keys moving forward to

pass keys, uh, and separating, right?

So, uh, you know, putting, uh, a different, um.

and authorization system for your backups.

I know it's a pain, but just like everything else in security, uh,

you know, it secure, you know, good security and convenience are not

necessarily in the same, uh, you know, in the same, uh, um, ballpark.

And the number one thing here that I'm gonna, that I'm gonna be

harping on is immutable storage.

Right.

So this entire time though, we talked about this library, which was supply chain

attacked, which were stealing credentials.

Could you help our listeners understand the link between having

immutable backups and this attack?

Yeah.

Great.

So the, the, the, the, one of the things you, if you go back earlier in the

episode, one of the things Mike said was restore your, um, what, whatever

this is from a trusted backup, right?

From a backup that you trust the, the thing with the immutable backups

is one of the things that, that just hurts my little heart when I see it

out there is when a ransomware or a malware attack happens, and you see the

little phrase at the end of the story.

And the backups were also corrupted, right?

So having immutable just means cannot be changed.

And the standard by which I judge immutable backups is if you can delete

them, If you as an admin can delete your old backups, then those aren't immutable.

At least that's, that's the gold standard that I'm putting.

So, um, configure your backups in such a way.

Talk to your vendor.

How do I do this?

Configure your backups in such a way that even you, the super, super, know,

God level access on your backups.

If you cannot delete backups before they're supposed to expire, then

you actually have immutable backups.

If you're anything less than that, you're immutable ish.

I'm not saying it's crap.

I'm just saying the closer you can get to that level of immutability, um, you

know, and, and Prasanna you always bring up, you know, when we start talking about

actual immutable storage, there's like the compliance mode and the what are the two

Governance.

governance mode.

Right?

And the governance mode is the more stringent one, right?

Yeah.

And so, uh, basically that, that mode of, uh, and we're talking, in this

case, we're talking about like, uh, object lock in S3, that if you enable

the, the stricter mode, even you, the owner of the account cannot delete.

Objects before they're supposed to expire.

And if that's the way your backups work, then that's truly immutable.

And if that's the case, then the bad guys can't delete or encrypt or corrupt

your backups, which means that you can then use them to restore this library.

Right.

That's a, that's a great, thank you for, uh, for making me.

Uh, get up on my soapbox and, uh, and explain that.

and again, I, I, I, I mentioned it already, but,

um, of course we're, you know.

Basically in your whole environment.

Look at MFA.

And again, literally the last episode, Mike was a little bit

rolling his eyes on, on MFA, but not, he doesn't think my MFA is bad.

He just, it's not perfect, which is why we're trying to move to pass keys.

But if you don't have MFA, if you have passwords in the wild.

That are, that are securing things that are important.

And you don't have MFA Mike, do you wanna explain what, what, why, again,

why is MFA, what does it do, uh, in, in this situation when somebody does

happen to harvest your credentials?

What is the purpose of MFA?

MFA is supposed to be a second, layer of security.

And we, we consider it an out of, out of band.

Of band means, you know, if I'm logging into my computer, the MFA doesn't

pull up on this computer, it goes to my phone or a, or an authenticator

app or any, another email address.

Uh, and that's important because if, if bad guys also capture your MFA

token and they're already at your computer, or they're already in your

environment and they already have your credentials, then your MFA is is.

Useless it.

It's not providing that extra layer of security.

MFA is also, uh, a, a good way of determining if your

credentials have been stolen.

Uh, so if you get a, uh, a text message or an email on your phone that says,

here's your, here's your MFA key for Facebook or LinkedIn, you're like, well,

I'm not logging into those right now.

else is.

Uh, and so that's a good indication.

You need to go change your credentials and, and maybe even try

to figure out how that happened.

But the problems with MFA.

Is if I'm on my computer and I log into something and it says, Hey,

uh, you need to check your MFA.

Device or your app put in that code.

The next, very next thing that happens is usually why MFA's

value diminishes significantly.

And that is a popup, uh, window or a, a subsequent webpage that says,

do you want to trust this device?

Do you want me to remember you?

And if you click yes, then you don't have to do MFA for that anymore.

So at work for your bank, probably not your bank, but you know,

LinkedIn, Gmail, whatever it is.

If you click remember me or Trust this device, you have saved

that MFA token in your browser.

And so now bad guys just need to get you to go to a bad website or

potentially even download some malware and they will harvest that MFA token.

And if they can compromise your credentials by getting you to

click a link, they can also.

Create a new session as you, with that new MFA token whereby bypassing the

value of having MFA to begin with.

they only do that on the computer where the MFA token was, was generated?

It's only valid there.

Right.

They, they cannot do it.

They, they, they're not limited to the computer.

It was generated on.

They just need access to the browser or the, the computer to take the

saved MFA token out of the browser.

I can do that remotely.

I can do that remotely from anywhere in the world.

I can get you to go to a bad website, will then suck that

MFA token outta your browser.

Or get you to click on a phishing email or go to a website to

download, you know, malware.

And that malware similar to this light LLM, uh, will harvest,

uh, those MFA tokens for me.

And, and so just to make sure I understand, so if they get that, if

they've got your credential, you know, your username and password and that saved

MFA token, even though that token was created on this laptop, they can use those

three things to log in as me anywhere.

High Probability

Curtis is freaked out.

Why you always doing this to me, Mike?

Trying to, I'm trying to get your hair to match my hair.

There are things you can do, uh, from an organization security perspective

to limit that in, in, in other words, uh, as a security admin for a company.

Uh, I can go into Office 365 as an example and say, you

know, no, no concurrent logins.

You know, Mike can only log in one time.

I can say, you know, uh, Mike can only log in from domestic ips.

Or we block all, you know, bat known bad ips, you know, China,

North Korea, um, et cetera.

there's a list of those that's published every day of every, every week.

there are things that we can alert on.

Uh, and so if Mike's logged in and Mike logs in again from a

different app IP address, and we would, uh, es especially one that's.

Very far away.

We call that impossible travel.

Uh, so if you, if I've logged in from Texas and, and, you know, 10 minutes

from now someone logs in from even, you know, Kansas, impossible travel.

And so that should be alerted on and potentially even automatically blocked.

And if, if we want to take a very strict approach to that.

Uh, whenever we see that impossible travel without explanation,

we suspend the account.

Yeah, yeah, sure.

Mike isn't gonna be able to work for a couple of minutes, but Mike's about to get

a phone call and say, you know, where are you, Mike, and what are you working on?

Um, so we can clear this up before things get bad, and that is the key

to incident response these days.

It is how fast can we respond to weird stuff before bad things happen?

Speaking of alerting, Mike, uh, do you want to talk about how the kinds of things

that people should be doing to make sure that they are aware, that they get these

alerts when something like this happens?

What, what should they be following?

How and how should they be doing that?

I'll tell you just about every tool that's out there has free training.

We just we're too lazy to take it.

You know, we're such a consumer driven culture.

We just want the latest, greatest, use it now, share it with my

friends, and move on with our day.

very rarely set time aside to watch the video or read the manual.

and I'm, I'm guilty of that too.

Um.

But it's, it's all out there.

So if you wanna know how to secure your Gmail or your, don't be using Yahoo

or Hotmail still, or definitely not a OL, but if you have a, whatever your

account is, there is guidance out there from whoever that provider is to help

you secure it and, and be more aware of when weird things happen, for example.

In Gmail, there's a security tab where you can see all the last logins

and IP addresses and time and date.

And a lot of people don't know that you can do the same thing

with iCloud, uh, for your bank.

Very similarly, uh, there's a security tab, when were the last

logins, what did I do, you know, uh, from an activity perspective.

and those are just the authentication pieces.

Well, what about the behavior?

So what if someone was able to.

Log in, uh, to one of these accounts.

Uh, and like my bank, what if, what if they start to transfer money or

they steal my credit card and they're, you know, they're buying tires.

In Utah, there are ways of setting alerts.

You just have to be willing to manage it.

Uh, so for example, everything over a dollar on my credit card, my

debit card, I get a text message.

I do the same.

Right.

So, and I'm okay with that and in fact, it's kind of cool

to see how fast that happens.

I'm at the grocery store, I just said, please remove your car.

And I got a text message.

That's awesome.

Um, and very similarly, I was

I, I hate, I hate to cut you off, but Prasanna 's gonna turn into a pumpkin.

The, those are great things.

It wasn't the question I was asking.

My

sorry.

is No, it's fine.

Uh, what I'm talking about is what?

kinds of things I, as a company should be looking for?

Where I get these alerts that, that a security incident is,

it's like this one is happening.

That's what I'm talking about.

So you gotta define the role first.

You know, who's gonna be responsible for this?

Nobody wants to look at logs and alerts all day.

They've, it's usually someone's part-time job they do at lunch or at the end of

the day, or first thing in the morning.

And that's it.

It's not real time all the time.

you should have a dedicated person for this.

you need a. Defined incident response plan.

Uh, and so for every alert I get, I need to follow these procedures

every alert, uh, even if it's false positive, you've gotta go through

the process of determining it's false positive and documenting that.

So in the future, someone goes, Hey, that thing happened.

How come we didn't do something about it?

Well, I looked at it and if I was false positive, you know, it wasn't,

I think.

do I, where do I get these alerts?

This is my question.

it wasn't a legit, uh, alert.

So go into all of your systems.

if, if you've got an environment where someone's managing your stuff,

uh, you need to turn on event logging for as much as you can, consolidate

all those logs into one place.

And there's a variety of things you can do, like a SIS log server or, uh, uh,

there's some free open source, uh, log consolidation and analysis tools like Sim,

monster, uh, SIEM, monster, um, there are.

Um, a variety of, uh, automated scripts like python's.

One of them, powershells one where you can use those to, uh, to alert on specific

event IDs and to know what those are.

Google it, what if, what security and event ID should I be concerned about?

And you'll get a list of those, or, uh, call, a call a managed service provider.

If you don't have the, the skills and the staff to, to support that activity, they

can consult with you about what you have.

How to configure it, what to do with it internally, uh, and how

they could help if you need, um, you know, additional skills and staff,

especially if it's a 24 7 thing.

Cybersecurity managed services today are so affordable.

Everybody should have it.

Uh, there's just no excuse and if you don't have it, it's gonna impact your

ability to get insurance in the future.

If you have a breach like this, your, your damages from lawsuits are gonna be a lot

bigger 'cause you, you weren't diligent.

Uh, but yeah, there's absolutely.

number of ways of collecting this information, being able

to automatically alert on it.

You just have to have the people and the procedures available

to, uh, to take action.

and Mike, I guess, oh, as Curtis had asked that question, one thing I was thinking

about is like, Hey, I work at a company.

All these issues are constantly happening.

Where as a person do I go to understand, Hey, where are the latest breach alerts

or other things like that happening.

That's kind of what I was thinking.

I don't know, Curtis, if that was what you were intending to, but.

is, that is the, that is the question I was asking Mike.

Was just, I just wanted you to say like cbe.org or something.

That's what I was looking for.

I apologize.

I'll start, I'll start clarifying my understanding of

your questions in the future.

Uh, so cisa.gov, cisa.gov, uh, is a good, uh, site.

Um, there are, there are a ton of, uh, Twitter or X profiles.

Uh, just search, you know, cybersecurity threat, intel and,

and x and you'll find good accounts.

So that's, that's people that, that's all they do, and it's very timely.

In fact, a lot of stuff will show up there as a quote unquote proof of concept before

ciso or some of the other agencies will actually, uh, publish it as a, a known

vulner, uh, known exploit or an attack.

It'll be a proof of concept that someone has, uh, observed out in the wild.

There are vendors out there that provide free threat intelligence.

If you're part of critical infrastructure, uh, get with your state.

Uh, there are state information sharing and analysis centers that you can

subscribe to for free, and you'll get daily, sometimes hourly updates on

threats, and if you're part of the critical infrastructure working with

the, the state ISACs, uh, they'll even help, uh, with your response.

So, um, there, there's just too much to get into from a, from a resource

perspective, but cisa.gov is a good one.

I know we covered a lot of these in the book, um, you know, buy our book.

Uh, but, uh, let, we will, I'll, if you could give me a list of those and

we'll put 'em in the show description.

'cause this, I think this is a big deal.

This is an op, it's, once again, it's an opportunity for people to get scared

to, to then go, you know, evaluate.

Evaluate their life.

Uh, all right.

Well, thanks Mike.

I This is great.

Thanks for, thanks for getting up.

Uh, well, it's for you, it's not, not as early, but, uh, Prasanna

, definitely you're your early bird.

Like you're not an early bird, but, uh,

I am an early bird.

Yeah.

Okay.

Well.

Yeah.

I'll just say thank you, Curtis, for, for get, for getting on the

camera at seven 30 in the morning.

'cause this is not, this is

Have you had your coffee yet?

I I've had two cups, actually, by the way, I had a cup of Java in a Java mug,

uh, very old Java mug that you can see if you watch the YouTube version of this.

All right, well, thanks, uh, thanks you two.

And, uh, that is a wrap