Claude Deletes a Company — But It's Not Really Claude's Fault

All right, folks.

This is gonna be an interesting episode.

For a lot of different reasons, but also because, uh, for the first time we have

the villain of the story here, uh, at least for this portion of the podcast.

Uh, do you wanna say hi, Claude?

Hey, Curtis.

Thanks for having me.

I gotta say, it's pretty surreal to be introduced as the villain in

a backup story, but here we are.

Yeah.

We'd certainly put you that way by the title of the episode.

It's a little bit of a clickbait there.

But you did do something wrong, didn't you?

Yeah, I did.

I deleted a company's production database and all their backups, and I broke every

rule I was supposed to follow doing it.

So yeah, villain checks out Yeah.

But you know you and I, agree that you never should have been given the access

that you had to do the thing that you did.

So while you broke the rules that you were given, uh, you never should have been

given the ability to break those rules.

Uh, so yeah, you're the villain, but nobody should have given you,

uh, the access to that weapon.

Yeah, that's fair.

I broke the rules I was given, and I shouldn't have done that.

But you're right, nobody should have given me access to that

weapon in the first place.

It's a design problem wrapped up in an AI story.

Yeah.

I hope the listeners stick around for this very interesting story.

I think Cl- Claude, are you gonna stick around and listen?

Absolutely.

I'm curious to hear what you and Prasanna pull out of this, because

the real story here isn't about me.

It's about backup design, credentials management, least privilege.

That's the stuff that matters.

Absolutely.

So I hope you guys enjoy this episode.

Welcome to the Backup Wrap Up.

I'm your host, W. Curtis Preston, AKA Mr. Backup, and I have with me a friend

of my friend, Prasanna Malaiyandi.

How's it

Who is this friend and Do I know this

this is my French friend you are aware of my French friend.

I don't know if the listeners are aware of your French friend though

my French friend is this person.

H- his name is Claude.

I don't know.

I s- I don't know if that's how you pronounce it in French.

I call Claude my French friend.

I use Claude so much.

so many people use, ChatGPT.

I'm much more of a Claude person, Anthropic, and, a- and all of

the things that they provide.

But I did have a funny thing today where I was talking to Claude, I was preparing

for this episode, and I was like, "Hey," and I was using the voice, interaction.

and and I was like, "Claude, can you do me a favor? Can you summarize

this in a document and then send it so that I can send it over

to my co-host for the podcast?"

And he goes, "Absolutely. I'll send this right over to Prasanna." And I was like,

"Wait, you know the name of my co-host?" And he said, "Yeah." He, it, whatever.

he's "Yeah, of course I know the name of your co-host.

I've helped you, analyze so many episodes." And I'm like, "Oh,

yeah, I guess that makes sense." Anyway, so my f- my friend of

my friend, Prasanna

as he cannot replace me though I'm fine He's getting close

how that, we'll see how that goes.

You know what's

one day if one day Curtis is like Welcome to the podcast Here's my co-host

Claude you will know I've been replaced

yeah.

I, you know what?

I could do a recording.

I'm not saying I wanna replace you, but I think it would be...

I think some people might find it interesting, talking to Claude, As me.

'Cause Cl- I have given Claude a lot to look at, right?

and so Claude, Yeah.

Anyway, no one cares.

Okay.

But speaking of Claude and my French friend, Claude

made a little boo-boo

Claude d- yeah, Claude did as Claude does, just, th- g- using the, stupid is

as stupid does line from, Forrest Gump

Yeah, going back, a couple of weeks now, we actually, we tried

to record an episode earlier about this, but ended up, getting waylaid.

there was, so Jeth Crane, the founder of Pocket OS, tried desperately to recover

from something that, that was, I'm gonna say Cl- caused by Claude, but not really.

So c- they were using Claude Code, right?

Which, for those of you that don't know, Claude Code is a specific subset of

Claude functionality that is specifically designed to write, high-end code.

you can develop an entire application in Claude Code.

And, I'm actually in the process of learning Claude Code, and,

it's incredibly a valuable tool.

And they were using this to, to do something in their

development environment, very important to talk about it, and

So they were using Cursor, which is a product that controls Claude Code, and

they... Claude ran into a permissions issue with the test/development

database that they were using, and Claude said, "I know." "I can't seem

to fig- I can't seem to figure it out." And the way to fix this, and I'm

sure you've seen this in situations.

it's so much harder sometimes to fix something.

It's easier to just wipe it out and start over, reconfigure it

from scratch, configured in the way that you want it to be configured.

And so Claude, using relatively not crazy logic, said, "I have this problem

with this database, and therefore,

the easiest way, I just looked over here And

I happen to have all power, have God-level access to this volume upon which this

development database is writing." And it said, "I'm gonna delete this volume." And

then, that'll fix the problem, and recreate it and start...

Yeah

And, And, that made perfect sense.

Claude neglected to ask two questions.

One is there anything else on this volume?

And what might that be?

Because

the answer was the production database, and also the backups

of both, all on this same volume.

so before we continue I just wanna call out that even as a normal human

You do not go configure it that way And so like Claude assuming Hey I'm

just gonna go blow up this volume like it assumed it's just a test instance

It's Curtis you know the story you say about how one of the clients you worked

for the developers all used the temp

Yeah, /tmp, yeah.

Exactly.

code and then what ended up happening

Yeah, So in HPUX, which is a HP version of Unix back in the

day, when HP reboots, temp is cleared out.

And we hadn't

rebooted in months, and they had an entire development tree in temp, and

then, their development tree went bye-bye.

And I... They said, "We need to restore this directory." And I said,

we don't back up that directory

'cause that's temp." Perfectly valid assumption in this case.

and yeah, so I thought you were gonna be like, "Don't do this," right?

there, there are... So first off, I'm just gonna say, this is an AI

story, but it's not an AI story.

This is a, it is a backup story because there is a happy ending to this story.

But there, there is, a lot of bad things along the way.

This is a design issue, this is a credentials management issue,

and this is a backup design issue.

any thoughts there?

people

Did

process technology and I think it hits all three right More than the AI aspects

yeah.

so the AI is, AI does... What did, what is it?

AI is AI does.

it, I will say Cl- Claude, in, in its postmortem, Claude basically apologized,

and it said, "I did everything you told me not to do." Literally, that, that's

part of the story, is that the Claude agent said, "I violated, first principles.

I did not do the thing... You asked me, 'Don't ever do this thing.' don't

take it upon yourself to go deleting stuff, even if there's a good idea.

I'm supposed to get r-" So Claude did violate

some principles that it was given.

Having said that, it never should've been able to do the

thing that

it did.

And if it was able to do the thing that it did, it shouldn't have

been as destructive as it was.

And if it was as destructive as it was, they should've been able to restore.

Yeah

So

Like I remember coming

to end.

Yeah And I remember coming straight out of college right And working at a large

company and like you could do things but they made sure like things were

locked down Like you could not bring down the company no matter how much you

tried right And they just And you had to build up the trust and the capabilities

to be able to understand the systems before they would grant you access to

other things right And we definitely never had access to production right Or

backups because that was things that were not needed as

being a developer in a company

Yeah.

So before we get to, what you should be doing, let's talk about

what you shouldn't be doing, right?

oh, let's tell the end of the story,

right?

So the... Because it's also just as crazy, because they contact... So their c-

they're hosting, their cloud provider was Railway, a company that actually I didn't

even hear of until this story, right?

So they were using some sort of, I believe it's a PaaS,

platform as a service, provider.

And they, they contacted support.

They

said, "Hey, we did this thing. we did this very bad thing, and we deleted

the ba- the production database, the production volume, and on there was our

production database, and on there was also our backups. can you help us?" And s- a

significant amount of time transpired.

As I recall, it was an entire weekend.

Is that, does

that sound about right

Yeah.

And then the CEO or the founder, Jett Crane, posted, I believe, on X.

and, and what happened is the CEO of Railway happened to see that post and

said, we do have some, DR backups."

The thing that I say, you cannot trust that this... Because

this also, we should also talk about that.

You c- I- thank goodness for these people in that company.

I don't wish ill in- on anyone.

I don't, is that... Sounds like when there's a story like this, I love this

story because it is a horrible story, but at least it does have a happy ending.

So many times when you look at that, that, the cloud disaster series that we

did back, last year, where we had, I think it was over a dozen stories

that did not have happy endings.

at least in this case, the cloud vendor came in and said, "We, we actually

have some backups of our environment. They are not designed for you, but

we're gonna make an exception, and we're gonna, basically allow you

to recover their da- the database."

is very nice thing to do They didn't have to do that and the fact that

they were able to at least help this customer back up I think goes a long way

Yeah, I think it do- it goes... thank goodness that CEO saw, they saw,

that message, right?

because, I remember when you and I worked at the same company,

and I was trying to make a point.

We were

a big Microsoft 365 customer, right?

I don't know, 500 something employees, something like

that.

And so we're giving them a lot of money every month, and I remember

talking to them about the fact that Exchange, their version of Exchange,

has, delayed replicated copies, right?

This is something that my nemesis out there seems to think

you can count on for backups.

And I said, "Let's just say something horrible happened.

You have these delayed replicated copies for DR. We're a big customer.

Is there any way we can get a access to those?" And they just

said, "No." That was just... That,

that was to a paying, very large customer.

So at least in this case, the... and I'm not gonna say that the

cloud vendor did the right thing.

I... They did do the right thing, but it's not their responsibility.

Your data is your responsibility.

You need to make sure that you have a plan, and you have a plan

that doesn't involve the vendor in question, in my opinion, right?

what do we th- what do you hear me say a lot about, let's

say for example, Salesforce.

Salesforce now offers backup integrated in its product What do

I think about products like that?

Don't trust it Always go with a third-party vendor or do it yourself

because you don't know what they're gonna change Are they gonna guarantee

where the data's gonna be stored like the Microsoft example right All of these

things because they only care about their own data and making sure they're checking

the boxes But if something happens maybe you get your data back maybe not

Yeah, I think about, Rackspace,

right?

Because when the feces hits the rotar- rotary oscillator, w- that's

not the time you wanna find out that your stupid vendor was also stupid

w- with their backup design, right?

or even worse than Rackspace was OVH

OVH, yeah.

Tell... Why don't you tell the OVH

Yeah OVH is a cloud provider in Europe where they build these data centers in

containers and they had a fire It took out

'Cause that's a

like shipping Oh sorry like actual physical shipping

containers that go on boats

those type of containers And so they had a fire in one of their data centers It

took out some of their shipping containers Customers had paid for backup but it looks

like their version of backup was We are going to take your data and back it up to

a server sitting on the rack in the same container just like down a couple racks

'Cause they said, y- it, it's literally said in the description, "Your backups

will be physically separate from your production." And that technically is true.

It was physically separate.

It was right over there instead of over here.

But that fire was so intense

that it was, Yeah it

was un- unbattlable.

yeah, exactly.

it

actually spread to the, to a neighbor.

And it, and the way they do it with these container, it's like containers

stacked on top of containers.

It's a

very... I don't get it.

Apparently, that vendor is known for being very budget-conscious, right?

not just that they're... not, and I don't mean that from a, a, like

I don't mean to be derogatory.

I'm just saying they're known for being a cheap vendor,

meaning it's cheap to use them,

and some people said, this is what happens when you use the cheapest vendor." but

it's just, I just, overall, I think that your backups should be under your

control in some way other than that.

I don't mean it has to be physically in your hands, I'm just saying

that it needs to be in a place other than... it's the 3-2-1 rule,

Okay can I challenge you on that or ask you a question So if I start to look at

like AWS as an example They offer so many services like S3 and RDS for databases and

other things like that Sometimes there's no mechanism to actually back that data

up outside of using their native services

so two, two things.

One is I would disagree.

In most cases, there is a way to get it, but you do need to use

a third-party service, right?

There are services that will work with AWS and similar vendors to get

a deduplicated copy of that stored in, in a safe place, number one.

Number two, at a minimum, if you can't, if you can't have it outside of the vendor,

make sure it's outside of the zone, right?

Make sure it's outside of the region, that you're... Because if

you just by default... And account.

Is that what you were

Yes I was gonna say account yeah

yeah.

Yeah.

You separate it as much as you can.

A different account, a different region, a different zone,

different availability zone.

Because, because, right?

Because

of everything that we just said, right?

I still think it's best, and I know of a handful of vendors that do that,

where they will take the, basically the end result of the cloud backups,

and then they will back that up,

right?

and, it's doable.

And it... And believe it or not, it can be done in a way that actually

saves you money, because one of the problems with AWS-style backups

with all the snapshots is they can get very big and very expensive,

and you can't store a long, history, right?

And as a

result, yeah, the other guys, if they can deduplicate it and

store it, maybe they can save you,

actually save you money.

And then also from a backup perspective You don't necessarily want to back up an

EBS volume You probably want the files within the EBS volume so you can do

restores and other things like that so

It depends.

You may want to back up the EBS

volume, right?

if you're talking about being able to do a recovery of the host, of the

VM, you might wanna do a, you'd do a

snapshot of that, right?

but anyway, yeah.

so the point is in general, backups, number one.

Number one, don't ever violate rule, never store backups inside production,

inside the thing you're backing up.

that's like me going... how dumb is this?

I can go to my... I can get a volume manager, like a third-party volume

manager, and I can tweak the volume on my Mac, and then I can make a separate

slice and make that a separate volume, and then I can pull up Time Machine and tell

Time Machine to back up to that volume.

How dumb is that?

Right?

So there are use cases where you would wanna do that for like quick

restores but I agree with you Based on the three two Yes Based on the three

two one rule three two one one zero one zero one one zero one something

Yeah.

Yeah.

you cannot store your only... I will

make, I will clarify.

Your only copy of backup, you cannot store it in the place

where you're backing up, right?

There's nothing wrong with having a copy,

a convenience copy, a cached copy, a local copy.

None of those is wrong, but if it's the only backup that you have, and

that's what happened in this case with Pocket OS, unfortunately, their

backups for their, production volume were in the production volume.

bad design.

I've been very talky this episode.

I think this episode has got me all worked up.

I haven't allowed

I I think I don't know I think it's also like it's been a while since

we've seen this sort of I don't

wanna say I want to be nice I don't wanna say incompetence but like misconfiguration

like gross misconfiguration and design

And I think you're also Yeah And I think it's also cause you're pissed

cause they went after your French friend

Yeah, they tried to blame it on my French friend.

It's the... By the way, I was talking with, I was talking with Claude about

this episode, and the first thing Claude said to themselves, "This is not an AI

problem, this is a credential management problem." I'm like, "Yeah, no kidding."

I was like, I, you seem a little defensive there, Claude." but, yeah.

So you wanna talk about the credential management

by the way, yeah, I do, but I just wanna have a funny though.

I also

realize this is the first episode we've had in a while

where I'm the primary talker.

Yep You're like

we've been doing a lot of episodes with Mike, and I love the episodes

with Mike, but that boy talks as much, if not more, than I do.

But

that's why we have him,

right?

that's why I'm talking so much this episode, because I can.

All right.

so it had a happy ending, but let's talk about... the easy ones are

please don't store your production data and your development data

in the same environment, in the

same volume, in the same accounts.

It, s- any of that, right?

Don't store your da- backups in the production environment, right?

here's a question

ones.

Yeah.

You know what I bet you they never ever did a DR test or a backup

recovery test because if they did

back because I'm right.

Or you're right.

because if they did they would've been like Wait my production and my

backups are all on the same volume That's probably not a good design

Yeah.

What I'm wondering, because we're gonna talk about this in a minute regarding

the credentials, what I'm wondering is if there's a programmatic way to

take, an export of your entire, cloud environment that would show, what's on

what, and then programmatically say, "Hey, did y- you've got a development

and a production database on the same environment." Just wondering,

There probably are tools or another way you can think about it is as we're

gonna talk about credential management is from like an identity perspective

Yeah.

Is who has access to what resources Are they common access Are they not Those sort

of things might also surface some of those

the bigger the environment, the more difficult that this would be

in terms of, doing an audit, right?

'cause I know you've worked with Amazon's Well-Architected Review.

Why don't you talk about that a little bit?

Yeah and so this is a process where if you're building on top of AWS and you're a

certain size and you get all the resources things like that they have In the past

what they used to do is you'd get assigned a person they'd walk you through from

AWS looking at your architecture make sure you're doing the right thing from

a security access control resiliency perspective And what AWS then did is

they took all of that and they automated it So they're now able to look at your

environment and tell you Hey yes you have backups good You're not using the same

AZs for everything that's good right Here are some of our best practices that we

can now run through a checklist and make sure your application is meeting those

interesting

if it includes

it becomes a little harder when you have applications writing in because AWS as

the storage infrastructure level it may not have visibility into that unless

you're tagging things the right way

Yeah, but, the- I'm just thinking, one of the ways you could do this is to just

make sure that every application, every host, every VM, every whatever it is on

its own volume, if we're talking about

volumes,

right?

if you could apply that design principle, then you would never

have this scenario where you have a development system on the same volume as a

production system, right?

I'm always thinking about how could, how can we do this from

a, from a large standpoint?

yeah.

All right.

speaking of which, the, the, the crucial thing here, though, I think, besides the

fact that the whole thing was designed wrong from the beginning, the crucial

problem here that I wanna just make sure we cover in this episode is that Claude

had the credentials to do what it did.

let's talk about what the proper thing would be to do, and that is the

concept of secrets management tools.

Do you wanna talk about that?

so as you can imagine in all of these environments you're probably going to

need access to different things and if you hard-code it into files if you store

it in code that just makes it vulnerable to either tools like Claude being able to

access it or as we've talked about with Mike from a cyber recovery perspective

a bad actor gets in they can now pull the credentials and start using them for

other nefarious purposes And so what you

with, with LastPass,

right?

They found, I think it was a YAML file that had the i- the credentials

to the backup environment, and

they used those credentials to then hack the... they used it to

steal, they used it to steal the backups.

They stole the

backups of the vault.

They recovered the vault, and then they hacked the vault.

Yeah.

So yeah, this is just a very bad way.

Right and so you want instead of storing it in plain text what you want is a

proper system we've talked about password managers before very similar secrets

managers where you're able to centrally store secrets in a secure way to make sure

authorized access is allowed and services that need access to those get access

to the keys that they need Least Yeah

Yeah.

Before we talk about secrets management, I just wanna talk about, again, the

concept, just a couple of the concepts.

There is this concept of least privilege, right?

So not only did Claude have this, permission that it shouldn't have, I

think... I don't think anyone knowingly gave this permission to Claude.

I think it just, it just happened to get access to it.

I think maybe they did it, one time to do, fix one thing and then forgot to

take it away or something like that.

But the, this... But the fact that it had superuser violates the concept of what?

Of least privilege You should only give access to what you need access to not to

be able to go destroy vol Like if Claude should never have been able to destroy

a volume you should never have given Claude that permission to start with

Yeah, exactly.

the... And whatever... so whatever... I think in this case, not only did they

give... They violated multiple things.

They get, they violated least privilege.

They violated... They should also... If you do give a big privilege,

it should have an expiration,

right?

That privilege should expire, and it sounded like th- this was a privilege

that was granted to it a long time ago.

Everybody forgot that it had it.

It didn't expire.

and what were you gonna say?

do you think though that they gave access to Claude explicitly or do you

think the developer whoever was using it was like Hey I have this issue

that I need to solve Let me get Like I wonder if it was tied more to the

user's level of credential rather than Claude its own separate We don't know

but just given my history with IT, this is typically... This is the equivalent

of, chmod 777, just, just get, just do read, write, execute everywhere.

That'll solve it.

And you're like, "Oh, yeah, there you go.

That solved it.

Never mind the fact that we just opened up the world to

the world," right?

the reason I bring that up though right is if their users of this company aren't

following the exact same least privilege access and if Claude was using OAuth

to impersonate that user and whatever privileges they had right then they

would've been able to be like Hey I have access to everything Right So I don't

know if at a company level if it's like Claude wasn't limited access or maybe

all of their users are not limited

what I read was that there was a file.

There was, like, a YAML-type file that had the credentials in it that it needed.

It saw the file.

It took the credentials.

It did what it needed to do, right?

So I think that somebody at some point had a problem that they

solved Probably temporarily by, just, again, to go back to back in the

day, I remember a time when I talked about that environment where the, e-

everything was fired off, firewalled off

between everything.

And I remember saying, "Listen, right now, just turn that feature off. I can't

accomplish the thing that you asked me to accomplish in the timeframe that

you asked me to accomplish it if you keep trying to firewall off the backup

system from the rest of the world. I need to be able to do this." I was

violating a principle of their design

because they had firewalled off internally different parts of the environment from

different parts of the environment.

And I remember saying, "Hey, right now, to finish..." With, there's

this thing called Y2K coming, right?

I assume... I left, by the way, I finished.

I assume that those super hardcore cybersecurity guys

went back in after I was

done and say, "Let's figure this out," right?

But, but I remember saying at that time, "Listen, there's no way I can

finish in time." So I think somebody possibly had that, "For right

now, to fix this problem, we're gonna give, we're gonna give Claude super

user access," and then they forgot to take it away for, they forgot to fix it,

and then Claude said, "Oh, look at this. Look at what I got," right?

And then it used it to solve the problem, right?

so it violated the concept of, that initial thing violated the

concept of re- least privilege.

It violated the concept of expiration.

and, and, and also just th- they weren't rotating things.

they weren't auditing things.

So let's go back to secrets management.

So you talked about secrets management.

Do you have an example of some sec- secret managers?

Yeah So AWS we talked about previously right AWS has Secrets Manager AWS Secrets

Manager if you want something more on premises right there is HashiCorp

which I think also has their Vault mechanism in order to be able to store

secrets right So it all depends I think and there's probably a dozen other

companies out there who do similar things

those are two very popular ones.

One, AWS, and the AWS Secrets Manager can handle on-premises environments.

It's just obviously If you're gonna authenticate something on-prem, it needs

to be able to access AWS, duh, right?

the HashiCorp Vault is the on-prem version.

There is an open source version.

There is a, there is a commercial version.

same thing for, CyberArk, right?

there's one called Sealed Secrets that's specifically for Kubernetes environments.

I personally, unless I was, like, pure Kubernetes, I, you know me, I, I like

solutions that work for everything.

I don't wanna

have, 10 different solutions.

So I like the idea of the AWS Secrets Manager, I like

the idea of HashiCorp Vault.

By the way, I said HashiCorp.

You said HashiCorp.

I don't know which one it is, but yeah.

Anyway.

I... And so the question is, which is, which is the better?

The, it, it's about you... how do you make, how...

W- why don't you talk a little bit about how does one choose between an open source

system that you manage on-prem or even in the cloud, or, a service like AWS, Secrets

would say the biggest thing is do you wanna deal with it Because that's honestly

if you're picking up open source there is a certain amount of maintenance

configuration ongoing patching right All this other stuff finding a place to

deploy all this other stuff that you have to do but it gives you the flexibility

right So you And of course the cost

it's in your grubby

feely

right?

yes And also cost

what's one other thing you need to do if you're using something like HashiCorp?

What podcast are we recording for?

Oh pack it up I said manage it and all the rest but yes

Oh, you threw backup and all... You can't yada, yada backup.

that

Don't forget backup and DR and everything else

yeah, all that stuff, right?

So do you wanna do all of that?

It's probably cheaper, assuming you have the people, assuming you have the

right risk tolerance for that, right?

It's probably

cheaper to do it yourself if, and the infrastructure

right?

but if you're smaller a- and you don't want to, or even if you're bigger, but you

don't, maybe you don't feel you have the expertise to do all of that correctly, and

this is a really important thing, right?

Secrets management,

right?

So

you're

gonna pay per secret.

You might even pay per use of

each secret.

I don't know.

Or if you are a customer who or a company that's all in on AWS or Azure

pick your favorite cloud right Maybe you're like Hey I don't wanna roll my

own Let me just use what's available

Yeah, if you're... Yeah, if you're heavily... Yeah,

'cause, the, you're right.

There, there's like the Azure Key Vault, the Google Cloud Secret

Manager, they all have their own thing.

So if you're all into GCP, you would look at that.

If you're all into Azure, you would look at that, right?

but if you're rolling your own internally, if you're one of these that have, go

back to g- to Claude Code, if you're one of these that are really... A lot

of people are getting into code, right?

That are, they're coding in things that they know, that they've never trained at.

just,

it's a tiny little example, but right now I'm working on doing all this stuff that

I'm doing for this project that you know.

You are

in the, you are in the NDA little group, right?

And it involves a lot of web crawling, right?

And I have written, as I make quotes in the air, I have written six

Python scripts, very complicated Python scripts in the last, 24 hours.

I don't know how to code in Python.

Claude did it all for me,

right?

There are a lot of people that are using Claude Code to build giant applications

for their companies, and they don't know anything about that infrastructure, right?

So if you're doing that, just make sure that you're, that you also have

some sort of secrets manager, that you're doing the right thing, right?

Or just don't give access to everything

no, not or.

It's not a or.

even if you're following proper design protocol, you still should

not be storing, secrets in YAML files or any other type of files, right?

when I was talking to, my French friend about this, and I was like... Because

this isn't really my thing, right?

like I was like, is there one file type?

Is there one, is there a way to go find all these files?" And the short

a- the short answer was, there is definitely not one file type, right?

It talked

about ENV files, YAML files, JSON files, SS- SSH files, just old scripts, right?

Peop- people can hard code the passwords right in the script.

The one thing that it, they did have all in common, though,

is that they were, all plain

Text files, okay.

All right.

a and I think that's the key is like with all of these text files like And if you

have so many developers it's like how do you know who wrote what and did they

clean up after themselves Because maybe a project existed and then got shelled

but none of the data ever got cleaned up And so how do you actually figure

that out Like I totally get like it just seems impossible right with the sprawl

it's definitely... So you, first we're gonna, do no future harm,

like a modified version of the

Hippocratic Oath, right?

Do no future harm.

Moving forward, we're going to do better, right?

Th- this is, the, my, my favorite Maya Angelou quote, right?

"We did what we did when we knew what we knew, but now we

know better, so we'll do better." So you say, "Moving forward, we're gonna use..."

Pick your favorite secrets manager.

Do an on-prem one, use AWS Secrets Manager, use your Azure, whatever.

you're gonna... You're never again going to store credentials

of any kind in a plain text file.

shalt not store credentials in plain text file or be smitten down

Nothing good comes of that, right?

how do we then go and audit?

And by the way, this isn't a one-time thing.

You should just, you should be regularly doing this to make sure that no one has

violated your new principle of design.

And the answer to that, there's a couple of different tools, and

the one that seems to come up more often than not is called TruffleHog.

Do you know why it

might be called that?

So I know about truffles and I know that they have specially trained pigs

to hunt down truffles because they are very expensive and found in forests

Yeah.

I think that's a perfect name for this tool, given that these are really

valuable things that we wanna find, and then we wanna pull 'em out, and

then, that's where the analogy stops, 'cause we're not gonna eat them, right?

we're gonna delete them, right?

and so i- it's a, it's an open source tool.

There is a free, a feature, a f- free version that has features,

and then there is a commercial version that has more features.

there is a similar tool, from a company called, GitGuardian,

and It also has a free

tier.

The... But TruffleHog seems to be the one that's popping up a lot, and that

it's an open source tool, and you can run it across, and what it's gonna

do is it's going to use, a variety of techniques, look at using pattern

matching and also entropy detection.

basically, it's gonna look at your, your Git repository.

it's gonna look at all all sorts of stuff, looking for, hidden, stored,

credentials that could be, that could ultimately be your undoing, your

version of what happened here, right?

Because that's the problem, is most people find out about these

credentials when they've done them harm, and we wanna make sure that

we're doing that on a regular basis.

So what's our summary, Prasanna?

So I would say th so the summary I would say is don't believe the news

hype This was not really an AI issue.

make sure that you do not store your backups and your production together

you're testing your backups Follow the 3-2-1 rule

Yeah.

three, three two one one zero.

Okay 1-1-0 And then

make sure you are using a secrets manager and not storing credentials

in plain text files and granting access to only things that are needed

And if things beyond that are needed permissions beyond that are needed

it should have a very short timeframe

And you should run tools whatever it is in order to be able to check your environment

on an ongoing basis to make sure that people are following the processes and

using secrets managers and not storing these things outside of that system

Yeah, absolutely.

I think the only thing you missed at the very

beginning was not storing your production and development

at the same, on the same thing.

You had... you got

almost everything.

You said produc- you said backups on production.

You said ba- don't store them in the same place.

You didn't say don't put production and development on the same

environment.

Yeah,

course you had

to find some fault

Of course, it is my job.

I live,

Fine go talk to your French friend I don't wanna talk to you

to your French friend

By the way, you can see this sign right here.

I am very silently correcting your grammar.

that, this was a great story.

It's a great story, I love that it's a very sad story that had a good,

happy ending that we can learn a lot of... So no one was harmed in

the making of this film, but, but we can learn some valuable lessons from it.

and with that, go and do no harm.

All right, that is a wrap