The REDCap Attack that Phishing-Resistant MFA Could Have Stopped

Can you imagine finding out that a Chinese state-sponsored hacker group

spent over a year inside your network and your MFA didn't do a single thing?

That's exactly what happened to some academic and medical research

institutions, and why phishing-resistant MFA isn't optional anymore.

Today, Prasanna and I break down exactly how attackers harvested credentials,

hijacked REDCap, uh, which is a thing for research and education, Google

Workspace with fake compliant rules, and exfiltrated sensitive research data

for months without anybody noticing.

We'll also talk about what you need to do right now so that

this doesn't happen to you.

Here we turn admins into cyber recovery heroes.

This is the Backup Wrap Up.

Welcome to the Backup Wrap Up.

I'm your host W. Curtis Preston.

I have with me a guy who is super jealous of the fact that in the

last couple of weeks I have used, I think, every tool that I own.

Isn't that true, Prasanna?

the one that I had you buy

It, that is true.

a significant amount of use, of the one that you basically really talked me

into it, and I was like, "What do I need one of those for?" And now I have it.

We're… What?

I found you the deal

You did find me a deal.

of course that, that, the tool we're talking about at

this moment is the planer.

and if you've never had a planer, it's a, it's an amazing device, but it's

if you've never used one before, you don't know why you would want one.

And then once

use

get one,

"Oh my gosh."

you're like, "How did I ever live my life without this?" like the air fryer.

It's like the air fryer of tools.

I don't know about an air fryer

meaning you don't know if an air fryer is awesome?

Or if I would justify having an air fryer

that's because you've never had an air fryer.

See?

This is what I'm talking about.

That and you know what else?

The West Wing.

if you would just watch The West Wing.

so wait, so here's another interesting thing about Prasanna.

Prasanna owns, because you got a deal on Apple, like Apple, what is it?

Apple Plus?

Yeah

TV, and you bought the entire series for some ridiculous price, right?

Like, how much was it?

It was like $30 or

something

yeah.

and that was, like, a year ago at least?

it might be going on a year and a half

Yeah, and you've yet to see a single episode because you know that the

moment you do, you're gonna get hooked and you're gonna, you're gonna

here's my problem though, Curtis, is this is not atypical for me where I

will buy something and it will sit

Yeah.

for

years

this is something you and I share

Yes.

I w- so recently I've gotten back into watching physical media, right?

Yeah.

and all the rest.

Yeah

I had some which were still sealed for the last, 15 years that had never been

Wow.

Yeah.

that was kinda like me in the wood shop with,

Yeah

some of the, my dust collection, and now that's my new obsession,

is my dust collection.

Yeah.

Anyway, people don't want… They won't, they don't wanna hear about this.

They wanna hear about, Google.

Are you sure?

Are you sure?

yeah.

so this is a, this was, this is another one of those stories where you're like,

"Man," like 'cause it, it's another, living off the land attack where

Wh- what's living off land for our

Yeah.

So basically, living off the land is leveraging, i- is an attack that leverages

y- the tools that you have in your arsenal, against you basically, right?

and, the, what happened here and, so first off, who are we, talking about?

Google's, their threat intelligence group, they have published a really

big report which we'll link, in the show notes, that really detailed, I

have to give them props, that they really detailed what happened here.

It's, it's actually a Chinese, PRC-sponsored attacker,

threat actor called UNC6508.

That is

Is

worst name ever

sometimes they have these Titles, nerd titles, I don't know, labels

yeah.

of these attacks.

But then the same organization, like what we might know externally

as LockBit or we'll take whatever

Yeah.

what's the name of…

ones

Yeah, what's the name of the one, the one that did the last attack that we covered?

Shiny hunters

It was the Canvas.

Go

the, so we didn't cover it, I'm sorry.

It was the last attack you and I talked about.

Yes

and it was the attack of Canvas, which is a learning platform, and, there,

because I saw an article that said it was the same, it was the same group.

I didn't check that, but, Google is calling them UNC6508.

and they targeted mainly, North American institutions, academic,

medical, things like that.

And via this tool, called REDCap, which is a Research Electronic Data Capture,

it's a web-based platform, and, they, th- which has a unique attribute that

allows you to run multiple versions of the software simultaneously.

And so even though you may be upgrading and patching more recent versions

of the software, if there's a, an exploit against an older version of the

software, they, they would be able to do it, and that's what happened here.

And once again, it started with harvested credentials.

we don't know where…

do they get?

they were, it was an admin.

it was an admin's credentials, but the, but again, this is they broke

some of the rules, and we're gonna talk, the meaning the victims.

they, broke some of the rules that, that we talk about, and we're gonna cover

them in the, that, that had they not done that, it would've been a different,

the attack would've been different.

but the scary thing about this is that they put in this, this malware

that's called Infinite, Infinite.red.

it sounds to me like that sounds like it's a s- a specific malware that

they wrote just for REDCap, and it sat there for how long, Prasanna?

It looks like from what I could tell this happened like November 2023

Good.

Yeah, it looks like literally they got in, and then they…

this is we talk about dwell time.

We talk about, just to go back to backups here for a minute, one of the real

problems with these types of attacks is that sometimes having a really good backup

isn't going to be enough because, what…

If they've been in there for over a year, like a year and a half, maybe even more,

probably isn't that far back.

no, right?

You're not gonna, you're not gonna have backups that don't have this.

And they… And what they did was they leveraged these, what do you call it?

v- vulnerable versions of RedCap to harvest their credentials, and

then, and then they somehow…

But wait, go ahead.

Go ahead

But even when they harvest the cred- credentials, it's not

like they wrote it to a file.

They wrote it back to a red capped database,

Yeah.

Yeah.

end product database.

Yeah,

they wrote it to

talk about living off the land, right?

and then they intercepted RedCap's upgrade process, so it then injected this

malware into every new version, right?

even if you were trying to patch things, you would get the new version, right?

It's an interesting question.

I know we talk about patching and, making sure everything's up

to date and having central IT.

Do you think a lot of these issues stem to the, from the fact that these were,

like, IT or software packages targeted at academia and researchers who sometimes

pull things together on their own?

It reminds me of, the shadow IT phenomenon, right?

n-

they're

no, actually, yeah, I don't think so.

From what I'm seeing i- is that, again, we're g- and we're gonna get

to the what could they have done better and what you can do better.

I think just, i- it's not the upgrade process.

it goes back to the harvested credentials.

It goes back to limiting the ability for products to do what they did,

and then we're gonna get to the next thing, which is once they had…

They had been harvesting credentials for a year, right?

for over a year, right?

And then they took some of those credentials, and they tried to log

into other systems, and this is where Google Workspace comes in.

And so what they had was, once they got… They were able to take those

harvested creden- they started with stolen credentials, then they used those

credentials to harvest other credentials.

Then they took those credentials, and they logged into Google Workspace,

Yep.

into, again, valid admin, as a domain admin, and they created

something called compliance rules.

You wanna talk about what those are?

Yeah.

So in an organization, you might have certain compliance requirements which

are, hey, if an email is from this person or to this person, or if it contains

these types of words, then forward it off somewhere else so then it can be secured

and can't be deleted and is protected for compliance auditing other purposes.

And so they created a compliance rule that said if it matched certain keywords

to send it out to a certain email address.

And this was ingenious because it just looked like normal email traffic.

So even if you were trying to detect something, it was like, oh yeah,

this is just part of your normal systems operations and all the rest of

Yeah

address was controlled by the attackers

And they would, and then it would BCC forward the, to a Gmail

address that they owned, right?

The keywords, very interesting.

The keywords, they're talking about geostrategic policy, military strategy,

advanced tech, and specific pathogens.

And by the way, that one was interesting because one pathogen that

they had on the list was chikungunya.

I don't know how to … Chika- You can say that?

Okay.

checking

I've never heard of that before.

So they're saying it's a mosquito-borne viral disease that's

responsible for an outbreak in China.

Yeah.

so interesting, right?

Oh, okay.

Gotcha.

hence your ability to pronounce that word.

I've never even heard of that word before.

anyway.

so they were… So on one hand, you're saying it just looks like normal

email traffic, but, I do think that perhaps somebody could have noticed.

By the way, the rule that they, the compliance rule they created was called

Patriot, which is interesting, right?

Didn't they misspell it?

You're right.

So they did sp- so it's spelled Patriot.

which again, might have been a, might have been a red flag, right?

but, and so the, basically anything that, that met these, the

filters that they were looking for gets forwarded to the bad guys.

And, d- Google, the Google Threat Int- Threat Intelligence

Group, is that what it's called?

Yeah, GTIG

Yeah, GTIG.

they, it says they disrupted the infrastructure, they disabled the

Gmail account, that was being used for exfiltration, and they notified everybody

and published, rules on how to stop this.

But, I always go back to, what could they have done differently, right?

and I don't wanna focus on just this particular account, but I don't think

Red Cap is alone in this idea of having legacy versions being, run side by side.

Can you think of any other products that work like that?

VMware

Oh, yeah.

Yeah.

Right?

'Cause when you were describing RedCap,

Yeah

I was like, "That sounds exactly like a hypervisor."

yeah.

you're right.

Is

and,

thinking or were you gonna give something

else?

no, I wasn't thinking.

My brain was blank.

but yeah, no, that's… I think that's a per- that's a perfect example, right?

where you leave, you leave many versions running.

and so really what they could have done in this case is, again, obviously

general password… we can start with general password, hygiene, right?

The, that if they had been doing normal password hygiene, the

passwords wouldn't have been harvested out there somewhere in the wild.

And then because the initial attack started with, passwords that have

been harvested out somewhere.

they don't know where that happened.

and if that means that they were… somebody was using a username and

password externally, potentially that, that then was used internally.

so if they had good password hygiene, that wouldn't have happened.

The next thing is to not…

Go ahead

I'm surprised you didn't start with the number zero.

What's the first thing that you do before password?

list of three things, if you did these three things,

Oh, patching?

You talking about

And so for patching, I was gonna say inventory management,

Yeah, absolutely.

and you knew what was out there and then you were able to patch

it, you probably may, or you may have avoided some of these issues

Yeah.

and then the big thing with the legacy version, support is to not do that, right?

you don't run things that you don't, at a minimum you disable them.

the best practice is actually to remove the old version completely.

don't leave it sitting around.

so another thing is, to not trust application-level

authentication by itself.

to use something like SSO, this is an important enough thing that you don't

trust the IAM system of just any old, piece of software that you use SSO to

log into important things like REDCap.

and had they done that… A- and by the way, this had, when we go back to

the, we talked about the Canvas hack, same thing there that, I'm currently

talking with a client that they were hit by the Canvas hack, but it was very

minor because they use SSO for the vast majority of accounts when logging into it

One other thing, and I don't know if we've necessarily touched on

it in past episodes, using SSO or a central identity provider,

you can enforce requirements.

So if a password needs to be rotated, do you use MFA, minimum number

or, minimum number of characters?

All the rest of these things you can enforce versus if Redcap had their own

password system and, say, didn't have this functionality, kinda limited.

Maybe someone hasn't changed their password in seven years.

Who

Yeah, or even if it had that functionality, that means you're

managing that functionality in every, SaaS app that you use, right?

Yep.

yeah.

Good point.

you're an organization, you really should have a central password system,

Yes.

system

There's an, entire, there's an entire industry built around

that and, thumbs up to that.

the next one… Go ahead.

What?

I touched on something which I think might be the next one

Okay.

Touch away.

one MFA?

Is it MFA?

It's always

the next…

like it's always DNS

specifically phishing-resistant MFA.

yes.

You wanna talk about what that is?

Yeah.

So like we've talked about, someone can guess your password or steal your

password, like what happened here.

But with MFA or multi-factor authentication, just having

the password isn't enough.

It should also send you a way to authenticate in addition to just knowing

the password, the username and password.

Typically, you see this as, a code that shows up on your phone.

email is not great, SMS is not great, but it's better than nothing.

but

you really

should be using like a one-time password

Yes

there.

Many versions are out there.

So that helps.

And then the other thing that you mentioned, Curtis, is phishing-resistant,

So you wanna make sure that someone doesn't go and steal your, MFA

token and then start to use it, that you do have … I know we've

talked about this in past episodes.

There's this notion of MFA fatigue, right?

Where people constantly keep pinging you, being like, "Is this you? Is

this you?" And then you click yes because you're tired of saying

no and responding all the time.

So you want

something that

is more resilient to those.

I think the big thing with truly phishing-resistant MFA is that MFA

that is tied to something, right?

The ph- it's tied to a security key, it's tied to a particular location,

it's tied to a particular device, so that even if, a- again, just

the… It's MFA for the MFA, right?

That, that, that if they steal a session cookie, which is what was happening here,

they were stealing session cookies, then they wouldn't be able to just replay

that because the system would notice that it was coming from other place.

Yeah.

Yeah.

Yeah.

and then of course you can and should investigate, passkeys, right?

and again, the beautiful thing about passkeys is that it solves all of this and

it's, they're tied to a location, right?

What's a passkey?

thank you.

So a passkey is basically a complete replacement for passwords

and, MFA, and it's basically a…

Think of it as a locally stored, I was gonna say password, but it is a key that

is stored locally with the device that you have, that is tied to that device and

tied to that account, and it's basically played on your behalf, whenever you

need to log into that, that account.

So passkeys fall under what's called FIDO, which is Fast Identity Online,

and the common thing that is stated…

There are multiple, there are many known attacks for passwords in MFA,

and you taught, you touched on one of them, which is the concept of,

becoming, an MFA fatigue attack, where they send you so many things

that you accept one of them, right?

And then they take the accepted one, and then they go do bad things, right?

there are known, there are no known attacks against FIDO-compliant passkeys.

So if you're not doing them everywhere you believe you can be doing them, you

should be investigating that right now.

if y- if you're familiar with that and you're working your way, I would just

say prioritize that wherever you can.

if you don't know anything about it, then, you should be looking into passkeys,

One of the things that GTIG pushed, in terms of what could have helped here

is the concept of device-bound session credentials and context-aware access.

And w- we talked about this a little bit.

Context-aware access, if you enable that on an account, it's going to do things

like, why is this coming from a different place than it was 30 seconds ago, right?

and why is it coming from this IP address?

Why is it coming from this country?

I thought this person was in California.

Whatever.

It's a context, right?

A con- and every time you go to access, it's gonna check that context.

And then you have the device-bound session credentials, or DBSC, and

what they do is, they're tying each…

'Cause when you authenticate with a, a browser, that creates a session,

and if you're using DBSC, those credentials only work in that session

and they tie it to that device.

It's a little bit like passkeys in that regard, and Google is just

talking about these especially for highly sensitive accounts.

If you turn on these two features, it would tie, once you authenticate,

it would tie that authentication to that session, and then context-aware

access would check that, so the two work together hand-in-hand.

And what that meant was if and when someone, like this malware, steals

your session credentials, they wouldn't be able to use those anywhere else

Yeah.

And specifically, this is the case where they're not just stealing

your password, but they're actually

stealing, say, your browser cookie

your cookie and trying to replay and reuse your cookie on a different device.

Right

By using device-bound session credentials, you're guaranteed that

even if they stole that session cookie, it's not gonna be usable anywhere else

again, it's specifically something that, that they, talked about,

or that Google talked about.

so the next is this idea of compliance rules, and they're not… Th- this is,

that's what Google Workspace calls it.

Every system has something like this, and the idea is look at any

standing rules that are there.

In a large company, you may have tons of them.

This is where I think AI can be helpful here.

Look for things that are created that are forwarding email out to

especially external accounts, right?

Didn't you

that's what's-

anytime something gets touched in these compliance rules?

yeah.

A- agreed, right?

Ha- have a compliance rule for the compliance rule.

that anything that's a new compliance rule, th- it should trigger some sort of

event so that you can then go check that.

That's a great, that's a great point.

So then the next one they recommend is something you might think

is obvious, but I guess a lot of people get this incorrect.

It's separating your credentials across security domains.

So as we looked at this case, RedCap, one security domain, Google Workspace, another

security domain, the two streams shall never cross, and yet someone used the same

credentials across the two, and that's what sort of allowed a breach of RedCap to

now impact their Google Workspace account.

Right

really should be keeping credentials separate if they need, if they don't need

to be common, or using SSO or an identity provider which deals with all of this for

you so you don't have to worry about it.

Yeah, or also PAM, so privileged access management, right?

and go ahead

Would… Here's a question for you, Curtis.

Yeah

this comes up a lot of times when we talked about backup systems, how keep

backup systems separate than your production systems, from a credential

management, active directory perspective.

When a company is using SSO, say O- Okta or someone else like that,

do you think it is important to still think about these security domains and

so have a different SSO user for Red Cap versus their Google Workspace account,

or is that overkill since you have Okta providing some of that control?

W- yeah, when it comes to admin accounts, I don't think there is

such a thing as overkill, right?

I think you, you need to treat admin accounts differently.

perhaps you do one thing for, regular users.

And again, going back to the client that I was talking about, that they,

the reason why they were only, they used SSO for all the normal users,

but admin accounts were separate.

so I think that is, that is important.

You can use SSO to layer on top of that, right?

So that they work together.

But I think, I still think you need to, keep that separate for the, for all

the reasons that we just talked about.

and then the next thing, we talk about here is get some logging, man.

Some sort of XDR, some sort of SIEM, some sort of, some sort of monitoring going on,

looking for the kind of things that were happening here, because they had to be

sending a butt ton of sensitive e- email out to external accounts, for a really

long time, and, nobody seemed to catch it.

I don't, that, that's quite disconcerting

did they say when they actually breached Google Workspace?

'Cause I know

I, it was towards the end.

It was towards the end of the attack.

Yeah.

It was the last thing they did after they'd been harvesting

credentials for over a year

Yeah.

Yeah.

Crazy

Crazy.

So here's a question.

So we talked about this, right?

And things you could do to prevent attacks.

Given the name of the podcast, is there anything that you would think

Did we forget something?

from a data protection, data recovery perspective, from a

cyber recovery perspective they could have, should have done?

Or is this even if you had the most amazing

cyber recovery plan in place, you didn't do any of these basics, you're hosed

I don't think, this is one of the situations where I'm not sure how much

backups would have actually helped, right?

Because it, you do need, obviously, you, you're never gonna be saying

you don't need backups, right?

one of the things we talk a lot about is just please, for the love

of everything, please make sure that you have regular, automated backups

of everything that are on a sec- separate security domain, and also that

use truly immutable backups, right?

That i- you, 'cause the worst thing is when you hear something

like this, and then you see that the backups were also impacted.

The downside here is that backups quite possibly aren't going to

help you because d- two things.

One is, many people don't store their backups longer than the amount of time

that this attack took, number one.

Number two, what are you going to recover?

Are you going to literally, like you found out that they first compromised

your system a year and a half ago.

Are you going to restore your database to a year and a half ago?

You're not going to do that.

so the related topic here is, from a system standpoint, one of the

things that we came to in the book, which we haven't mentioned the book.

if you're not familiar with the book, that should be over my,

right shoulder in your view.

if you're watching this, it's my left shoulder, but it's

right in the video, right?

I think I'm pointing right.

Yeah, it's to my right in the video.

Anyway, the, is, Learning Ransomware Response and Recovery,

which, I wrote with, Dr.

Mike Saylor, who is a frequent podcast, guest.

that, Dang it, I lost my train of thought.

We were talking about the… Oh.

So you really have to approach backup a- as two things.

We really came to the r- reality that th- there, there were three ways to

restore after something like this.

One is to just restore back to a point in time before you, you know that you

were attacked, and that's like the last thing we want you to do, actually, i- in

terms of restoring the operating system and the applications, because it's really

hard to know when that point is, and it's really easy to reinfect your systems.

So that's one method.

The other is this idea of, restore and then go in and surgically

clean each system before you release it to the public, right?

And that is better than just knowingly restoring, without doing any cleaning.

But again, it still could be really difficult to find some

of this very pesky malware.

You talk about the malware here that could- kept reinfecting.

Sometimes the malware will be, in the firmware, and it will

reinfect you after a reboot.

So really, I think the only real method here isn't just, I guess it's

technically backups, but it's really an automated, like infrastructure as

code type environment, where what…

you know what your… when we talk about VMs and applications, you know what

version of the OS you're running, you know what version of the app you're running.

You should be able to push a button and boom, you have a new

VM of that from a trusted source

yeah

that is, that was created when you first made that system.

This isn't something you're backing up.

So what you're backing up is the application data, the database and,

and the documents and all of that stuff, and then you, when it's time

to, to rebuild, you rebuild the OS and the application, and then you restore

the actual data for that application.

And generally speaking, the data itself won't be infected.

Go ahead.

I have a question for you.

Yeah

Do you know, and maybe this is also a Dr. Mike question, any ransomware actors

who have attacked the Golden Images?

I am not aware of any, right?

I would I think I would argue… You mean the actual s- so

where the images are stored?

in the sense of if they infected the golden image, right?

The, like you said in your

Like a supply side type attack, basically.

Yeah

right?

Or it during the attack, yeah.

If they, if they infected those images as you're trying to recover

and rebuild, like you said, right?

Rebuild the OS and everything else, you're basically reinfecting yourself.

Now, hopefully

Yeah.

testing those as well before they

Yeah

them during rebuild, but maybe that's something

yeah, I, that's, I, it's a good risk to think about and I guess, in my mind,

any golden image is stored either on worm media or on, immutable… Like

worm media like a DVD-type situation or on truly immutable storage.

But because again, you gotta think of it like a backup.

you gotta make sure that there's no way they can attack it, right?

And

so this is something that you're creating right in the very beginning.

Yeah.

ahead

do wonder how many people actually think about that, 'cause

I don't think they do

until you were actually just talking through it and

Yeah.

Yeah, you have to think about it like a backup, and that backup has to be stored

in a way that no one can ever get to it.

And again, my standard there is if you can't delete it even

if you want to, then I'm happy.

If it's anything less than that, I'm not so happy.

and there are a bunch of ways to do that, right?

You c- there are truly immutable storage platforms that even you can't delete.

There's different modes.

sometimes you'll see compliance mode and retention mode.

you want the more restrictive of the two.

Again, the standard is i- if you can't delete it, even if you want to, then,

then that's what you want, right?

and, you could also do that with worm tape, and you could

do it with worm, optical media, one of which we'd covered here.

M-Disk, right?

the media that's meant to be around forever.

I wanna see hackers attack that.

good luck with that.

It's a, called a

blowtorch.

Which is what I was saying.

Unless, like the only way you can this media is, like you said, if you

have physical access, all bets are off

Yeah, you, yeah.

That, that's true with any element of cybersecurity, right?

so you, that's why physical access is, paramount.

and I go back to the best that I've ever seen there, where it

took multiple layers to, to get, to just to get in the building.

I remember

Yeah.

I did this a couple years ago with a client where it took a half hour just to

get in the building, and then once you were in the building, you were monitored,

because physical access is king, right?

all thanks again for, chatting about, one of my favorite topics

I know.

I love these topics.

Come on.

I was talking about woodworking

Oh.

I'm glad to see at least you're using the tool which I

guilt-tripped you into buying, so

Yeah, it was a lot of fun.

I made, for a preschool that I work with, I made these, plexiglass

windows that were… I m- I made window frames, out of two-by-threes.

Yeah

They were, yeah, two-by-four plexiglass window frames.

I made seven of them, and they were incredibly inexpensive because I

did them with all of my own tools.

And, yeah, it was very cool.

All right.

folks, thanks for listening.

if you didn't listen, why would we even do this?

that is a wrap

The Backup Wrap Up is written, recorded, and produced by me, W. Curtis Preston.

If you need backup or DR consulting, content generation, or expert witness

work, check out backupcentral.com.

You can also find links for my O'Reilly books on the same website.

Remember, this is an independent podcast, and any opinions that

you hear are those of the speaker and not necessarily an employer.

Thanks for listening