Ransomware Recovery: Lessons from Surviving an Attack

The unpleasant reality of the modern digital environment means that you can do everything right and still get hit with ransomware. Prevention is important, but it’s just as important to know what to do if and when it happens. This real story of a ransomware attack reveals how these incidents unfold behind the scenes and what leadership wishes they knew before everything went down. And it’s full of important advice to make you better prepared for ransomware recovery.

See Surviving a Ransomware Attack with Zach Lewis for a complete transcript of the Easy Prey podcast episode.

Zach Lewis is the Chief Information Officer and Chief Information Security Officer for the University of Health Sciences and Pharmacy in St. Louis, Missouri. His career has spanned multiple industries, from tech startups to energy plants, including about a decade in higher education. A few years ago, he guided the university through a ransomware attack, and now his goal is to tell the world about what happened, why, and what we can all learn from it. He wrote a book, Locked Up: Cybersecurity Threat Mitigation Lessons from a Real-World LockBit Ransomware Response, about the experience, and has spoken at events across the United States and internationally.

Why Zach Shares His Ransomware Recovery Experience

When Zach joined the university, he quickly realized that they didn’t have a security program. So he started assembling tools and people and building one from the ground up. Then they were hit by a ransomware attack from LockBit, who in 2022-2023 were the most prolific ransomware group in the world. LockBit stole some data, eventually claiming to have several hundred gigs of university files that they would release on the dark web if the university didn’t pay. Zach’s small team had to activate what resources they could to deal with the situation.

Everyone in cyber will probably run into ransomware at some point in their career. But when Zach looked around, he didn’t see a lot of conversations about how teams decide whether or not to pay or any of the logistics of dealing with it in the moment. After getting through the incident, he started talking about it. A lot of people had questions. He would get cut off at conferences because there was so much interest. So he decided to write a book, where he laid out everything he and his team explored, went through, and even felt.

There’s an attitude in the industry that if you were attacked, you’re not a great practitioner. So people hide it. But the criminals are out there sharing techniques and tips. If we can normalize talking about it and share our resources, we might be able to get ahead. We haven’t been overly good at preventing ransomware. We’re good at responding, and getting better. But as these attacks keep happening, we have to get good at ransomware recovery.

The threat actors are out there sharing what they’re doing and who they’re compromising. If we can get ahead on this, maybe we can bring attacks down.

Zach Lewis

The Start of a Ransomware Attack

Zach got the initial phone call at 4:30 AM on April 13. A bunch of on-site resources weren’t available and some of his team couldn’t get into things. They were dealing with a lot of end-of-life hardware at the time, and as the world was in the middle of the covid pandemic, it was hard to source materials. Zach immediately assumed that their old equipment had stopped working, and they would have to find a way to manage until they could get new equipment.

He assembled the team, and they go into disaster recovery mode. They spent a couple days getting servers set back up and recovering from backups. They actually managed to get almost the whole university running from their backups. The university was a big SaaS university, so the main resources students and faculty used were still functioning. Most of the campus didn’t know that they were struggling on the back end. The team got everything back up, and the next day, it went back down. When they went back in to do recovery again, that’s when they saw everything was encrypted and a README ransom note.

Zach thinks that at that point, the attackers thought they were on to them. Something in the recovery process made them think they were caught. So instead of continuing to steal data, they encrypted everything. The first time, nothing was encrypted – the attackers probably accidentally knocked something over while they were poking around. They later found out through forensics that the hackers emailed eleven people saying “We have your data, pay a ransom.” But everybody thought it was spam and didn’t even report it.

The Ransomware Response

When they initially thought it was a hardware issue, the restoration process they went through worked really well. They recovered almost everything and got it back up. When it went down the next day, Zach’s first thought was, “Wow, when they say end-of-life equipment, they really mean it.” Going back in and poking around, they found the README file. It said not to contact law enforcement or cyber insurance, just pay.

Zach remembers the feeling of the pit in his stomach when he found the note. When you’re having an issue in your systems and find a note from a ransomware group, it’s one of the worst feelings you can get. He wrote in Locked Up about what he was thinking and feeling. At that point, they stopped disaster recovery and moved into incident response and ransomware recovery.

Zach had to notify the leadership team, and they came together to determine next steps. Most of these were already laid out in the incident response plan he’d developed, which mostly said to contact insurance. Standing in front of executives and the board and explaining that even though they have a security program, the bad guys got in and they got some data but you don’t know what or how much, is never a conversation you want to have.

So Zach’s first call was to cyber insurance and let them know they had a ransomware attack and needed help. Then he reported it to CISA and the FBI through an IC3 report. Then he called his wife and told her that he’d be late – and to try to cut down on spending, because this might be a resume-generating event.

Cyber Insurance Responds

The next day, the university’s cyber insurance connected them with forensics experts, negotiators, and lawyers with lots of cyber experience. They moved fast, which was great. The FBI called within a few days looking for more information. Zach sent what they had, including the README file and some IP addresses they’d managed to find. The FBI were building a case against LockBit, and Zach likes to think some of the info helped, because just a few months after the attack they took down LockBit. CISA called after that and offered resources. Zach didn’t take them up on it out of the gate, because they were using stuff from the insurance company first. But they were there if needed.

Regardless of whether or not you use the FBI or CISA’s resources, Zach thinks it’s incredibly worthwhile to notify them as part of your ransomware response. When the FBI took down LockBit, they recovered some ransoms and decryption keys. They ended up reaching out to the university and asking if they still needed them for anything. At that point, they didn’t. But if you had been attacked recently, that’s a huge win for you. And if you’ve been attacked, they can help.

Making Ransomware Recovery Decisions

The main factor that would affect the ransomware recovery process was what data the attackers had. LockBit was ransoming both the encrypted systems and the data they claimed they’d stolen. Zach and his team needed to know what the data was. So early negotiations focused on getting lists of files and trying to verify what was in them. If none of the data was sensitive, they might not pay.

LockBit gave a dark web link to a forum chat to negotiate with them. They had great support, operating like a well-oiled crime machine. While the negotiators were trying to get more information, Zach’s team focused on figuring out where that data could have come from. If they could figure that out, they might be able to figure out what was in it even without LockBit’s cooperation.

A big question was when to notify faculty, staff, and students about what had happened. They ended up waiting a bit because they had that flexibility. They had some time to figure out where the files came from, what was in them, and decide if they were going to pay. Waiting until they had a better idea and could answer questions seemed smart.

Whether or not they would pay the ransom came down to what data the attackers had. The university didn’t stop operating – people were still taking classes, doing exams, and teaching. From a business standpoint, that was great. The big question was whether or not the attackers had intellectual property, research information, or student PII. They had systems specialized to keep that data safe, but at the time they didn’t have a picture of what was on their servers. Just because you have a policy doesn’t mean people will always follow it. So they could have stolen some extremely sensitive information.

To Pay or Not to Pay?

The negotiators managed to get a text document with file paths. LockBit said it was 5-10% of the files they took, and they said they had some good stuff. Since the systems were encrypted, all Zach and his team had to go on were the file paths and names. They were trying to figure out where they were from and what they might have.

But they also couldn’t figure out where all this data had come from. LockBit claimed that they had almost 300 gigs of data that they’d stolen. Zach and his team didn’t think there was a way for them to have taken that much data without setting off some kind of alarm. They couldn’t find any indication that 300 gigs of data moved from the university to somewhere else.

LockBit eventually put up a drop data date on the dark web. This is when a threat actor posts on a dark web forum your company’s name, logo, a bit about them, and the date that they’re going to release the data they’ve stolen. For Zach’s university, it was sometime in June. But after going through the file list and looking at where the data came from, the university decided not to pay. And when the data eventually did drop, they had only 2.5 gigs. The file listings they had given were literally all they had. They had just been bluffing.

And the files ended up not having much in them, either. Zach went through them, and all they had of importance were some grades and four social security numbers of past students that shouldn’t have been stored on the servers anyway. The university reached out to those students, gave them recommendations, and did credit monitoring. In terms of a ransomware incident, this is the best-case scenario.

Notifying Campus

It was about five weeks between the incident and notifying campus. At that point, they had a pretty good idea of what was going on. But they didn’t yet know what data LockBit had and couldn’t tell people if they were compromised or not. LockBit told them the drop data date was coming, and they wanted to get ahead of that. Some people scrape that data, and that’s when the media starts asking questions.

So they let campus know, including faculty, staff, students, and alumni across multiple emails. Lots of questions came back, and Zach’s team didn’t have many answers yet. And something that happens when people know there’s been an attack that every problem comes from the attack. Can’t find a file? They blame the attack. Keyboard isn’t working right? It’s the ransomware. Some people claimed that there was money missing from their bank account and that was the university’s fault because of the ransomware. IT got a lot of heat for a bit.

The Final Cost of Ransomware Response

All together, the university probably spent around $300,000 on everything. The cyber insurance deductible covered a lot of resources. But that number also includes cost in personnel – working late hours, pulling technology and marketing team members off other projects, getting internal council involved, having leadership team meetings, and all the lost productivity while people focused on mitigating the issue.

They also had to buy some hardware and set up some new services. Even some back-end things like building automation and lighting control got affected by the incident and had to be rebuilt from scratch. Insurance covered a lot of costs, though not all, and were able to get additional people in to help. It definitely mitigated some of the costs. This isn’t true of every policy. Make sure you know your policy and what’s in it, because that will be crucial to your ransomware response.

Sometimes Paying the Ransom is the Right Move

In this particular incident, the attackers got 2.5 gigs of not very important data. That’s not very much in quantity, but what’s in it is crucial. If it’s some bit of intellectual property or research that’s going to make you millions, you don’t want that exposed. Or if it’s the keys to your castle or you need it to operate, that should all be taken into consideration. Nobody wants to pay the ransom. But there are times when it can make sense from a business standpoint. Universities have closed because of ransomware. People have died because a hospital was attacked. These are things you have to consider.

There are times where, as a business decision, paying that ransom could be justified, even though you really, really don’t want to.

Zach Lewis

In this incident, the university was fortunate that they could continue running while it was happening. But that’s not the case every time. Attacks are increasing every year, and we’re not getting any better at stopping it. You’re probably going to run into it at some point. Zach’s university managed to recover from this ransomware incident. But it wasn’t as simple as wiping over the encrypted data and reinstalling from backup. Ransomware recovery is more complicated and has more factors involved than that.

When You Can’t Rely on Backups

Finding the README ransom note wasn’t the only sinking feeling Zach had through this incident. The other one was realizing that the attackers had managed to find one of their backups and completely erased it. The university had run a tabletop exercise with leadership and the IT teams earlier in the year. They had a plan and everyone was familiar with what they should do. But they didn’t count on having issues accessing the backups.

They had another backup. But in order to access it, they had to authenticate it. Normally they authenticate with Active Directory, but that was encrypted and down. So they tried to do a local login. That required a password. And because they ran a local password manager, that was also encrypted and down. The backups were just one login away, but they couldn’t access it.

What saved them was a tertiary, fully offline, fully separate backup. And they even got lucky with that, because it was also password-protected and the password manager was down. But by happenstance, one of the team had the login saved somewhere else. It was probably against policy, but it was a massive benefit in this case. There’s no easy way to get into a backup from the outside, because otherwise attackers could do the same. If they hadn’t had that offline backup and hadn’t had another way to access that password, they wouldn’t have been able to restore everything.

Ransomware Recovery Lessons

One interesting lesson was how incidents happen. In this case, it happened because of a perfect storm of issues. The attackers got in through a compromised account, accessed the VPN, moved through the system laterally, found cached credentials on a machine, and got access that way. It’s very rarely zero day issues – it’s more often misconfigurations, compromised accounts, social engineering, and other basic stuff.

The university’s hardware woes were one factor. They also had problems with their firewall, and had to switch manufacturers multiple times until they got one that functioned right. Because of all this switching, the team had to do a lot of copying and pasting configurations. One of them didn’t come through quite right.

Internet interruptions were also an issue, and the only thing they could do at the time to get a stable internet connection was a co-managed solution with another provider. They got everything set up and working, but when it came time to secure it, there weren’t as many options. Because the co-managing solution was so new, it didn’t have all the features enabled yet. Multi-factor authentication (MFA) over VPN wasn’t an option.

Normally, MFA would have been able to prevent an attacker from getting in the way they did. And if the firewall had had the correct configuration, it would have stopped the infiltration. But everything lined up just right for the attackers. And even though the window for all of those issues was only a few months, LockBit still managed to find it, get in, and steal data.

Validate Everything

Just one or two small changes could have prevented this whole incident. Make sure you have MFA on all accounts everywhere. And aside from that, check and validate everything. They were focused on getting the internet stable and the firewall functioning. They didn’t go back and double-check that all the configurations came through properly. Any time you want to do something, build in time to check, validate, and make sure it’s working as it should be. Don’t just trust the copy and paste worked, you have to check. It’s a good lesson learned.

Build a little time into your projects to validate those configurations are in place. … Don’t just trust that the copy-paste worked.

Zach Lewis

Have a Plan

You’ll find it extremely difficult to do any sort of ransomware recovery if you don’t have an incident response plan. And it needs to be a good one with detailed information. It should tell you where critical data is, what servers you restore first and in what order, and the number for your cyber insurance. Zach always makes sure he has a printed copy, and now he keeps key local login passwords there, too. He never wants to be in a situation where the backup is there but inaccessible. They’ve changed how they handle password managers now, but he still keeps the passwords in the printed plan. He recommends keeping both your incident response plan and your disaster recovery plan in hard copies in a safe place.

Know How You Recover

Having the processes is great, but you also need to dig into the granular details. One of the tabletop scenarios they had run previously was what to do if one backup was compromised. They stood up another backup and connected to storage to pull stuff. But in the middle of doing it live, it needed a product registration key. That wasn’t something they’d covered in the tabletop scenario. Nobody knew where the key was, and it took time and stress to track it down. Think about those kinds of things, down to the smallest detail, in advance. It will make the whole process much faster and less stressful.

Have Communication Options

Having out-of-band communications was a huge factor in this ransomware recovery. If your environment is compromised, attackers could be reading your emails and chats – if you can even access them in the first place. You have to pivot to other forms of communication so they don’t know what you’re doing. Which means you need to have access to that information, ideally somewhere it’s not going to be encrypted in the attack.

Backups and Cross-Training

Have lots of backups in lots of places. Zach does quarterly backups now, and makes a point to follow the instructions to spot where they’re outdated and needs adjusted. And make sure your security team, IT team, network infrastructure team, and everyone who might be involved is cross-trained. Everybody has to know how to do things. Otherwise you run into situations where John’s on vacation and Steve is home sick and nobody knows how to implement any of the ransomware recovery processes.

Prepare as Much as You Can in Advance

We can’t always prevent ransomware. But we always want to be able to successfully recover from ransomware. Have multiple backups in lots of places. Have multiple hard copies of your plans in multiple places. Even keep one of them outside the IT department. Zach gave one to the Chief Operating Officer. That way, even if for some reason IT blows up, they have a copy to give to outside consultants to help with ransomware recovery.

I might not be able to always prevent ransomware, but I want to make sure I can recover from any incident that pops up.

Zach Lewis

Have a plan for contacting people. Zach’s incident response plan also includes communication. Do you know who your cyber insurance provider is? What about how to reach the outside IT contractors you normally use? Do you know how to contact all the employees and third parties you’ll need to contact? And not just you, but does your team know?

Numbers, systems, any information that might be useful if you can’t access anything digital anymore should go into this document. It makes it a big document, but it’s useful when you need it. Ransomware is stressful. The last thing you want is to be flying blind. Having the plan in advance and everything written down will help you find the information you need and know exactly what to do. When you don’t have to think about it, it reduces some of that stress and helps you get moving.

Talking About Attacks and Recovery

Zach’s main goal with his book Locked Up is to have conversations about what really goes on and how ransomware recovery works. He’s talked about this incident all over the place. When he gives presentations, one slide asks the audience to raise their hand if they’ve ever had a cyber incident. Nobody raises their hands. But when he provides a QR code to report it, suddenly most of the room has. It’s way more common than we think, and we need to talk about it more.

It was more common for people to share information. CISA had a control that protected companies from antitrust laws when sharing this info, and that’s expired. So from a legal standpoint, people are more skittish about sharing again. But in the private chats Zach has had with people across the country, people still want to share. He’s trying to break down walls.

In the book, Zach goes through the details and explains why they made the choices they did. Every incident is different. Hopefully from the book, you can get an idea of what you should be thinking about when you respond to ransomware and why.

You can connect with Zach Lewis on LinkedIn, where he is very active. He also has a website, homesteadingciso.com. The book, Locked Up: Cybersecurity Threat Mitigation Lessons from a Real-World LockBit Ransomware Response, is available wherever books are sold.