Deepfake Technology is More Realistic (and Dangerous) than Ever

We’ve all heard the old phrase, “Seeing is believing.” But deepfake technology has gotten us to a point where that’s not true anymore. Scammers are using cloned voices, digital impersonations, and even AI-generated faces on live video calls. And these fakes are so good that it’s almost impossible for humans to spot them. Learning how these attacks work and why they’re so hard to identify can help you stay ahead.

See Convincing Deepfakes with Tom Cross for a complete transcript of the Easy Prey podcast episode.

Tom Cross is the Head of Threat Research at GetReal Security, where he leads research in deepfake-based social engineering and digital impersonation. He’s been researching internet security, security vulnerabilities, and how people exploit software issues to break into systems for over twenty-five years. Tom has worked at a variety of computer security software companies, and he often speaks at conferences about understanding threats and vulnerabilities, building mental models to think about threats, and complicated or sophisticated attack scenarios. In his role at GetReal Security, he specializes in deepfake technology, researching deepfake-based social engineering and digital impersonation and building software that detects it. His job is to understand how threat actors are using it and develop protections for those threats.

The Computer Security Mindset

Tom’s interest in computers started in the 1990s, when he ran a bulletin board system (BBS) for his friends. They played games, shared text files, and as was common in those days, had a lot of discussions of hacking. He found it fascinating. In college, he studied computer engineering, and found the security parts of it especially interesting.

A lot of his interest in computers boils down to what Tom would call the hacker mindset. It isn’t necessarily criminal – people use it in a lot of different days. It’s just a particular way of looking at things. Tom describes it as trying to see technology as what it really is, as opposed to what it’s meant to be. People expect computers to work in particular things – that’s what they’re meant to be. If you see them for what they actually are, that can let you see ways to get them to do things they’re not supposed to be able to do. Tom has always found that fascinating. And he’s tried to use it for good, doing things like vulnerability research to spot those exploits and fix the problem before someone malicious uses it for evil.

People expect computers to work a particular way … if you see what they really are, sometimes that reveals ways to get them to do things that they’re not supposed to be able to do.

Tom Cross

Part of this mindset is what makes Tom always want to be on the leading edge of technology. That’s how he’s ended up working with deepfakes. They’re fascinating from a technical perspective, and he gets to work with interesting people. Hany Farid, one of the co-founders of GetReal Security, is a professor at Berkeley and spent his career on image forensics – techniques to determine if an image has or hasn’t been photoshopped. The team at GetReal Security includes experts in audio and video processing, stream processing, stenography, machine learning, and more.

They’re operating on the leading edge of AI tech. It’s a very interesting set of things to learn about. It’s also very relevant, because deepfake technology is an increasingly dangerous threat. The tech keeps advancing, and every few months it comes out with new capabilities.

Different Types of Deepfakes

Audio deepfakes, even real-time ones, are pretty easy to pull off. If you have 30 seconds of someone speaking, you can easily train an AI model to replicate that voice. These voice samples are pretty easy to get if someone does any kind of public speaking – or if they’ve posted a video on social media. The AI is open-source, so it’s easy to download it and train it with that voice. It’s only a little more complicated to have a computer turn your voice into someone else’s in real time.

Fully AI-generated videos exist. But they take a lot of processing power, so they can’t (yet) be done in real time. But we’re still seeing real-time deepfakes. One way is through face-swapping, where the AI puts someone else’s face over yours. You can talk and move, and the face will do everything you do. It’s sophisticated technology, but there are plenty of free and open-source apps that will let you do it. Recently we’ve started to see fully AI-generated avatars. And just a few months ago, we saw the first example of injecting deepfake video into a live video call.

There are lots of legitimate uses for this kind of technology. But they’re also something criminals are using. And as these tools become more common, they’re going to open up more opportunities for bad guys. This tech is just going to get more sophisticated as time goes on.

There are lots of legitimate uses of [deepfake] technology. People are doing all kinds of creative things with it. But unfortunately, it is something that criminals are also taking advantage of.

Tom Cross

How Good are Deepfakes, Really?

The short answer: Really good.

Especially when it comes to audio, humans aren’t able to distinguish deepfake voices from the real thing. Someone recently did a deepfake of the voice of the CEO of the cybersecurity company Wiz using the audio from a public presentation. It didn’t work, but only because the employees had all talked to the CEO personally, and he had a very different affect in conversation than when presenting. But the voice was identical. It’s a situation where you had to know the person personally to identify something was wrong.

A human being is not going to be able to distinguish [deepfake] voices from the real voice that they are simulating.

Tom Cross

Deepfake technology can also fool biometrics. Research has shown that AI-generated faces and voices pass biometric authentication. Doesn’t matter if it’s open-source or commercial, the deepfakes pass.

Research shows that deepfake faces and voices pass biometric authenticators.

Tom Cross

Ultimately, it is possible to detect deepfakes. But humans can’t do it. You have to use AI to spot AI. Specialized models can detect subtle traces in audio and video processing and signal that say there was AI involved. But they’re so subtle you can’t even really describe them to a human. It’s the same thing with AI in medical diagnostics. AI can spot patterns subtle enough that it’s hard for human doctors to identify it even when they know it’s there. For instance, if a background is real, the camera will get a slightly different number of photons in each frame. But a virtual background is rendered by the computer, so it’s the same in every frame. Humans can’t really see the difference, but it’s something that software can detect. These are the kinds of tools we need to detect deepfakes.

The Arms Race of Deepfake Technology

If a computer-generated image is identifiable because it’s too perfect, getting the deepfake software to add imperfections could make it harder to detect. That’s part of the “arms race” of cybersecurity. Criminals come up with a technique, security people find something to protect against that, and they update or change the technique to get around the security. Hopefully over time we’re reducing the problem. But there’s such a desire to commit fraud that it will keep getting more and more sophisticated.

For the most part, the people making these AI tools aren’t making them specifically for criminals. Most of them have legitimate use cases. People want to talk to AI agents, so AI agents can listen to and translate human speech and talk back. They’re designed to look and sound human to humans, but they’re not designed to trick detection software. Legitimate use cases don’t need to fool AI detection. The people building deepfake technology don’t necessarily want to find a way to make it undetectable.

A number of these companies are actually including ways to make their technology detectable. They’re adding digital watermarks in content their AIs produce so detection software can easily tell it’s AI generated. There’s a watermark right there that says so. Tom thinks that’s a responsible thing to do, and would like to see more companies do it. Some criminals build their own AI, but many more get access to legitimate software or use open-source software for crimes. Adding these safety features would help.

How Criminals Use Deepfake Technology

The most well-known criminal use of deepfake technology is scamming consumers. Romance scams and investment scams are big ones. Candidate fraud against companies is a big one, too. North Korea especially likes getting remote jobs at US companies and funneling those salaries into their weapons development programs. They use deepfakes to disguise themselves – sometimes having multiple people doing the work but with the same AI-generated face to look like they’re the same people.

Deepfake impersonation against financial services also happens. When you set up a bank account online, the bank has to go through a Know Your Customer (KYC) process to verify your identity. Criminals use deepfakes to get around KYC checks so they can have verified bank accounts. They often use these for money laundering. Identity thieves can use deepfakes to even more thoroughly pretend to be the real deal. They may build up a person’s credit to later take a big loan in their name, or may use a deepfake with a stolen identity to fool KYC processes.

There’s this wide swath of ways that deepfakes are being used, and sometimes they’re used to impersonate a specific person.

Tom Cross

Deepfakes are also great for social engineering, like in the case of the Wiz CEO. They can let you look like someone else, whether you’re trying to pretend to be a specific person or just trying to hide your identity. They can even make you more attractive for a romance scam.

Everyone is Vulnerable to Scams

Criminals exploit people’s desires. According to FBI statistics, investment scams were the most lucrative type of internet crime last year, stealing something like $8 billion. It’s because everybody wants to make money. If someone’s telling you there’s an opportunity to make a lot of money, who doesn’t want that? When you want to believe something, you’ll look past red flags because you’re emotionally invested in it being true. Romance scams are the same. We want to believe Keanu Reeves is in love with us, so we rationalize away the fact that he always seems to need money.

Scammers are putting out a huge volume of attempts. Most of it doesn’t work, but that doesn’t matter, because it’s cheap to fire stuff out and if even one in a thousand people takes the bait, they win. And the people who take the bait aren’t just dumb or ignorant, either. If the attack happens to line up with what we expect, everybody will get caught.

A common scam targeting businesses is a criminal contacting a company and saying, “Hi, I’m your supplier, my bank account info has changed, next time you pay me please send it here.” Companies often have good processes around this. But when an attack hits when the CFO happens to be on vacation, it gets through. Every once in a while, there’s the perfect opening, and the scam works. You may not fall for a fake customs message on an average day, but if you recently ordered something from China, it fits with what you expect, so you’ll click.

Every once in a while, there’s an opening … every once in a while you get a phishing email that’s exactly aligned with what’s going on in your life right now.

Tom Cross

Deepfakes are Letting Criminals Scale

The biggest benefit that deepfake technology gives criminals is scale. Criminals want to do crime at as big a scale as possible, because that’s how they make money. Right now, there are scam compounds in places like Southeast Asia where people are held against their will and forced to scam. Thousands and thousands of people, literally with guns to their heads, are out there trying to turn others into victims. They’re using deepfake tools to hide their real situation and trick people into sending money to scammers.

Remote work also provides an opportunity. North Korea has been putting people in remote roles in US companies using deepfake faces. These people work in small groups with the objective of just making money. There’s often more than one person supporting the job. Sometimes there’s a person in the group whose entire job is applying for jobs, getting interviews, and getting hired for more jobs. These groups may be supporting four or five full-time jobs per person. With deepfakes, they can appear as the same person in every meeting while not always being the same person behind the camera.

In Tom’s profession, he thinks a lot about vulnerability and where weaknesses are. But the fact that a weakness or vulnerability exists doesn’t necessarily mean criminals are going to exploit it. They want things that they can build that work over and over again. They want to build processes and scale it. A vulnerability that doesn’t work repeatedly isn’t very interesting to them. Then it becomes a trickle-down effect – other criminals see one group doing something and decide they want to get in on the action.

The fact that a hole or weakness or vulnerability exists does not necessarily mean that a bunch of criminals are going to start exploiting it.

Tom Cross

Techniques to Spot Deepfakes

Years ago, the advice was always to jump on a video call, because scammers wouldn’t do that. Then it shifted to they would do it, but would have excuses about why they couldn’t turn on their camera. Now, deepfake technology lets them look and sound like whoever they want in real time. The advice there was to have them put their hand over their face, or open their mouth, or interact with something from the background. But deepfakes have worked out most of those issues now. And the tech is advancing so fast that whatever works now to spot the videos may not work in six months.

Regardless of the technology, though, scammers will always pressure you to make exceptions to your policies. That comes with some guilt. They might say they can’t turn on their video because they’re at their mom’s place because she’s not feeling well. With fake employees, they will often give a story like “I had a family emergency so I’m out of town, can you ship my work laptop to this address instead?” In reality, the home address they gave is a random one they have no access to, but the other address is a reshipper who will pass the laptop onto the fake employee. You feel bad and want to agree. But that’s how scammers get around security policies.

One thing you can do is to ask people on video calls to turn off their virtual background and show the room they’re in. If they refuse, something’s up. It might reveal they’re in a call center and not who you thought.  That’s an important sign, so it’s important to be aware.

Inaccurate Confidence in Spotting Deepfakes

A survey of IT professionals recently showed that most of them thought they’d have no problem detecting a deepfake. But that confidence is misplaced. Tom thinks most people don’t understand just how fast deepfake technology is advancing. If you’ve been exposed to them at some point, you may be confident in detecting them – but if six months have gone by, whatever signs you think you can spot are now obsolete. AI images and videos used to have weird issues with basic human anatomy, but that’s not the case anymore. They’re incredibly realistic now.

People don’t understand how fast this technology is moving. They may have had exposure to something at some point, and … six months has gone by and that thing is totally obsolete.

Tom Cross

You used to be able to spot a deepfake face on a video call by asking the person to put their hand or something else in front of their face, or move around a little bit so the lighting changes. Today’s deepfakes have no problems dealing with that. Even complex visual things like around the edges of objects isn’t going to be an issue a year from now.

The people building these tools want them to be as realistic as possible from a human perspective. Any things humans could spot are going to go away soon, if they haven’t already. And academic studies show that humans can’t tell the difference between deepfake and real in a controlled study situation. It gets even harder if you’re emotionally invested in believing something. If you think you’re about to make a lot of money, have finally found the perfect candidate, or met the love of your life, you’ll find ways to rationalize it.

What Security Professionals Need to Do

Tom thinks the security community needs to integrate the ability to determine real from not real into the internet itself. That’s a big project with a lot of facets. It’s not just about scams, but also disinformation that manipulates people’s understanding of reality. Tom wants his computer to protect his ability to understand the universe, not erode it.

Douglas Engelbart, one of the fathers of modern computing, thought that computers were supposed to augment human intelligence. Tom doesn’t know if that’s happening – right now, they might be making us dumber. Computer scientists need to think about that. They need to consider how computers currently operate, what they can do to support us and make us smarter, and help us determine what’s real and what isn’t.

As computer scientists, we’ve really got to think about … how our computers can do things to support, to make us smarter, and make sure that we know what’s real and what’s not real.

Tom Cross

The Future of Deepfake Technology

Tom and his team recently did an experiment where they took an old internet softphone, connected it to a voice deepfake controlled by an AI, taught the AI how to do social engineering, and had it run IRS scams. To Tom, it was a bit of a future shock thing. Everybody has at some point called a customer service center and gotten to an AI that was just stupid. We all want to get through to a human because the AI just isn’t helpful. But this technology was much more sophisticated. It was actually emotionally responsive – if the person on the phone was compliant it would be nice, and if they were aggressive or said it might be a scam the AI got aggressive back.

Most people haven’t experienced an AI that can interact conversationally and adapt like that. But it’s easy to do. Tom’s team didn’t spend a lot of money to do it. It was just a little project they wired up. And they also found different levels of guardrails with different companies. Some caught onto what they were doing quickly and stopped it after just a few hours. Other companies and models never seemed to realize what was going on.

The Tech is Just Going to Get Better

The deepfake technology we see today is already so sophisticated. But that power is going to double in the next year or so. Whatever seems exclusive and high-tech today is going to be commonplace next year. Expect to see more criminals doing more automated scams. Scam compounds might even shift over from mostly kidnapped and confined people to mostly computer systems targeting hundreds of thousands of people a minute. We’re going to see so much more volume because it’s going to become so easy to produce.

The power of this technology that we’re seeing today, it’s going to double in 12 to 18 months.

Tom Cross

It’s also going to get even better at convincing humans it’s real. We’ll soon get to the point where any identity could be fully automated – talking to you in your voice, moving through whatever setting you want in real time, responding to you. We’re right on the edge of all of that being available in video and audio. Imagine what the next few years are going to look like. It’s going to be scary.

Anybody’s identity could be fully animated, talking to you using their voice, moving around in whatever setting you want in real time and responding to you.

Tom Cross

Machine learning is also going to give it a boost. Humans just don’t have the capacity to comprehend the scale of this. It will become a self-evolving threat. We’re on the edge of a situation where computers improve themselves. In lots of ways that’s good, but when it comes to threat actors that’s very, very bad. Scammers will be able to try millions of things, see what works best, and double down on the most effective ones. Scams are going to get very, very good very, very quickly.

AI Lowers the Bar for Criminals

In order for a potential criminal to start doing crime, they have to have three things: Means, motive, and opportunity. The biggest holdup for many of them is they don’t have the means. They have the motive, and there may be an opportunity, but they lack the time, space, or energy to develop a full criminal enterprise.

AI-driven threats are going to lower the bar. Before, getting into cybercrime required being able to write software or knowing people who do. But these days you can get a free AI chatbot to write the code for you. It gives a lot of people opportunities to do great, creative things. But it also makes it much easier for criminals to get started. Lots of AI companies are adding guardrails so models won’t do certain things. But there are still ways around those – and if you run a model locally on your computer, there are no guardrails at all.

Once people started being able to run their own email servers, criminals started spamming, and the internet is still full of spam. Email is no longer really effective person-to-person communication because of all the spam. Deepfakes may do the same things with phones. Tom is already annoyed with the amount of spam texts and calls on his phone. Is it going to become unreasonable to have a phone number anyone can call? Will we need a phone social network where you have to be friends with someone before they’re allowed to call you? It’s possible.

We Need to Certify Deepfakes

A big thing that we need is a way to certify content. There needs to be a way to identify if this video on the internet is real or if it’s a deepfake and that person didn’t really say that. But on the other hand, if a celebrity or politician puts a deepfake on the internet, we need a way to verify it’s an “authorized” deepfake. Because there are legitimate uses of deepfakes.

In India, politicians often use deepfakes for official communications to their consistencies. The country has a huge number of languages. Deepfakes will let the politician give a speech in a single language, but then generate twenty deepfake versions in other languages so all the communities in their area can hear what they’re saying in their own language. These are authorized deepfakes used in a political context. So we need a way to identify when a deepfake is a definite fake versus when it is an AI-generated video but legitimate and authorized by the person being represented.

A number of responsible AI companies are embedding digital watermarks in their deepfake videos so people know they’re fake. Tom focuses on threat research, so the benefits of deepfakes can sometimes get lost. But it’s interesting to think about how there’s a lot of benefits to these things. We need to find a way to meet in the middle and make an internet where we can better understand what’s real, what’s fake and duplicitous, and what’s fake but authorized.

Follow Tom Cross for professional updates on LinkedIn. You can also visit the GetReal Security blog for more information on deepfakes. Tom recently wrote a blog post dissecting a Chinese criminal deepfake tool primarily used for romance scams and explaining how it works.