AI Phone Scams: Here’s How to Protect Yourself from Voice Cloning Scams

Phone call-based scams are nothing new, but it seems like scammers are working overtime to find new ways to cheat and steal from their victims. The latest phone scams involve AI voice cloning to trick people into thinking that they are being contacted by a loved one. 

Protecting yourself from AI phone scams requires an understanding of what these scammers are doing. 

AI is getting more sophisticated, and so are the scams. It’s important not to assume that you could never be tricked! The more you know about how these scams work, the better you can protect yourself. 

How artificial intelligence scams use voice cloning

AI is used to create text, images, videos, and audio. There are a lot of positive ways to use AI, like improving productivity and assisting people with disabilities, but there are also risks and drawbacks. 

One of those drawbacks is the ability to copy or clone a real person’s voice and make it sound like that person has said things they never said. This is dangerous because scammers can use this to trick people into thinking their loved one is in trouble and needs money to get out of an emergency situation. 

In early 2024, The New Yorker published an article about a stressful and traumatic experience a couple went through because of scammers like this. 

Robin and Steve, a 30-something couple from Brooklyn, received a late-night phone call that appeared to be from Steve’s mother, Mona. They heard what sounded like Mona’s voice wailing, followed by Steve’s father Bob’s voice telling them to get Steve on the phone. When Steve answered, a man claimed he was holding Steve’s parents hostage and demanded money.

Believing the situation was real due to the familiar voices, Steve complied with the demands while secretly contacting a colleague experienced in hostage negotiations. The scammer asked for $500 via Venmo, then an additional $250, which Steve sent.

After the call ended, Steve and Robin contacted Bob directly and discovered that his parents were safe in bed at home, unaware of any incident. The couple realized they had fallen victim to a sophisticated AI-powered voice cloning scam.

Unfortunately, this type of scam is becoming increasingly common as voice cloning technology improves. Scammers can now create convincing voice replicas with just a short audio sample, often obtained from social media videos.

More examples of how scammers use AI voice cloning 

The New Yorker article goes on to explain examples of other scams, including: 

• A corporate attorney in Philadelphia received a call from what he thought was his son, claiming to need $9,000 for bail after a car accident, but his son never actually made that call.
• Jennifer DeStefano from Scottsdale, Arizona, got a call that sounded like her daughter Brianna, claiming she had been kidnapped. The scammer demanded money and made threats.
• DeStefano’s mother received a call from someone claiming to be DeStefano’s brother, asking for money to pay a hospital bill related to a car accident.
• RaeLee Jorgensen, a teacher’s aide, received a call from what appeared to be her 14-year-old son’s number and voice, followed by a threat from another person claiming to have kidnapped him.
• A friend of DeStefano’s received a call that sounded like her 9-year-old son claiming he’d been kidnapped.
• Another mother at DeStefano’s daughter’s dance studio had a cousin who was scammed just two weeks earlier, receiving a call from what sounded like her daughter’s voice and sending $1,500.

As you can see, these scams are complicated and tricky! When a person believes that their loved one is hurt, scared, or in danger, they will often do anything they can to try to help. Scammers do everything they can to take advantage of that instinct to protect and help your family members and close friends. 

How does voice cloning work? 

Voice cloning uses AI to record and then copy a person’s voice. Here’s an overview of how it works: 

1. The scammers obtain a sample of the target’s voice. This usually comes from audio recordings (like videos, voicemail messages, and other recordings), but they can also get someone’s voice by calling and talking to them. 
2. Audio processing is next. The collected audio samples are cleaned up and segmented into much smaller units called phonemes. These are the smallest units of sound in speech. English only has 44 of these phonemes, which means they only have to access enough audio to capture these 44 sounds — then they can turn someone’s words into just about anything! 
3. Once these smaller pieces are obtained, scammers will use AI to analyze various aspects of the voice. The AI model is trained to recreate specific patterns and nuances of the targeted voice. 
4. Text-to-speech programs will then synthesize the voice so that it can say anything it is prompted to say.  

Using these steps, today’s scammers can produce remarkably accurate voices with only a little bit of data. Sometimes all it takes is a few minutes of a conversation or social media video to make this work. 

How to recognize scammers using AI 

Now that you understand just how sophisticated these scams can be, how are you supposed to avoid them? 

To spot these scams, NYC’s Department of Consumer and Worker Protection says to watch out for the following: 

• Calls that come out of nowhere. 
• Pressure to act immediately. 
• Urgent requests for money in unconventional means, like a wire transfer, gift card purchase, payment app (Zelle, PayPal, Venmo, etc), or cryptocurrency. 
• Calls that demand your personal information. 
• Instruction to keep the call a secret.
• Signs that the audio is faked, like jerky language, awkward pronunciation, or strange expressiveness.

The importance of a family password

One of the best ways to avoid falling for one of these scams is to establish a family password or passphrase and make sure that everyone knows it. 

For example, you could all agree that in a real emergency, you will use a phrase that everyone knows means, “Hey, this really is me.” It could be an answer to a question, a specific name, or a phrase. 

Choose a phrase that is easy for family members to remember, won’t come up in normal conversation, and doesn’t reveal any personal information if it’s overheard. 

Now that you are informed about the ways that these scams work, know how to recognize them, and have put a plan in place to protect your family from being conned, you should be protected against these scams!