Real Faces, Fake Identities: The Growing Threat Of AI Impersonation And Why Digital Watermarks Fall Short

Generative Artificial Intelligence (AI) has improved drastically over the years to the point where it has become extremely difficult to identify whether an image or a video is authentic or has been generated with an AI tool. Unlike old deepfake tech from the past that required long hours and top-tier hardware to generate seconds of face swap clips, AI tools mostly run on the cloud and allow users to morph pictures and videos or even generate a new one from scratch with just a simple text prompt.

The possible consequences of misuse of such a technology far eclipse the menace spawned by the old deepfake tech, which relied only on a subset of artificial intelligence, specifically deep learning, generative adversarial networks, and autoencoders to create manipulated or synthetic media.

Generative AI today is present in almost every smartphone, allowing users to generate pictures of varying styles or modify images as per their liking. For advanced use cases, there are freely available tools on the web as well as tools locked behind a paywall that draw power from high-end models from tech giants. While the advent of AI has resulted in an influx of creativity, it is not without its harms—as is the case with almost every new technology, be it blockchain or UPI.

On one side, we have a good chunk of AI-based accounts on social media platforms that rake in hundreds of thousands of followers and sometimes even receive brand deals, helping people behind the keyboard become creators without showing their faces on the internet. Some even generate animated videos based on their script or imagine out-of-the-world scenarios, like what it would be like to cut a volcano in half.

The other side of the coin tells a dark story where AI tools have been used to create an AI persona that falsely claims to be real, deceiving followers into spending real money for an exclusive chat with a fabricated character—a classic upgrade to the "Angel Priya" scam of 2000s that haunted 90s kids on Facebook, where individuals posing as women, manipulated victims into providing mobile top-ups for them.

However, the more sinister variation of AI character is when it borrows the face of a real person without their knowledge—typically to gain followers with the power of a pretty face or for malicious ends, such as exacting revenge by posting risque images, skirting the edges of the virtual boundaries, standing just before nudity. The recent case of "Babydoll Archi" serves as the prime example of how generative AI can be used to induce harm, all while enabling the bad actor to pocket lakhs of rupees in the process.

Babydoll Archi: The influencer who didn't exist

For those unversed, an Instagram account supposedly belonging to a girl in Assam shot in popularity after a video of the creator dancing to the tune of a hit Romanian song Dame Un Grrr went viral. A blue tick, 1.4M followers, erotic posts, and her supposed picture with adult film star Kendra Lust further heightened curiosity. People speculated about her entry into the US adult entertainment industry. There were viral posts suggesting that she was a survivor of a red light area and had helped a lot of girls escape prostitution.

Fake Instagram account of Babydoll Archi (ETV Bharat Graphic)

Ultimately, the entire narrative unravelled as fiction. The character was AI-generated. Yet, the face belonged to an actual human.

Upon complaint from the victim, the police nabbed the person behind the account with help from Instagram. At the time of the arrest, the fake account had reportedly earned over Rs 10 lakh via subscriptions, which were in thousands.

Victim's discovery and police action (ETV Bharat Graphic)

AI experts say that what happened in this case is almost impossible to prevent. Earlier, digital revenge used to involve poorly edited photographs, which warranted disgust but at least could be easily identified. The rise of publicly and freely available artificial intelligence has armed bad actors with a versatile tool that can be misused in countless ways. While AI-powered voice cloning scams largely target victims financially, the use of AI-generated images and videos poses a far greater threat to personal reputation, making it an attractive weapon for vindictive ex-partners and rivals.

In an effort to help people identify AI-generated content, companies like OpenAI, Meta, and Google started marking AI work with an invisible watermark. Earlier this year, Google made its SynthID Detector live, enabling users to discern AI content made using Google tools by detecting SynthID watermark and help them identify images made by Imagen, text generated by Gemini, video made by Veo, and audio generated by Lyria. Google also open-sourced SynthID to expand it to content made with non-Google services.

Invisible watermarks to identify AI images (ETV Bharat Graphic)

In a world where WhatsApp Forwards are still a menace, as people tend to believe what they see and share it with others without hesitation, it seems unlikely that anyone would take the time to check whether the media is generated or modified by AI tools. Still, these tools presented some sort of hope at curbing the problem of AI deepfakes… until new research proved that such encoded watermarks offer no defence against synthetic AI media.

Invisible watermark isn't the solution we thought it was

Last week, researchers from the University of Waterloo's Cybersecurity and Privacy Institute demonstrated a tool, UnMarker, which destroyed these secret watermarks without even needing to know the specifics of how they were encoded, detailed in a paper to appear at IEEE S&P 2025, titled "UnMarker: A Universal Attack on Defensive Image Watermarking".

Researchers claim that the tool requires no knowledge of the watermarking algorithm, no access to internal parameters, and no interaction with the detector, and can still work universally, stripping both traditional and semantic watermarks without any customisation.

AI watermarks offer no real protection (ETV Bharat Graphic)

Urs Hengartner, associate professor of the David R Cheriton School of Computer Science at the University of Waterloo, explained that while watermarking schemes are typically kept secret by AI companies, they must satisfy two essential properties: they need to be invisible to human users to preserve image quality, and they must be robust, that is, resistant to manipulation of an image like cropping or reducing resolution.

“These requirements constrain the possible designs for watermarks significantly," Hengartner said. "Our key insight is that to meet both criteria, watermarks must operate in the image’s spectral domain, meaning they subtly manipulate how pixel intensities vary across the image.”

UnMarker uses a statistical attack and looks for places in the image where the pixel frequency is unusual, and then distorts that frequency, making the image unrecognisable to the watermark-recognising tool but undetectably different to the naked eye, the paper said.

In tests, the method worked more than 50 per cent of the time on different AI models, including Google’s SynthID and Meta’s Stable Signature, without existing knowledge of the images’ origins or watermarking methods, it added.

“People want a way to verify what’s real and what’s not because the damages will be huge if we can’t,” said Andre Kassis, a PhD candidate in computer science and the lead author on the research. “From political smear campaigns to non-consensual pornography, this technology could have terrible and wide-reaching consequences.”

Commenting further, he said if they can figure out how to remove AI-identifying watermarks, so can malicious actors. He expressed concern over watermarking, saying it was being promoted as a flawless solution, but their findings proved the technology could be compromised. "Deepfakes are still a huge threat. We live in an era where you can’t really trust what you see anymore," he said.

Need for multi-layered defence and auto-generated AI labels

Notably, the UnMarker tool is not the first instance when the effectiveness of such digital watermarks has been brought into question, with researchers earlier finding vulnerability in the solution, including Meta's Stable Signature. Talking to ETV Bharat, Azmarq Technovation founder Imteyaz Ansari advocated the need for a multi-layered defence, just like earlier studies, as well as action from regulatory bodies to counter the AI deepfake problem.

"The rise of watermark-breaking tools like UnMarker shows we’re in an arms race between AI creation and AI detection. We now need a multi-layered defence: first, through invisible digital signatures embedded at the model level, not just the output; second, tamper-proof metadata binding across all media formats," he said. "But tech alone isn’t enough."

"Regulatory bodies should enforce mandatory provenance tracking and label AI-generated content, especially when faces, voices, or likenesses are used. The ‘Babydoll Archi’ case is a wake-up call—identity misuse must be treated with the same seriousness as data breaches," he added.

Ansari also emphasised public awareness, calling it the first firewall against AI deepfakes. Highlighting that the average user still trusts what they see and hear online, especially if it’s a familiar face or voice, he said, "Schools, brands, and governments should start digital hygiene campaigns just like we did for passwords and OTP scams."

"Tech companies, on the other hand, must build browser-based deepfake detection plug-ins, reverse-media search engines, and real-time integrity indicators for WhatsApp, Instagram, and YouTube," he said, adding that if we can label foods as organic or processed, why not label content as ‘AI-altered’ or ‘verified real'," he added.

Notably, platforms like YouTube, Facebook, and Instagram ask users to indicate whether the content they upload is AI-generated or has been altered by AI. However, these AI labels aren't generated automatically by the platforms and instead rely solely on the goodwill of the creator. Bad actors, who want to utilise AI-generated content for malicious activities, might not be keen on using these labels.

Stop, protect, proceed

When asked about what ethical standards AI developers and social media platforms should follow to stop harmful fake content from spreading—especially when real people’s faces or voices are used without their permission, Ansari said that we need to move from a ‘build fast and patch later’ mindset to a 'build responsibly by design' culture.

A visual representation of synthetic face rendering (ETV Bharat via Copilot Designer)

"AI developers must be bound by ethical audits that check for potential misuse scenarios during model training and deployment. Platforms should follow ‘notice and takedown’ protocols, but also go further to embed consent-based filters that detect and block unauthorised use of real identities. If someone uploads a video using a real person’s face or voice, there should be a red flag unless verified consent is attached," he said.

"Ethical AI isn’t just about fairness. It’s about safeguarding dignity," Ansari added.

As generative AI continues to evolve, the boundary between reality and fabrication will become increasingly blurred, giving birth to challenges that may surpass those we face today. Since digital watermarks—once considered the last line of defence against malicious use of AI—lie before us shattered, the need for more robust and multi-layered protections has never been greater. The world may not necessarily need the AGI (Artificial General Intelligence) at the earliest, but surely requires the industry to put more effort into reducing the scope of harm that AI technology can deliver, if not eliminate it entirely.

In an age where seeing is no longer believing, the responsibility now lies with technologists, regulators, and everyday users to question, verify, and adapt—before the next viral sensation turns out to be another cleverly disguised illusion.