Weaponized Deepfakes Raise Alarming Risks for Society and Democracy

Artificial intelligence-generated deepfakes—videos, images, or audio that impersonate real individuals—have become an urgent concern. While experts have predicted for years that these AI forgeries would be misused, improvements in generative models and widespread, affordable access to such tools have made faking reality easier and more convincing than ever.

Deepfakes no longer consist only of the so-called "AI slop" that is easy to identify. Weaponized versions can resemble authentic media, fueling disinformation, targeting individuals, and undermining trust. Examples from around the world have shown deepfakes inciting violence, spreading political propaganda, and generally eroding public confidence in institutions and each other.

A particularly troubling area involves the use of deepfakes in producing sexually explicit and non-consensual imagery. Recent research found that as of 2023, 98% of deepfakes were pornographic in nature—and 99% of those depicted women—disproportionately impacting women and marginalized groups. For example, with the launch of the “edit image” function in the Grok AI chatbot, users reportedly created millions of sexualized deepfake images, mostly of women and even children. xAI, the company behind Grok, initially limited the feature to paying customers following backlash, and has now blocked nudity features where it is illegal.

Political deepfakes have also seen a surge. In the United States, members of the Trump administration have shared AI-generated images and videos, some of which are designed to either sway opinion or embarrass political rivals. In January, Texas attorney general Ken Paxton posted an AI-generated video showing his opponent, Senator John Cornyn, seemingly dancing with a political rival—an event that never took place. Such content was shared without making clear that it was artificially generated.

Proposed solutions to address deepfakes include adding technical safeguards and detection mechanisms to AI models, encouraging users to be more cautious online, and enacting new or applying existing legislation such as copyright law. However, each approach has limitations. Technical solutions can be sidestepped by using open-source models without safeguards. Changing user behavior at scale has proven unrealistic. Regulatory measures require enforcement, which is often lacking or inconsistent. Notably, while deepfake pornography has been criminalized in some jurisdictions, other forms of harmful AI-generated media continue to circulate, even from official sources. In one instance, the White House posted an altered image of a Minneapolis civil rights lawyer, darkening her skin and exaggerating her expression.

Looking ahead, the risks associated with deepfakes are expected to intensify—particularly with major elections approaching and fact-checking organizations facing challenges. Without stronger and more coordinated responses, the proliferation of weaponized deepfakes could further erode societal trust and weaken democracy.

Source: technologyreview.com