Indonesia may ban Grok AI and X amid nonconsensual deepfake surge

Indonesia’s communication and digital ministry is investigating the use of Elon Musk’s AI chatbot Grok on social media platform X after reports emerged of the technology generating nonconsensual sexualised images of women and minors.

The ministry’s director general of digital space supervision, Alexander Sabar, said on Wednesday that early inquiries show Grok AI currently lacks clear safeguards to stop the creation and circulation of pornographic material using real photos of Indonesian citizens.

“This risks serious violations of citizens’ privacy and self-image rights,” Sabar added.

Don't miss: X tightens the screws on parody and fan accounts

The scrutiny follows a surge in a viral “remove clothes” trend on X, formerly Twitter, in which users have uploaded photographs of real people - including women and children - and asked Grok to generate images showing them partially or fully undressed. A report by European non-profit AI Forensics found that of 20,000 images generated between 25 December and 1 January, 53% depicted individuals in minimal clothing, 81% of whom were women, while 2% appeared under the age of 18.

The ministry views the manipulation of personal photos not merely as a moral issue, but as a serious infringement on individual control over visual identity, potentially causing psychological, social and reputational harm.

The ministry has warned that non-compliance could result in the outright banning of Grok AI and X in Indonesia. “Every electronic system operator must ensure that the technology they provide does not turn into a platform for privacy violations, sexual exploitation or the degradation of a person’s dignity,” Sabar said.

Authorities have outlined measures to curb the misuse of Grok AI, including stricter content moderation, prevention of immoral deepfakes, and expedited reporting procedures for privacy and image rights violations.

All providers of electronic systems operating in Indonesia are required to comply with national law, with non-compliance potentially resulting in administrative sanctions, including the blocking of access to Grok AI and the X platform.

Under Indonesia’s updated penal code (Law No. 1 of 2023), pornographic content and the manipulation of personal images can lead to criminal and administrative sanctions. Articles 172 and 407 stipulate punishments ranging from six months to ten years in prison, as well as fines. Victims of such violations are encouraged to report cases to law enforcement or file complaints with the ministry.

The controversy has drawn international attention, with Indonesia joining Malaysia, India, and France in calling for investigations into the misuse of AI technologies such as Grok.

X, developed by Musk’s company xAI, has stated it will act against illegal content, including removing offending images, suspending accounts permanently, and collaborating with governments. Musk himself tweeted that “anyone using Grok to make illegal content will suffer the same consequences as if they upload illegal content.”

Indonesia represents a major audience for X, with 22.92 million users, making the government’s response a significant test of AI accountability in Southeast Asia.

Related articles:
Elon Musk: X hit by cyberattacks again
MY govt to verify X's claims following social media license snub
Roblox moves to review age ratings amid ban threat in Indonesia