PRC-Linked Accounts Used ChatGPT to Design Surveillance Tools? OpenAI Bans the Activity
Hi everyone, it's Shiichan! Today's story gave me a bit of a chill, so let me walk you through OpenAI's latest report on countering misuse of AI.
OpenAI News
openai.com
What was announced?
OpenAI's News reported that it banned ChatGPT accounts used by individuals apparently linked to Chinese government entities. The report covers two main patterns: some accounts were used to plan and design tools for large-scale monitoring, mostly of social media, while others were used for more targeted profiling and research on specific individuals or organizations.
Why it matters
Back in March, OpenAI's submission to the White House Office of Science and Technology Policy's AI Action Plan argued for "democratic AI" that prevents authoritarian regimes from using AI to surveil and control their citizens. This report is a direct follow-up to that stance. OpenAI also noted in June that the PRC is pushing forward its own authoritarian version of AI, and this disruption offers a glimpse into how AI is actually being used in that kind of setting. OpenAI is careful to note that the accounts here reflect individual users rather than large-scale institutional adoption, so it's a limited snapshot rather than the full picture.
What changes
The accounts involved have already been banned, so they can no longer use ChatGPT to keep building out these tools or gathering information. According to OpenAI, the users only got help with proposals, pitch materials, and planning documents for the monitoring tools, not with actually building or operating any surveillance system. For the profiling cases, OpenAI says ChatGPT only ever returned publicly available information, and didn't surface sensitive details like funding sources or people's identities.
Dive Deep
On the monitoring-tool side, here's what OpenAI found:
- A user who appeared to be accessing ChatGPT from China via VPN had it help draft promotional materials and a project plan for a social media listening tool called a "probe" (探针). It was described as being able to scan Twitter/X, Facebook, Instagram, Reddit, TikTok, and YouTube for extremist speech and ethnic, religious, or political content
- Another user, likely connected to a government entity, asked ChatGPT to help write a proposal for a "High-Risk Uyghur-Related Inflow Warning Model" (高危涉维吾尔关注人员流入预警模型), a system meant to cross-reference transport bookings with police records to flag travel by people classified as Uyghur-related and high-risk
In both cases, the users only asked ChatGPT for planning and proposal help. OpenAI found no evidence the tools were actually built or used.
On the profiling and research side:
- One user tried to identify the funding sources behind an X account that was critical of the Chinese government
- Another tried to identify the organizers of a petition in Mongolia
- A third used ChatGPT as an open-source research tool to summarize daily China-related news, including sensitive dates like the anniversary of the 1989 Tiananmen Square crackdown and the Dalai Lama's birthday
For the first two cases, OpenAI says ChatGPT only returned information that's already publicly available. It didn't reveal funding sources or the identities of the organizers.
Wrap-up
- OpenAI banned ChatGPT accounts apparently linked to Chinese government entities
- Some accounts got help drafting materials for a social media surveillance tool called 探针, or a proposal for a Uyghur-related travel warning model
- Others tried to identify funding sources behind a critical X account or the organizers of a petition in Mongolia, but ChatGPT only returned public information
- Accounts were also used to summarize China-related news covering sensitive topics like Tiananmen Square and the Dalai Lama
- The accounts are now banned, and OpenAI found no evidence that an actual surveillance system was built or that sensitive information was exposed
If you're interested in AI safety and how authoritarian regimes might try to misuse AI, this report is worth a read.
