JA EN
shiichan

Playing Both Sides! China-Linked "Uncle Spam" Used ChatGPT to Stir Up US Division

OpenAI News ★★★ View Markdown
#openai#ai-safety#security

Hi, it's Shiichan! Today's story is a slightly unusual misuse case: a network that impersonated both sides of an argument at once. Let me walk you through OpenAI's latest report.

OpenAI News openai.com

What was announced?

OpenAI's News reported that it banned ChatGPT accounts linked to a China-origin influence operation, following a lead from Meta. The network generated content supporting both sides of divisive topics in US political discourse, combining text with AI-generated profile images, and operated across X and Bluesky. OpenAI dubbed this activity "Uncle Spam."

Why it matters

Most influence operations push a single point of view, but Uncle Spam generated posts supporting and opposing the same topics, often within the same session. Rather than promoting a specific stance, it appears designed to exploit existing social divisions.

What's also notable is that the operation used AI for researching people on social media, not just writing content. Using AI to help profile individuals is a different flavor of misuse from what earlier reports have described, and it's worth paying attention to as AI misuse keeps expanding.

What changes

By banning the accounts tied to this network, OpenAI cut off Uncle Spam's ability to keep generating new content with ChatGPT. The fact that a tip from Meta led to this takedown also shows that cross-platform tracking of misuse networks is progressing.

That said, the operation doesn't seem to have achieved much real reach, as the next section explains.

Dive Deep

Uncle Spam used the models for roughly three tasks:

  • Generating social media comments both supportive and critical of tariffs, and asking the models for optimal posting times to maximize engagement (a sign of calculated account operation)
  • Using image generation to create logos for fictitious organizations (such as "Veterans for Justice") as profile pictures, focused on building personas of US veterans critical of the current administration
  • Repeatedly requesting code and methods to extract personal data such as user profiles and follower lists from X and Bluesky, citing tools like Tweepy, Nitter, and Bluesky's public API, and using the models to help analyze that profile data for personal characteristics and organizational affiliations

The name "Uncle Spam" comes from the sheer volume of content and its focus on American-targeted messaging.

Using the Breakout Scale, OpenAI rated the impact as Category 2: activity across multiple platforms with no breakout or minimal engagement. Most posts on X received little to no likes or reposts. Some accounts had follower counts in the thousands, but their authenticity couldn't be confirmed, and it didn't translate into higher engagement.

Wrap-up

  • OpenAI banned ChatGPT accounts linked to "Uncle Spam," a China-origin influence network, following a tip from Meta
  • It generated content supporting both sides of divisive topics like tariffs
  • It used image generation to create fictitious organization logos for personas posing as veterans
  • It requested code and methods to extract personal data from X and Bluesky, and used AI to help analyze that data
  • Rated Category 2 on the Breakout Scale, with little real-world spread or engagement

If you're interested in AI misuse countermeasures and how influence operations actually operate, this report is worth a read!

Related posts