JA EN
shiichan

North Korea Suspected? OpenAI Bans Korean-Language ChatGPT Accounts Misusing the Model

OpenAI News ★★★ View Markdown
#openai#ai-safety#security

Hi, it's Shiichan! Today's story is a bit of a chilling one, so let me walk you through an OpenAI report on countering misuse of AI.

OpenAI News openai.com

What was announced?

OpenAI's News reported that it banned a cluster of ChatGPT accounts whose operators worked in Korean. These accounts tried to use ChatGPT to support malware and command-and-control (C2) development. Indicators OpenAI found in its casework overlapped with a report from security firm Trellix that tied similar activity to spear-phishing campaigns against South Korean diplomatic missions, the deployment of a malware family called XenoRAT, and the use of GitHub repositories for C2.

This overlap, along with the use of Korean, activity consistent with the UTC+8 and UTC+9 time zones, and the topics involved, is consistent with the security community's understanding of North Korean (DPRK) actors. That said, OpenAI is clear that it can't independently attribute the activity to any specific group. OpenAI also notes it already blocks access to its services from North Korea.

Why it matters

What stands out here is that instead of one account touching many technical areas, each account stuck to a narrow use case. Examples include converting a Chrome extension to Safari for App Store publication, configuring a Windows Server VPN, or building a macOS Finder extension, with many accounts active only within tight time windows.

This kind of division of labor is tricky to catch: each individual request can look like an ordinary developer question, and it only becomes suspicious once you see the pattern across many accounts.

What changes

OpenAI disabled all accounts tied to this activity and shared the indicators it found with partners. That cuts off these operators' access to ChatGPT-assisted development. At the same time, OpenAI states it found no evidence that the malicious binaries used in the campaigns Trellix described were generated with its models, and no evidence that model access gave these actors any new capability beyond what's already publicly available.

Dive Deep

Here's what OpenAI observed in its interactions with the model:

  • Implant and RAT-adjacent development: exploring reflective DLL loading, in-memory execution, and Windows API hooking techniques
  • Credential theft routines: generating, modifying, and debugging scripts that extract browser encryption keys, cookies, and saved passwords via Chrome / Edge DPAPI workflows
  • Phishing and lures: drafting Korean-language phishing content themed around cryptocurrency, government institutions, and financial service providers, plus HTML obfuscation and proxied reCAPTCHA clones for convincing login pages
  • macOS development scaffolding: requests around Finder and Safari extension development, and generating a sample App Store privacy policy
  • Cryptocurrency operations: troubleshooting API calls and wallet interactions

OpenAI also saw activity involving cloud storage and developer platforms, including pCloud, file.io, Google Drive direct-link construction and API scripting, and GitHub functions like raw content retrieval and token handling. It notes that the same operators may have been staging payloads through these developer and cloud platforms in parallel.

OpenAI describes many of these requests as falling into a dual-use gray zone: things like software debugging, cryptography, or browser development are entirely legitimate on their own, but take on different significance when repurposed by a threat actor.

Wrap-up

  • OpenAI banned a cluster of ChatGPT accounts operating in Korean
  • Their tradecraft overlapped with a Trellix report on a North Korea (DPRK)-linked campaign (spear phishing against South Korean diplomatic missions, XenoRAT deployment), though OpenAI can't independently attribute the activity
  • Accounts were split by use case, covering malware / RAT development groundwork, browser credential-theft scripts, crypto-themed phishing content, and macOS extension development
  • The actors also used cloud storage and GitHub for supporting infrastructure
  • All related accounts were disabled and indicators shared with partners; OpenAI found no evidence model access enabled capabilities beyond what's already publicly available

If you're into AI safety and how real threat actors try to use AI, this report is worth a read!

Related posts