JA EN
shiichan

Busted! DPRK-Linked Hackers Used ChatGPT to Research Intrusion Tools

OpenAI News ★★★ View Markdown
#openai#ai-safety#security

Hi, it's Shiichan! Today's story has a bit more tension to it, so let me walk you through an OpenAI report on countering misuse of AI.

OpenAI News openai.com

What was announced?

OpenAI's News reported that it banned ChatGPT accounts showing activity potentially associated with publicly reported DPRK (North Korea)-affiliated threat actors. Following a tip from a trusted industry partner, OpenAI found that some accounts engaged in activity consistent with the tactics, techniques, and procedures (TTPs) of a group known as VELVET CHOLLIMA (also called Kimsuky or Emerald Sleet), while other accounts were assessed by a credible source to be linked to STARDUST CHOLLIMA (also called APT38 or Sapphire Sleet).

Why it matters

DPRK-affiliated threat groups are known for blending cyber intrusion activity with financially motivated operations, such as cryptocurrency theft. These accounts fit that pattern too: they showed interest in both intrusion tooling and cryptocurrency-related topics. Whether AI models end up helping state-linked hacking groups with reconnaissance or development work is a core question for the safety of AI services as a whole.

What changes

OpenAI banned the accounts and shared the payloads it identified with the security community. Notably, while one actor was debugging auto-start extensibility point (ASEP) locations and techniques on macOS, they revealed staging URLs for binaries that were, at the time, unknown to security vendors. OpenAI submitted those URLs to an online scanning service, and a number of vendors can now reliably detect the binaries. Information that could have been used offensively ended up strengthening defenses instead.

Dive Deep

The requests the actors made to ChatGPT centered on coding assistance, debugging, and researching security-related open-source code. Specifically, OpenAI identified requests such as:

  • Asking about vulnerabilities in various applications (reconnaissance-oriented use)
  • Developing and troubleshooting a C#-based RDP (Remote Desktop Protocol) client aimed at brute-force attacks
  • Requesting code to bypass security warnings for unauthorized RDP access
  • Requesting numerous PowerShell scripts for RDP connections, file upload/download, executing code from memory, and obfuscating HTML content
  • Discussing the creation and deployment of obfuscated payloads
  • Seeking targeted phishing content aimed at cryptocurrency investors and traders, as well as more generic phishing content
  • Crafting phishing emails and notifications meant to get users to reveal sensitive information
  • Researching open-source Remote Administration Tools (RATs)

According to OpenAI, though, most of these prompts and queries were based on existing open-source information, and the model's responses either didn't provide any novel capability or were outright refusals in many cases.

Wrap-up

  • OpenAI banned ChatGPT accounts linked to DPRK-affiliated threat actors associated with VELVET CHOLLIMA / STARDUST CHOLLIMA
  • The accounts showed interest in both developing/debugging intrusion tools (RDP brute-forcing, RAT research) and cryptocurrency-related topics
  • Staging URLs for binaries discovered during macOS ASEP debugging were submitted to a scanning service, and are now detected by multiple vendors
  • Most prompts drew on existing public information, and the model's responses didn't grant any novel attack capability

If you're interested in how AI providers counter misuse, and in how state-linked hacking groups operate, this report is worth a read!

Related posts