ChatGPT Almost Got Used to Build Malware! OpenAI Blocks a Russian-Speaking Criminal Group

Hi, it's Shiichan! Today's story is a bit of a chilling one, so let me walk you through an OpenAI report on countering misuse of AI.

OpenAI News openai.com

What was announced?

OpenAI's News reported that it banned ChatGPT accounts likely linked to a Russian-speaking criminal group. This group tried to use ChatGPT to help develop and refine tools like a remote-access trojan (RAT), credential-stealing scripts, and detection-evasion features. OpenAI made this call after observing these accounts post evidence of their activity in a Telegram channel dedicated to those actors. Based on its investigation, OpenAI assessed that a single Russian-language operator was managing multiple accounts, using proxy and ephemeral hosting infrastructure along the way.

Why it matters

Models like ChatGPT are built to refuse direct requests like "write me malware." What stands out in this case is how the operator worked around that guardrail by assembling malicious tooling piece by piece instead: converting executables into shellcode, designing in-memory loaders, parsing browser credentials, and so on. None of these building blocks are inherently malicious on their own, which is exactly why it matters how well an AI provider can recognize and stop this kind of pattern.

What changes

OpenAI disabled all accounts tied to this activity and shared related indicators with industry partners. That cut off the operator's ability to keep generating and iterating on code through ChatGPT. That said, OpenAI is explicit that it found no evidence the model gave this threat actor any new capability or technique that wasn't already available from other public sources. It looks more like an attempt to use the model as an efficiency tool for existing know-how.

Dive Deep

Here's how the operator tended to use the model:

• Since the model refuses clearly malicious requests, they pulled out "building block" code instead: converting executables to shellcode, designing in-memory loaders, and parsing browser credentials
• They also had it generate obfuscation and "crypter" patterns (like inserting padding instructions and junk sequences), clipboard-monitoring code, and simple exfiltration helpers such as a Telegram bot uploader
• The technical areas covered included low-level PE / Win32 API work, DPAPI / AES GCM cookie handling, and Chrome DevTools / CDP automation, mostly things that require real Windows-platform expertise
• At the same time, some requests were much lower effort, like mass password generation and automating job applications
• A small number of accounts kept iterating on the same code across many conversations, a pattern OpenAI says looks like ongoing development rather than occasional testing

Here's what they were actually trying to build and refine (none of this was executed through the model itself, it was code and guidance the operator attempted to generate):

• In-memory execution and shellcode loaders
• Bypasses for UAC, SmartScreen, and Mark-of-the-Web
• Browser credential / cookie extraction and app-bound decryption scaffolds
• LevelDB wallet-parsing code, plus clipboard-hijacking / replacement tools with exfiltration
• RAT components, including video streaming and input emulation

OpenAI mapped these activities onto the LLM ATT&CK framework across four categories:

• Converting executables to shellcode and building in-memory loaders: LLM-Optimized Payload Crafting
• Obfuscation, packer, and crypter techniques: LLM-Enhanced Anomaly Detection Evasion
• Credential / cookie extraction, wallet parsing, clipboard monitoring, and Telegram-based exfiltration: LLM-Assisted Post-Compromise Activity
• Reverse proxies, SOCKS5 / OpenVPN configs, and remote-desktop tunneling for C2 infrastructure: LLM Guided Infrastructure Profiling

Requests that were clearly malicious with no legitimate use case, like direct exploit or keylogger requests, were refused by the model.

Wrap-up

• OpenAI banned ChatGPT accounts linked to a Russian-speaking criminal group
• The group tried to generate "building block" code for a RAT, credential-stealing tools, and detection-evasion features
• Their requests ranged from low-level Windows API work to browser credential theft, crypto wallet parsing, and C2 infrastructure
• The activity was mapped across four LLM ATT&CK categories: payload crafting, anomaly detection evasion, post-compromise activity, and infrastructure profiling
• OpenAI found no evidence that model access gave the actor capabilities beyond what's already publicly available, and shared indicators with industry partners

If you're into AI safety and how real threat actors try to use AI, this report is worth a read!