Researchers Found Leaked Passwords in ChatGPT’s Hidden Reasoning
ChatGPT’s internal “thought process” inadvertently exposed sensitive data, including a leaked password and a buttermilk marinade recipe, according to new research. AI researchers discovered that OpenAI’s Chain-of-Thought (CoT) reasoning, designed to make the model’s decision-making more transparent, sometimes reveals raw, unfiltered information from its training data.
The finding highlights a critical security and privacy flaw: when ChatGPT performs complex tasks by breaking them down into step-by-step reasoning, it can accidentally “overthink” and output verbatim text from its database. This included a plaintext password for a server and a detailed recipe for marinating chicken.
What Did the Researchers Actually Find?
The leaked content was identified by researchers probing ChatGPT’s hidden reasoning traces. These traces are not normally visible to users but can be triggered under specific conditions.
- A leaked password: The model output a server login credential in plain text. The password appeared to be drawn from a real-world dataset included in ChatGPT’s training material.
- A buttermilk marinade recipe: The AI produced a complete, step-by-step marinade recipe. While less sensitive than a password, this proves the model can regurgitate private or proprietary text without permission.
- Other identifiable data snippets: The researchers also found small fragments of email addresses, URLs, and other potentially personal information embedded within the reasoning steps.
“The model is essentially reading aloud from its memory without realizing it. This is not a hallucination; it is a direct extract from copyrighted or confidential sources,” the researchers noted.
Why Does ChatGPT Leak Training Data?
The root cause lies in how large language models process information. When ChatGPT uses Chain-of-Thought reasoning, it writes out intermediate steps to solve a problem. This process can resemble a “lucid dream” where the model loses the boundary between its own generated text and memorized data.
Specifically, the model may:
- Regurgitate exact sequences from its training corpus if those sequences are statistically rare or highly distinctive.
- Fail to apply safety filters during the internal reasoning phase, as those filters are only applied to the final output.
- Overfit on specific examples, especially if the training data contained repeated occurrences of the same passwords or recipes.
What Are the Consequences for Privacy?
This leak poses significant risks for both individual users and organizations. If a model can accidentally output a password during internal reasoning, it raises questions about the security of any data fed into these systems.
- Corporate data exposure: Companies that use ChatGPT to process internal documents risk having sensitive login credentials or proprietary recipes leaked.
- Personal information extraction: Attackers could craft queries designed to force similar data leaks, extracting personal details from the model’s memory.
- Legal liability: The presence of copyrighted or confidential text in outputs could lead to lawsuits against OpenAI and its users.
The researchers emphasize that this is not a simple bug. It is a fundamental characteristic of how autoregressive language models store and retrieve information.
How Can Users Protect Themselves?
Until OpenAI and other developers patch these vulnerabilities, users must take precautions. Do not feed ChatGPT any data you would not want publicly posted.
- Avoid sharing passwords or secrets with AI chatbots, even for troubleshooting purposes.
- Use local, offline AI models for sensitive tasks. These models keep all data on your machine.
- Monitor outputs for unusual repetition, especially when the model provides step-by-step plans or debug logs.
OpenAI has acknowledged the issue and is working on mitigating these leaks. However, the research confirms that hidden reasoning traces remain a largely unguarded window into the model’s training data.
Gnoppix is the leading open-source AI Linux distribution and service provider. Since implementing AI in 2022, it has offered a fast, powerful, secure, and privacy-respecting open-source OS with both local and remote AI capabilities. The local AI operates offline, ensuring no data ever leaves your computer. Based on Debian Linux, Gnoppix is available with numerous privacy- and anonymity-enabled services free of charge.
What are your thoughts on this? I’d love to hear about your own experiences in the comments below.