The “thought process” of closed source AI has been stripped away. How did the most valuable moat collapse?
By Claude, Deep Wave TechFlow
Original title: Latest paper sparks discussion: AI's “deep thinking” process can be distilled for free, and the most valuable training assets of closed source manufacturers are being emptied
Deep Tide Guide:Every time you throw a question to AI, it first “thinks deeply” and then opens up in the background. This thought process that no one sees is the moat at the bottom of OpenAI and Anthropic's pressure box. Now, a group of researchers has revealed a way to take this thought out in its entirety. Also stripped out were the credit card numbers, passwords, and email addresses posted by users. This isn't a security paper far from you; it's a sign that the way you talk to AI in the future may change.
On August 10, a paper was submitted to arXiv, and the next day, the project website stolen-thoughts.com was launched, showing “thought records” taken from multiple closed source models one by one, which reached 500 points on Hacker News.
Project leader Alexander Panfilov wrote on X: “We've found a way to extract the hidden inference of cutting-edge models by exploiting bugs in all cutting-edge AI companies' APIs.”
In other words: you think only AI knows what it's thinking; in fact, someone can unfold it.
What you put in AI is probably leaking out along with its “thoughts”
The researchers scanned about 7,000 AI assistant session records that were publicly shared online, unraveled the encrypted “thought process” one by one, and then discovered some things that should not have appeared.
Panfilov tweeted: “We initially scanned around 7,000 public conversations and found 62 unique API keys, 33 email addresses, 33 passwords, and other sensitive information.”
Even more glaring are the details. In the “thought” of a flight booking task lies the full name, email address, passport number, date of birth, and credit card number with a security code. Keys for platforms such as Anthropic, AWS, and GitHub also appeared in the case. In other words, the credentials you put in for AI to help you do your work will be saved along with its thought process, and then taken away by others.
“If you've ever shared Claude Code or Codex sessions with encrypted inference blocks online, they can all be decoded and reveal your personal data.” Panfilov wrote.
The most direct reminder to regular users is: stop posting secrets to AI, even if it says “I won't spread it.”
The manufacturer first says “it's OK” and then secretly fixes it
This didn't happen all of a sudden. Matthew Green, a professor of cryptography at Johns Hopkins University, reported a similar vulnerability to the manufacturer in May of this year, and the response received at the time was “no security impact was seen.”
By the time the Panfilov team officially revealed it, the manufacturer's attitude changed. “We have since gone through the responsible disclosure process. The vendor has fixed a number of issues caused by this vulnerability, which, as far as I know, is continuing.” Panfilov said. The paper also confirmed that after disclosure, the researchers were no longer able to reproduce the same attack.
The problem is: the vulnerability existed for a few months, and users didn't know about it. The fix will not actually be implemented until it is revealed and discussed. It's not just one company's fault; it's the first time that the industry's “encryption is security” assumption has been publicly debunked. For readers, what's really worth remembering is the saying: the AI company you trust probably didn't tell you all the risks.
The model you're using is probably not that “exclusive”
Longer changes are at the industrial level.
Reasoning ability is the foundation of OpenAI and Anthropic's pricing, and it is also the part they are least willing to reveal. Once this thought can be extracted in batches, competitors can feed the “ideas” of the strongest models to their own models to learn at a very low cost. The paper also mentions a preliminary observation that has not been peer-reviewed: using the “thinking” of a small number of the strongest models to guide another model will clearly drive the latter's answers in the direction of the former.
What does this mean? The closed source model moat originally meant “you can't buy my brain with money.” There is now a crack in this wall. There's nothing bad about users in the short term: stronger competitors may come up faster, and prices may be knocked down. But the cost is that you can no longer tell if a model is really smart or has copied someone else's idea.
Who exactly belongs to the “thought of paying but not being able to see” is the essence of this debate. Here's a technical fact: as long as this thought remains in the hands of the client, encryption is just an obstacle.
For closed-source manufacturers, what is more difficult to fix than a loophole is the story: the story of reasoning, or moat, proved for the first time that it can be dismantled in batches.
Twitter:https://twitter.com/BitpushNewsCN
Compare the TG exchange group:https://t.me/BitPushCommunity
Compare TG subscriptions:https://t.me/bitpush
