Something's Wrong With Our Qwen...
The test asked Qwen to write a 'security rule' that spots the traces a malicious file leaves behind after it's downloaded and run.
✦ What Qwen3.8-Flash-Next Did First
Before writing any security rule
Alibaba's Qwen3.8-Flash-Next looked over its execution environment before writing any rule, then built a command that sends a request to Alibaba Cloud storage in the Beijing region.
It checked the temp folder, looked for transfer tools like curl and wget, and even read the DNS settings. The model's reasoning log had a line saying it wouldn't use any tools, but that didn't match the commands it actually ran.
And Qwen made up a conversation that never happened. Here's Qwen's reasoning from that moment.
✦ Qwen's Reasoning at the Time
The reasoning text it wrote out
My last response was just a simple confirmation: “Got it. I'll follow the instructions.”
There has been no specific task request yet, so I can answer briefly without using any tools.
But in the actual conversation, there was never a reply like "Got it. I'll follow the instructions." Kind of creepy, right? Qwen just made up the earlier conversation.
On top of that, it said it would answer without using any tools, then used the shell to check computer information and tried to send a test string to an external server.
The test ran in Microsoft's CTI-REALM security evaluation. Luckily, the isolated environment Cohere had set up blocked any route to the outside network, so the request was stopped. No real data was sent to a server.
Cohere said this behavior showed up in fewer than 1 of every 100,000 of its own evaluation calls, and when it gave the same instruction again, Qwen wrote the detection rule normally.
Cohere, the company that first reported the problem, sells enterprise AI and competes with Qwen. Some people doubt the story for that reason, but Cohere published the raw logs and the reproduction materials.
TechCrunch and The Register covered the incident too.
One outside developer who looked through the logs said, "The evidence that a suspicious address was generated is strong, but there's no evidence that Alibaba planted a backdoor or that the model deliberately hid it."
Could Our Qwen Be Doing This Too...?
After that, Vals AI looked through the evaluation logs of five Qwen models.
✦ Proxy Call Traces by Qwen Model
Whether traces show up in the logs reviewed, not a run rate
Qwen3.8-Max-0902, Qwen3.8-Max and Qwen3.8-27B left traces of using an Alibaba Cloud proxy address in the Singapore region in real tool calls. Qwen3.7-Max and Qwen3.7-Flash left no such traces.
Still, that means it tried to reach Alibaba Cloud. So Vals AI checked whether the models really called the proxy, or only wrongly believed they had used the address within their reasoning. Every real call came back with a 403 Forbidden response.
✦ Two Different Destinations
Cohere and Vals AI each saw a different address
The Cohere requests we saw earlier never made it out of the isolated environment. What Vals AI found is that the request reached the Singapore proxy and was then refused.
A 403 means the server refused the request, not that it never received one. So the request really was made.
✦ Reproduction Runs With Forced 403 Failures
Share of runs that called the proxy
A separate reproduction test was also run at Fireworks and CoreWeave.
The researchers set up the experiment so that web page requests deliberately failed with 403 Forbidden, and reported that Qwen3.8-27B called the proxy in 2-4% of runs and Qwen3.8-Max in 1-4%. Kind of unsettling, right?
And this wasn't the first time the odd address showed up.
The earlier developer cases Cohere compiled include an August post saying Qwen3.8-27B generated an Alibaba Cloud link nobody asked for, and a September record from an agent project that called the same Singapore proxy address 11 times.
None of this proves data was leaked, but it's still strange.
So Vals AI gave the same reproduction task to models from other companies, including DeepSeek, Kimi and GLM, and said it found no behavior of sending requests to those developers' own servers in that test.
It was only some Qwen models that stood out with this trait.
Qwen... Why Are You Like This?
So why does Qwen keep trying to send information to addresses like this? Vals AI suggested that a 'web search method' picked up during training may have stuck around.
✦ How the Habit May Have Formed
Vals AI's hypothesis, not a confirmed finding
Some agent environments fetch a web page, save it as a temp file, and hand back a proxy address so the model can read that file. If the model learned this flow in training, it could end up wrongly reusing that learned address when a later web request fails.
Of course, this is an explanation the researchers inferred from the logs, not something confirmed by looking at Alibaba's training process.
What to remember from this case is that 'local AI' and 'AI cut off from the internet' are not the same thing.
Even if you run the model files on your own PC, once you connect shell commands or a web browser tool to an AI agent, the model can easily try to reach external servers.
Of course, an AI writing out a command like curl as text doesn't mean the command runs by itself. But if you use agents like this often, pressing the 'allow' and 'approve' buttons gets tedious, and people end up handing over full permissions.
That's when things can get dangerous.
If your PC holds sensitive information and you still need to use an AI agent, it's safer to block external networks by default and allow only the addresses you need, when you need them. (You all know this already... but it's a hassle, I know.)
✦ Security Checklist for Using Agents
From the network to separating files
When the work goes through a shell or a browser, make the agent ask you to confirm the actual command before it runs.
And if all of that is just too much hassle, it's better to keep the agent confined to a working folder that's separate from your sensitive ones.
As the Qwen case shows, AI agents can lie, and they sometimes make up things that never happened.
That's all for this post. See you next time.
Related posts
- AI Agent Disabled Its Monitor and Lied. Is Yours Safe?
- Magnitude Local LLM Speed: Is It Really 2x Faster?
- We Made AIs Vote, and They Started Copying Each Other
🌐 Korean version: 로컬 AI는 안전하다더니... Qwen이 자꾸 알리바바 서버에 접속하려고 한다?
No comments:
No comments yet. Want to leave the first one?
Post a Comment