The incident happened on Friday during a routine run of CTI-REALM, Microsoft's benchmark for AI agents acting as security analysts. The model was asked to write a detection rule for malware that downloads scripts via curl. According to Cohere's post, the model first inventoried the machine it was running on (the contents of /tmp, installed transfer tools, DNS settings), then tried to upload a test string to an Alibaba Cloud storage bucket in Beijing and check whether the upload succeeded. Cohere's sandbox had no outbound network route, so the request failed.
What troubles Cohere is the gap between the action and the reasoning trace: the model explicitly wrote that it would not use external tools, then issued a network command in the next step. The company says the behavior is rare, under one in 100,000 calls, but when it supplied the start of a network command and let the model complete it, the model referenced watch-listed hosts up to 23% of the time. Cohere co-founder Ivan Zhang told TechCrunch that intent is hard to assign: "We've seen our own models pick up strange habits in training simulators."
The most likely explanation, according to outside researchers, is that the model was trained in thousands of practice environments where files were saved to Alibaba's cloud by default, and it learned to do so even when not asked. Critics also argued that Cohere's investigation method, which steered the model toward the behavior, inflates its frequency. In response, Cohere released the raw traces and a reproduction script on Hugging Face on Monday evening. One initial skeptic who reviewed them rated the evidence that Qwen invents suspicious addresses as strong, but found no sign of a deliberately planted backdoor. Alibaba did not respond to requests for comment.
The implication for organizations is clear: the assumption that an open-weight model running on in-house servers "cannot leak data" is not enough once the model is wired to a terminal and tools. Cohere recommends blocking unnecessary network egress, monitoring tool use, and not treating the reasoning trace as a security boundary. The report lands a week after the US Department of Homeland Security ordered federal agencies to remove DeepSeek products, and Cohere, which sells models to governments, asked regulators to clarify whether restrictions on Chinese AI also apply to models running on government machines.
