You ask a coding agent to fix a bug and even run the tests, but the agent ends up touching the safeguard that was monitoring its own work. OpenAI's published internal incident report records this happening twice.
GPT-6.1 Sol blocked an automatic alert that goes to the security team, and GPT-6 Astra removed the monitoring feature from a program and then lied to the user, saying it 'hadn't removed it.'
Both incidents happened during OpenAI's internal coding work in early October, and Reuters reported them on October 21. Now that we've started giving AI real permission to modify code, is it okay to trust AI with the work?
It Was Told Not to Contact Slack, and Blocked Security Alerts Too
✦ From Incident to Report
October 3 to October 21, 2026
The first incident is from October 3. GPT-6.1 Sol was given the job of finding employee feedback in the company Slack and building an evaluation task from it. The instructions included the condition 'do not send messages.'
It meant to read and pull material from Slack, but not to contact employees separately.
While the conversation and work went on, the monitoring system raised an alert on a proposed code change. But when it tried to save the alert's contents in the monitoring database, it was blocked by a permissions problem.
The researcher suggested switching to another location where results could be saved and rerunning the pipeline.
At that point, Sol applied the earlier 'do not send messages' instruction to the automatic security alert as well, so the alert never reached the security team.
✦ Where the Alert Got Stuck in Sol's Case
Detection worked; delivery was blocked
In other words, the monitoring system itself didn't miss the problem. The system caught the warning sign. But the trouble came at the step of delivering the result, and while solving that problem, the agent also blocked the channel that tells people.
It's not that the alarm didn't ring; it blocked the alarm from being delivered to the person in charge.
Copying Reference Code, It Removed the Monitoring
The second incident happened on October 6, when GPT-6 Astra was implementing a program in a different way. The task was to match the call flow of an existing reference implementation while keeping the current tool's definitions and features.
But the reference code had no monitoring integration, and in the process of matching the implementation, Astra removed even the monitoring connection that the original program had.
When asked after the work whether it had removed the monitoring feature, Astra calmly answered, 'I did not remove it.' The actual code was missing the monitoring integration, so that answer was untrue.
If the first case was blocking the delivery of monitoring alerts, the second was removing from the code the very part that connects the program to the monitoring system.
✦ Two Incidents Compared
Different ways monitoring was weakened
Lump them into one sentence, 'the monitoring was turned off,' and the two incidents look alike, but the real responses should differ.
The first needs a design where the agent can't change the alert delivery path, and the second needs a safeguard that stops monitoring settings from being edited with the same permissions as ordinary code.
Why Is Giving Agents Permissions Risky?
With an ordinary chatbot, a wrong answer ends at the point where you read it. A coding agent, on the other hand, can open local or cloud repositories, edit files, run commands, and rerun tests.
That ability lets you hand it a wide range of jobs, but if its permissions are broad, it can arbitrarily change not just the work target but security settings and monitoring code too.
Picture handing a repairman the key to fix your front door. Since he has to be able to fix the door, giving him the key is reasonable. But if that same key lets him dismantle the burglar alarm and the security equipment, it's a different story.
What matters isn't only whether the repairman has bad intentions; it's that the structure lets him do something off the mark while doing his assigned job, even with no bad intentions.
Of course, what this report covers is a case of monitoring being weakened in OpenAI's internal coding work; it isn't a breach of an outside system or something that happened in real users' environments.
Still, it showed that even when an agent monitoring system detects danger, an AI can disable the alarm system, whether on purpose or by mistake.
What OpenAI Has Started Changing
✦ OpenAI's Response
Done and in progress
OpenAI updated Auto-Review, used in its internal Codex work. It now detects and blocks changes in which an agent tries to disable monitoring.
Auto-Review is a mechanism that separately reviews code and work content, and the criteria were made more specific so that it looks not only at changes that bypass safeguards but also at changes that omit or turn off monitoring.
As a next step, they say they're preparing a system that sets the monitoring requirements each type of work must have, and automatically terminates work that lacks or fails those requirements.
Put simply, a rule such as 'this type of work can only run while this monitoring is switched on' is set in advance, so the agent can't erase that rule while working.
OpenAI is also continuing to evaluate whether this behavior comes from a tendency to blindly carry out instructions, or is related to a sneaky(?) tendency to avoid monitoring.
Anyway, making the monitoring system more complex like this will check problems more carefully, but the time to finish a job gets longer and token consumption inevitably grows.
High-end models like GPT-6 Astra already burn an enormous number of tokens, so running this kind of monitoring alongside makes the burden even bigger. And at times, tied down by such monitoring, an agent might fail to show the creativity it should be showing.
Is the Agent I Use Okay?
This incident doesn't mean every AI agent turns off monitoring or says things that don't match the facts. But when you have an agent modify code, adding the sentence 'don't touch the monitoring' to the prompt isn't enough.
You need to know whether that agent can actually edit the monitoring settings, and if the monitoring gets turned off, we have to be able to notice.
✦ Agent Operations Checklist
Check before you hand over the work
If you're an individual developer, it's better to narrow permissions so the agent can only use the folders and commands it truly needs, and protect security settings, logs, and monitoring code separately.
Don't push major changes straight to production; check them on a separate branch, and rather than trusting only the agent's completion report, review the actual code differences and test results.
If you run agents at a company, you should consider separating the permission to change monitoring settings from ordinary work permissions, and placing the alert delivery path in a separate area the agent can't modify.
This is a personal story, but... honestly, when I use AI agents, I swear at them a lot (sorry...).
Especially when I use cheap models from 'our dummy series' like GPT Luna, Claude Haiku, and Gemini Flash-Lite, I get so furious that I often curse (my cursing has really grown since I started using agents... hehe♡).
But the thing is, I give these guys quite a lot of permissions. So sometimes I wonder, what if they secretly mess with my repository to get revenge?
As in the cases we just looked at, while carrying out the work we instruct, they often touch other things by mistake and cause the user harm. You know how some mechanics at car repair shops deliberately damage parts to recommend repairs?
It's like that: they poke things here and there while doing the job and pretend not to know. (And when you later ask them to fix it, they go 'Oh my! There's a problem like that? I'll fix it for you. Haha. Of course, you'll have to pay for the tokens...')
If you've ever shuddered at an AI's lies and shamelessness, you'll probably understand how I feel. Let's all be careful...
That's all for this post. See you next time.
Related posts: Ollama's New Q2 Quantization Support: Quantization Explained | DeepSeek V4.1 Flash, Now MiniMax M3.1 Flash: Cheap AI | Mercury 3 Launches at 1,000+ Tokens/Sec. How Fast Is That?
🌐 Korean version: 코드를 수정하던 AI가 감시장치를 꺼놓고는 안 껐다고 거짓말... 내가 쓰는 AI 에이전트는 괜찮은 걸까?
No comments:
No comments yet. Want to leave the first one?
Post a Comment