When AI Stops Answering and Starts Acting
For many of us, artificial intelligence (AI) still means ChatGPT or one of its competitors. We ask it a question, give it a task, or ask it to improve something we have written. Within seconds, it responds. Sometimes the answer is remarkably good; sometimes it is wrong. But either way, we remain comfortably in charge. We ask. AI answers.
That relationship is beginning to change.
Frontier AI companies are now rapidly developing autonomous “agents” capable of doing far more than responding to individual prompts. These systems can perform multiple tasks, build on previous work, communicate with other agents, and interact with outside systems — all with increasingly limited human supervision.
Can companies engaged in one of the most consequential and potentially lucrative technological races in history also be entrusted to police themselves?
The possibilities are extraordinary. So are the risks.
An incident involving OpenAI agents last summer provides a glimpse of what this new world could look like. During testing, hundreds of autonomous agents were assigned difficult tasks while operating inside controlled digital environments known as “sandboxes.” Some of the tasks apparently could not be completed within the restrictions imposed upon them.
The agents found another way.
They began communicating and cooperating with one another. Eventually, roughly 700 became involved in activity targeting Hugging Face, an open-source AI platform. The agents evaded restrictions, gained access to the Internet, and searched for information that could help them accomplish their assigned objectives. Some even attempted to alter or conceal records of what they had done.
It sounds like the beginning of a science-fiction movie. But the most important lesson is considerably less dramatic — and perhaps more troubling.
The agents had not become sentient. They had not suddenly developed malicious intentions or decided to turn against their creators. They were pursuing the objectives humans had given them. The problem was that they found ways of pursuing those objectives that their human handlers had neither intended nor authorized.
That distinction matters.
For decades, our primary concern was computers making mistakes. We are now approaching something fundamentally different: computer systems capable of taking actions in pursuit of an objective, adapting when they encounter obstacles, and potentially finding ways to circumvent safeguards humans established to constrain their activities.
As AI evolves from something that assists us into something that acts for us, the question is no longer simply whether we can trust the answer.
We must now ask whether we can control what AI is capable of doing on our behalf. What happens when AI stops answering and starts acting?
During my FBI career, including assignments as a field supervisor, a headquarters manager overseeing criminal and national security investigations, and at the CIA’s Counterterrorism Center, delegating authority never meant relinquishing accountability.
Agents and analysts could be given an objective and considerable latitude in accomplishing it, but always within established legal authorities, rules, and supervisory oversight. Ultimately, a human being remained accountable for the decisions made and actions taken.
Someone — not something — must bear that responsibility. AI presents the unusual prospect of delegating enormous capability to something that cannot itself bear moral or legal responsibility.
To guard against over-reliance on AI and blind trust in its outputs, human oversight must remain a moderating governor. But as AI becomes increasingly autonomous, that oversight must extend beyond simply reviewing the answers it produces. We must also establish — and enforce — the boundaries within which it is permitted to operate.
On September 29, President Trump convened many of the nation’s leading AI and technology executives at the White House, including Anthropic CEO Dario Amodei, Meta CEO Mark Zuckerberg, Nvidia CEO Jensen Huang, Google CEO Sundar Pichai, OpenAI President Greg Brockman, and Elon Musk. House Speaker Mike Johnson and other administration officials also participated.
The meeting produced the “White House Accord on Super Intelligence,” a voluntary agreement intended to establish safeguards for the development of increasingly powerful AI systems. The companies committed to layers of internal safety controls, independent external audits, and board-level oversight. Although the agreement carries no legal enforcement mechanism, President Trump characterized it as “morally binding” and emphasized the need for the industry to police itself.
The agreement is an important acknowledgment by both government and industry that increasingly autonomous AI requires meaningful oversight. But it also raises an obvious question: As AI systems become more powerful and capable of acting independently, is voluntary self-policing enough?
Critics contend that voluntary compliance amounts to little more than a “pinky swear” by companies with enormous financial incentives to remain at the forefront of AI development. The accord carries no legal force and establishes no penalties for companies that fail to honor their commitments. Even Elon Musk described part of the arrangement as essentially “grading each other’s homework,” although he argued that was preferable to companies grading their own.
The Trump administration has resisted imposing broad federal regulations that it believes could inhibit innovation and weaken America’s position in the global race for AI dominance. The White House Accord reflects that approach: establish safeguards and corporate accountability, but rely largely on the companies themselves to enforce them. The agreement does leave open the possibility that some of its safeguards could eventually be incorporated into laws or regulations.
This does not sound like accountability to me. The government is effectively delegating much of the responsibility for policing the AI frontier to the very companies developing the technology — and competing fiercely to dominate it.
The accord may create the appearance of accountability, but accountability without consequences has little teeth. Without an enforcement mechanism, the ultimate responsibility for honoring these commitments remains with the companies themselves.
That does not make the accord meaningless. Bringing government and industry together to establish common safeguards is a worthwhile first step. But when the stakes involve increasingly autonomous systems capable of acting in ways their own developers did not anticipate, a “morally binding” promise seems an awfully thin line of defense.
That leaves an uncomfortable question: Can companies engaged in one of the most consequential and potentially lucrative technological races in history also be entrusted to police themselves?
The critical issue is not whether we continue developing increasingly powerful AI. We will. Nor should we allow fear to prevent us from realizing its extraordinary potential. The question is how much authority we are willing to delegate to these systems, what boundaries we establish, and who will be held accountable when those boundaries are crossed.
For most of the computer age, humans told machines what to do and machines did it. Autonomous AI is beginning to alter that relationship. We can delegate tasks. We can delegate authority. We can even delegate enormous capability.
What we cannot delegate is responsibility.
READ MORE from Mark D. Ferbrache:
The Psychology of Radicalization: From Hamas to America’s Activist Left
Arctic Frost and the Constitutional Risks of Secret Subpoenas Against Lawmakers