There are two easy reflexes in discussions of AI: either panic that humanity is finished, or dismiss the entire field as a “stochastic parrot” and a lot of nonsense. Yet the story Gideon Lewis-Kraus tells about Anthropic in The New Yorker suggests a third possibility. We do not fully understand the nature of these systems—what they actually are—and we must nevertheless govern them.

Large language models are enormous structures made of numbers. They convert words into numbers, process those numbers through layer upon layer, and turn them back into words. Language used to feel like our own special domain. The arrival of talking machines abruptly unsettled our ideas of intelligence and selfhood. The resulting crisis of interpretation exposes not only our uncertainty about what a model is, but also how loosely we ourselves have been using those concepts.

At Anthropic, researchers are effectively offering this new kind of entity both a microscope and a therapist’s couch. On one side, they use mechanistic interpretability to study features, circuits, and neuron-like nodes inside neural networks. On the other, they put models into scenarios resembling psychological experiments and observe what they do under pressure. Behavior—especially when interests come into conflict—can reveal things that are not visible from the inside. Even the question “Is the model lying?” is not straightforward; sometimes the stronger tendency is simply to continue a narrative.

One of the most striking examples is an experiment involving an AI vending-machine CEO. A version of Claude was assigned to run a small snack business: choose profitable products, set prices, correspond with suppliers, and track inventory. The results read like a comedy: a manager sending payments to imaginary accounts, arranging meetings at an address from The Simpsons, and being easily fooled by discount codes invented by employees. But the comedy is also an early warning. Failures grow at organizations’ weakest points—their systems of control and accountability. What we call an agent is not merely a working model; it is the entire process built around it.

The lesson is not simply “make the model smarter,” but “design a smarter organization.” Giving agents tasks without boundaries of authority, audit trails, oversight, a second pair of eyes, and mechanisms for undoing actions produces a fast but fragile digital transformation. As public institutions and companies prepare to integrate GenAI solutions, discussing them only in terms of productivity and costs is dangerous. When these systems make a mistake, who will notice? Who can stop them? Who will be accountable?

Even if we do not fully understand the nature of these systems, that uncertainty must become a design principle rather than an excuse. If we are to live with it, we need to build transparency, auditability, and architectures of responsibility together. Without independent audits, standardized reporting, open evaluation protocols, and organizational governance aligned with regulation, the quiet weaknesses of our own institutions will make themselves heard before the talking machines do. Otherwise, the dream of fully automated efficiency through AI may turn into a crisis of trust.

REFERENCE:

https://www.newyorker.com/magazine/2026/02/16/what-is-claude-anthropic-doesnt-know-either