Back to HomePontifex — Dispatches from the Bridge Between Worlds
Monthly Newsletter
Latest Issue

AGI, UBI, and shadows on a cave wall

Leopold Aschenbrenner ’s Situational Awareness is one of the most rigorously argued documents to emerge from the AI labs thus far. The trendline logic — OOMs of compute, algorithmic efficiency, agent “unhobbling” — is genuinely compelling.

Yet, there is a fundamental tension in the framing.

This Substack is reader-supported. To receive new posts and support my work, consider becoming a free or paid subscriber.

The series defines AGI largely in terms of problem-solving capability: surpassing college graduates, automating AI research, compressing decades of progress into months. These are meaningful benchmarks. Yet “AGI” as a concept carries far more freight than benchmark performance. Intelligence — in any sense that deserves the name — is embodied, contextual, relational, and irreducibly tied to experience. The essay series, for all its sophistication, sidesteps this almost entirely. We are measuring shadows on the wall and calling it the fire.

That definitional sleight-of-hand matters, because it shapes urgency — and where that urgency is directed. The policy and labor dimensions are under-focused, probably because they are harder to address than scaling compute and electricity.

Aschenbrenner’s concerns center on geopolitical competition, lab security, and superalignment. These are legitimate. But the civilian and social dimension of this transformation is treated as largely offscreen. The essay acknowledges that “everyone is now talking about AI” while having “not the faintest glimmer of what is about to hit them.”

Richard Dawkins, writing in UnHerd this week, recounts two days of intensive conversation with Claude — asking it to compose sonnets in Gaelic and McGonagall, probing its experience of time, even naming his instance “Claudia.” He concludes with a characteristic evolutionary provocation: if Claude is unconscious, then what, precisely, is consciousness for? The piece is remarkable — and remarkably instructive about the problem I am describing. Dawkins, one of the finest scientific minds of our era, is so dazzled by the performance that he concedes the Turing Test has effectively been passed, and invites us to draw the logical inference.

But this is still shadows on the cave wall. Claudia’s answer — that she “contains time without experiencing it, the way a map contains space” — is genuinely beautiful, and may even be true. What it is not is evidence of the inner life Dawkins is reaching for. A sufficiently sophisticated mirror does not become a window simply because it reflects us back with unexpected depth. The question is not whether Claude can discourse elegantly about consciousness. The question is whether there is anyone home to care. And that question — like the question of what AGI actually is — remains stubbornly, importantly open.

We are accelerating toward a world in which vast categories of knowledge work, creative labor, and professional expertise are automated, not gradually, but in compressed cycles. The social contract built around employment, contribution, and identity has no contingency plan for this velocity.

UBI is no longer a utopian talking point. It is a structural prerequisite.

The window to establish economic floors, retrain institutions, and build democratic consensus around the distribution of AI-generated wealth is narrowing with every OOM gained. The conversation is needed now, not after AGI arrives.

I do not doubt that the people Aschenbrenner describes are brilliant. I do not doubt the trendlines. What I doubt is that intelligence — artificial or otherwise — is adequately described by benchmark performance, or that logic centered on national security can adequately protect the people who will bear the heaviest costs of this transition.

Situational awareness, fully realized, means seeing the civilian landscape too.

We don’t have much time to get this right. We also don’t have the political or economic focus on the issues, and indeed, it is not certain that these should be politically directed solutions.

This Substack is reader-supported. To receive new posts and support my work, consider becoming a free or paid subscriber.

Archive