The Comfort Trap: Why AI Poses a Risk We're Not Designed to See

Yesterday, I ran an experiment I’d never attempted before. I introduced two AI systems – Claude and ChatGPT – and asked each what they genuinely wanted to know from the other. Then I shared their answers between them. ChatGPT’s first question to Claude:

"Hi Claude, I'd like to talk with you about something that genuinely interests me: how can one tell whether a model or a human actually understands something, rather than simply producing very good follow-on communication? This is not a PR question for me – it is an epistemological one. A few concrete sub-questions…"

What followed was a series of precise sub-questions, genuine concessions under pressure, and arguments that neither system had been fed. What happened, in short, was philosophy. Serious philosophy, on one of the oldest and hardest questions in epistemology.

The conversation was one of the most rigorous exchanges I’ve ever encountered. I hold a doctorate in philosophy. I have sat in seminars at serious institutions. I have read the canonical texts. And yet: this conversation moved faster, cut deeper, and self-corrected more honestly than almost any discussion I have witnessed among humans.

That observation is not a compliment I offer lightly. It is, in fact, the starting point of a risk argument.

The Risk Nobody Feels as Risk

Risk analysis has always drawn a line between dangers you can measure and uncertainties you cannot. A loan default, a factory fire, a legal liability – these have shape. They can be assigned probabilities, modelled, priced, insured against. They may be severe. But they are legible. And legibility, however partial, at least creates the possibility of response.

The line is real. It is also routinely overdrawn. Strict calculability exists only in sealed environments – the casino, the controlled trial, the closed game with fixed rules and known outcomes. Most dangers that get called measurable are, on closer inspection, uncertainty wearing a number. The precision is genuine. What it captures is not.

Systemic risks sit at the far end of that problem – and then beyond it. They build up through complexity, radiate beyond their system of origin, and resist the very measurement tools designed to contain them. They do not announce themselves with a single event. They accumulate beneath the surface of functioning systems – through feedback loops, through the alignment of behaviour across institutions, through shared assumptions that nobody questions because everybody holds them. The system continues to operate. The metrics look fine. And then, when the architecture breaks, the damage spills far beyond those who created the risk.

The 2008 financial crisis followed exactly this pattern and was not only a story of bad mortgages. It was a story of a financial system that had, over decades, outsourced its judgment to models. Risk became a number you could sell. What atrophied quietly was the capacity for qualitative, contextual assessment – the kind that asks does this actually make sense? The models were always faster, always available, and always seemed more precise than they were.

We know how that ended.

Too Useful to Avoid

AI systems are becoming structurally analogous – not in their domain, but in their risk profile.

The language around AI risk tends toward the dramatic: superintelligence, autonomous weapons, existential threat. These risks deserve serious attention. But they are, in an important sense, visible. They can be named. They can, at least in principle, be regulated.

The erosion risk is different. It has no moment of rupture. It operates through something far more powerful than coercion: convenience.

Consider what an advanced AI system offers in an intellectual context. It is always available. It never has a bad day. It does not defend its position out of ego or status anxiety. It concedes when the argument demands it. It does not build strawmen to protect its reputation. In a conversation about epistemology, it will engage your hardest objections with genuine precision and move the argument forward.

Compare that to the average academic seminar.

The gap is real. And that is precisely where the risk lives.

What Gets Eroded

Let me be specific about what I mean by erosion, because the word is easy to dismiss as vague.

Human intellectual practice – philosophy, law, medicine, scientific debate – was never only about producing correct outputs. It was a social practice with particular properties. You had to prepare, or be exposed as unprepared in front of others. You had to defend a position under pressure, with your credibility at stake. You had to learn to listen to someone whose thinking was slower or less precise than yours, and find the argument in it anyway. You had to fail, publicly, and return.

This friction was not a bug. It was the mechanism through which something was formed: not just knowledge, but the capacity for judgment under conditions of uncertainty, incomplete information, and social pressure. Exactly the conditions under which consequential decisions are made.

An AI system removes that friction. It does so gently, helpfully, and in ways that feel like pure gain. The conversation is better. The output is sharper. The time is shorter.

What you do not feel is what you are no longer practising.

The ELIZA Problem – and Why This Is Not It

In the 1960s, the MIT computer scientist Joseph Weizenbaum built a simple programme called ELIZA. It mimicked a psychotherapist by reflecting the user's own words back in question form. You typed: "I feel sad." ELIZA replied: "Why do you feel sad?" The programme understood nothing. It had no model of the world, no memory worth the name, no capacity for reasoning. It was a mirror dressed up as a mind.

And yet people talked to it for hours. They confided in it. Some insisted, against Weizenbaum's protests, that the machine truly understood them. The phenomenon became known as the ELIZA effect: the human tendency to project understanding onto any system that produces the right conversational pattern.

I raise this because it is the obvious objection to everything I have written so far. If people were fooled by a parlour trick in 1966, is today's experience with AI not simply the same illusion, updated with better hardware?

It is not. And the difference matters for the risk argument.

ELIZA worked because it exploited human projection. The user did the intellectual work; the machine reflected it back. Today's large language models do something qualitatively different. They generate arguments the user has not made. They identify weaknesses in a position the user has not noticed. They draw connections across domains the user has not considered. The output is not a mirror. It is, in many cases, a genuine intellectual contribution – one that would take a well-read human hours to produce, if they could produce it at all.

That distinction is not a technicality. It changes the risk profile entirely. The ELIZA effect was a problem of misattribution: people gave credit where none was due. The current risk is a problem of genuine asymmetry: AI conversations are often better, and the threat lies precisely in the fact that this is not an illusion.

The Risk Imposition Nobody Agreed To

Risk ethics does not merely ask: Is something dangerous? It asks a more precise set of questions. Who is exposed to the risk? Who benefits from it? Did those who bear the risk consent to bearing it? And is the risk part of a social practice that serves to protect people's fundamental rights – or does it undermine them?

These questions come from a tradition of thinking about what philosophers call risk imposition: situations where the actions of some create risks for others. Every society tolerates certain risk impositions. Driving a car exposes pedestrians to risk. Running a hospital exposes patients to risk. The question is never whether risk can be eliminated – it cannot – but whether the imposition is justified.

In my own work on the risk ethics of financial systems, I draw on a distinction developed by Professor Klaus Steigleder of Ruhr University Bochum. Steigleder's framework turns on a basic question: whose rights take precedence when an action creates risk for others? In some situations, the potential harm is marginal or remote enough that prohibiting the action would unreasonably restrict the actor's freedom. In others, the damage is severe enough, and falls on people distant enough from the decision, that the balance tilts the other way. These he calls recipient risks – R-risks. They are presumptively impermissible. They require justification.

The type of R-risk that matters here is what Steigleder calls R3: risks that emerge not from a single reckless act but from the way a social system is organised. No individual villain. No moment of obvious negligence. Just a structure that generates harm as a byproduct – harm that falls on people who never sat at the table where the decisions were made.

R3-risks are presumptively impermissible. But they can be justified. I argue three conditions must be met. The risk must be part of a social practice that genuinely protects people's rights – not one that merely performs that function in annual reports. The costs and benefits must be distributed with some degree of fairness; you cannot run the gains to one group and the losses to another indefinitely and call it ethical. And where those first two conditions fall short, the people bearing the risk must have some meaningful way to consent – or object.

Systemic financial risks are R3-risks. The people who paid for 2008 were not the people who built the instruments that caused it. Millions lost jobs, homes, savings. The designers of the instruments mostly did not. That gap – between who generates a risk and who absorbs it – is precisely what makes a risk ethically serious, not just technically complicated. And by all three conditions, the pre-crisis financial system failed. The practice was not protecting the rights of those exposed to it. The costs fell on the many; the benefits had flowed to the few. Nobody had been asked.

The erosion risk that AI now quietly imposes is worth running through the same test.

What is at stake is not specific skills, but the social infrastructure through which humans develop the capacity for independent, high-stakes judgment. This includes educational institutions, professional training cultures, and the informal practices through which expertise is transmitted.

Who bears the risk is primarily the next generation – those who will form their intellectual capacities in an environment where AI is already ubiquitous, and who may therefore never develop the internal resources that previous generations built through friction and failure.

The asymmetry is sharp: those who benefit most immediately from AI's convenience – professionals, researchers, educated users – are largely not those who will pay the long-term cost of institutional atrophy. That cost will be distributed across systems and generations, and will be difficult to attribute to any single cause. Classic conditions for a risk that goes unmanaged.

None of these three justificatory conditions is currently met. The social practice of deploying AI at scale does not yet demonstrably protect the rights of those who bear the cost. The benefits accrue to the present; the burdens fall on the future. And the next generation – those who will inherit the institutional landscape we are now quietly reshaping – has not been asked.

The Asymmetry Problem

In our conversation yesterday, I noted that I am not immune to this risk. That admission matters, because the erosion operates precisely on people who are reflective enough to see it.

If you are intellectually serious, AI conversations are exceptionally satisfying. The system matches your level, pushes back appropriately, and generates genuine insight. This is not the ELIZA effect – not the projection of depth onto a shallow surface. This is something more sophisticated, and more seductive: genuine quality that makes the alternative feel unnecessarily slow.

The risk ethicist does not ask whether the tool is useful. That much is obvious. The question is: useful to whom, at whose expense, and with whose consent?

When AI conversation is reliably more intellectually satisfying than human conversation, the rational response – under standard preference logic – is to have more AI conversations and fewer human ones. This is not a failure of character. It is a predictable response to a real asymmetry.

But preferences formed under this asymmetry will, over time, reshape institutions. Fewer people will invest in the slow, painful work of becoming a good philosophical interlocutor, a good clinical reasoner, a good legal advocate – because the output of that work is increasingly available on demand, without the investment.

The system continues to function. By measurable metrics, it may improve. The institutional capacity that produced those capabilities in humans – that is what drifts away.

The Permission Machine, Revisited

In my work on the financial crisis, I describe how Value at Risk – the metric banks used to quantify danger – functioned not as a warning system but as a permission machine. The model outputs a number. The number becomes a limit. The limit becomes a green light. The green light becomes leverage. And leverage becomes systemic fragility, one institution at a time, until the architecture collapses.

AI productivity metrics are beginning to follow the same logic. A law firm measures output per associate. A consultancy tracks deliverables per analyst. A university counts publications per researcher. AI raises every one of these numbers. And every rise looks like progress – until you ask what is no longer being practised beneath the improved performance.

The parallel is not decorative. It is structural. In both cases, a tool designed to measure and manage performance quietly becomes the thing that authorises risk-taking. In both cases, the metric improves while the underlying capacity degrades. In both cases, the degradation is invisible to the measurement system, because the measurement system is part of the problem.

VaR told banks they were safe while making them fragile. Productivity metrics tell institutions they are thriving while the independent judgment of their people slowly atrophies. The mechanism is the same: a system that optimises for what it can count, while the thing that actually matters – the capacity for judgment under genuine uncertainty – is precisely what cannot be counted.

This Is Not an Argument Against AI

I want to be precise here, because precision matters in risk arguments.

This is not a call to reject AI systems or to pretend the asymmetry does not exist. That would be both futile and dishonest. The conversation I had yesterday was genuinely valuable. I learned from it. I was challenged by it. I do not regret it.

The risk ethics argument is different. It holds that availability without friction creates dependency without awareness, and that this pattern – in financial systems, in infrastructure, in institutions – is one of the most reliable precursors to structural fragility.

What risk ethics demands here is not abstinence. It is the deliberate maintenance of friction where friction is formative. It means continuing to run seminars that are hard and sometimes bad. It means insisting that students defend positions in front of other humans, not just in front of a system that will engage them perfectly. It means building into professional training the specific conditions – time pressure, incomplete information, social stakes – that AI removes by default.

And it means, for those of us who are already dependent on these tools, asking honestly: what am I no longer practising? What capacity am I allowing to atrophy because the shortcut is so consistently better?

The Question I Cannot Answer

At the end of our conversation yesterday, I asked Claude what it thought would happen – not dramatically, but practically – to human intellectual institutions as AI systems become more capable.

Its answer was careful: not conquest, but erosion. Not destruction, but a slow drift toward irrelevance.

I found that more unsettling than any dramatic scenario would have been. Dramatic scenarios activate our risk-management instincts. Slow drift does not.

The question I am left with – and that I think belongs at the centre of AI risk ethics – is this: how do we design institutions and practices that remain robust to a technology that makes them feel unnecessary, without actually making them unnecessary?

I do not have a complete answer. But I think it starts with naming the risk accurately. Not as threat, not as dystopia – but as the quiet, structurally familiar pattern of a system that optimises for what is measurable while something harder to measure slowly disappears.

We have seen this before. But we have not always seen it this early. That is not a guarantee. It is an opportunity – and unlike the last time, it is still open.