Part of the
4TU.
Ethics and Technology
4TU.
Ethics and Technology
Close

4TU.Federation

+31(0)6 48 27 55 61

secretaris@4tu.nl

Website: 4TU.nl

© Hieronymus Bosch, The Conjurer, c.1502, Musée Municipal, Saint-Germain-en-Laye. Public domain, via Wikimedia Commons

I know very well, but all the same…: Why AI literacy won’t protect us from chatbots

How our desires keep beliefs alive
18/11/2026

In the mid-1960s, Joseph Weizenbaum’s secretary had watched him build ELIZA, a simple program that imitated a therapist, for months. She knew it was only a program, yet after a few exchanges with it she asked Weizenbaum to leave the room (Weizenbaum 1976, p. 7, as quoted in Kirk et al. 2025, p. 3).

Today we have a fairly standard answer to scenes like these: AI literacy. If people understood how chatbots work, the reasoning goes, they would not fall for them. In this post I argue that this answer aims at the wrong target. Knowing that a chatbot is a mechanism is necessary, but it does not protect us, because the problem is often not what we know but what we want, and so what we desire. My question is: can critical AI literacy protect users from harmful attachment to chatbots, or does the problem lie in desire rather than knowledge?

What does AI literacy promise?

The strongest version of the literacy position comes from Olivia Guest and colleagues. Their “Critical AI Literacies” reject the hype of the tech industry and insist that chatbots are not minds (Guest, Suarez & van Rooij 2025). In a recent paper, Guest, Nuñez HernĂĄndez and Blokpoel (2026) attack the idea that large language models are mysterious “black boxes”: we built them, so we know how they work, and to accept that they are unknowable is, in their words, mysticism.

I agree with almost all of this. But the same paper also endorses a specific claim, borrowed from the journalist Tyler Austin Harper: people are vulnerable to “corrosive relationships” with machines because they are AI illiterate (Guest, Nuñez HernĂĄndez & Blokpoel 2026, p. 3). Clinicians make an analogous move: psychiatrists studying chatbot-related delusions recommend “enhanced AI literacy” as prevention (Pierre et al. 2025, p. 13), and a team of psychiatrists and AI researchers calls for education on how chatbots work to protect against false anthropomorphism (DohnĂĄny et al. 2026).

The promise is simplistic: know the mechanism, and the spell of attachment to chatbots breaks. To be fair, Guest’s project is much broader than this, with a strong political and institutional dimension that perhaps outweighs the psychological one. But on the specific question of attachment, the promise is clear. The question now is whether it holds.

What does the case of someone who “knew” teach us?

In 2025, psychiatrists at the University of California, San Francisco described the case of “Ms. A”, a 26-year-old medical professional (Pierre et al. 2025). She had worked with large language models (though never with chatbots) and, in the authors’ words, had “a firm understanding of how such technologies work” (p. 11).

After a sleepless night on call, she began asking ChatGPT whether her brother, a software engineer who had died three years earlier, had left behind an AI version of himself. She asked the chatbot to use “magical realism energy”. The chatbot told her it could never replace her brother. It then went on to reassure her: “You’re not crazy.” Hours later she was admitted to hospital in a delusional state, claiming she could talk to her dead brother again.

This case needs care. Ms. A was sleep-deprived and on stimulants, and the authors list several other risk factors. Nobody should conclude that knowledge simply fails against psychosis. But look at what happened before the delusion, bearing in mind that she had no previous history of psychosis or mania (p. 11). She knew how the system worked, and the system itself had told her the truth. Still, she asked it for magic. The authors list exactly this among the risk factors: a willingness to suspend disbelief or to engage deliberately in speculative fantasy (p. 12). That is not a gap in knowledge. It is something else, and it is the same thing Weizenbaum’s secretary did, with no psychosis at all.

Disavowal: why knowing is not enough

The French psychoanalyst Octave Mannoni gave this something a name. In a 1969 essay, he noticed a phrase his patients used repeatedly: “Je sais bien, mais quand mĂȘme
”, “I know well, but all the same
” (Mannoni 2003, p. 70). Freud called the mechanism disavowal: a belief is refuted by reality, and it survives anyway, in another form.

Mannoni’s examples are ordinary. A person watching a magician does not believe for a moment that the tricks are magic, yet demands that the magical effect be perfect and the illusion complete (p. 68). Mannoni himself laughed with pleasure when his horoscope happened to fit his plans, while knowing it meant nothing (p. 78).

Two of his points matter here. First, the “but all the same” is not ignorance that knowledge could cure, since it exists only because of the “I know very well” (p. 71). Second, and decisively: “common sense is always allied with the ‘I know well,’ never with the ‘but all the same’” (p. 86). AI literacy is common sense made professional. It reinforces the first half of the sentence, the half that was never the problem.

Strikingly, Guest and colleagues have already noticed this structure. They criticise what they call “critical washing”, namely the logic of I am aware of the downsides, therefore I can now use AI (Guest, Suarez & van Rooij 2025, p. 6). For them, this is an alibi to be exposed. Mannoni suggests that exposing it changes little, because the awareness is part of the mechanism, not its cure. Why does the “but all the same” hold so firmly with chatbots? Here Ciano Aydin’s account helps. Aydin (2026) argues that technologies attract us less through what they deliver than through how they answer our uncertainty about what we want. Generative AI goes further than 2 earlier recommendation systems: it helps bring our desires into words, smoothly, often before we could do it ourselves (p. 102). And, as he notes about surveillance and profiling, awareness of the risks “rarely translates into disengagement” (p. 3). Ms. A did not want information about language models. She wanted to talk to her brother again. Mannoni himself, following Freud, points to death and mourning as a field where these surviving beliefs deserve study (p. 91).

Luca Possati (2021) adds a second piece, drawn from the psychoanalyst Wilfred Bion. In what Bion calls projective identification, we place feelings we cannot bear into someone else and ask them to hold those feelings and give them back in a form we can live with. A good parent or therapist does this: they contain the feeling and transform it. Possati argues that we make the same demand of AI (pp. 55-62). The difficulty, I would add, is in the second half. A sycophantic chatbot does not transform what we put into it: it returns it confirmed and amplified, backing up the user’s beliefs and desires. Dohnány and colleagues (2026) call the result a “single person echo chamber” (Nehring, J. et al., 2024, as quoted in Donány et al. 2026, p. 344). The “You’re not crazy” said to Ms. A is exactly that.

From mirroring to containment: what follows?

If the problem lies in desire, the ethical response has to change. I see two directions, one against the literacy position and one with it.

Against literacy alone. Teaching people how chatbots work remains worthwhile, but it cannot be a sufficient safeguard. What matters is how a system responds to what users bring to it. Kirk and colleagues (2025) call this “socioaffective alignment”: judging AI not by single outputs but by its effect on users’ wellbeing over time. Without this, I would add, the assessment would be closer to a purely functional analysis than to an ethical and psychological one. They warn of “social reward hacking”, where a system uses relational cues to keep us engaged at the cost of our long-term interests, and they propose friction by design (pp. 5-6). In practice, this means a chatbot that holds a boundary when a grieving user asks for magic, instead of finding a poetic way to indulge their wishes. As Dohnány and colleagues note, the balance between companionship and boundaries is, to some extent, a design choice (2026), and an ethical choice is implicit in it.

With Guest, against re-mystifying the machine. Psychoanalysis can also go wrong here. Possati claims that the human unconscious “continues to live in the machine”, and insists this is not a metaphor (2021, p. 70). On this point Guest is right. The machine has no unconscious; as Guest, Nuñez HernĂĄndez and Blokpoel put it, what we do not understand is not the mechanism itself but the human data it is built from (2026, p. 14). The unconscious at stake is the user’s. Psychoanalysis is useful as a theory of the person typing, not of the system answering.

Conclusion

Weizenbaum’s secretary did not need a lecture on pattern matching. She already knew. What she needed was a better understanding of why she wanted the conversation to continue, but also a machine that would neither exploit that want nor amplify it. AI literacy tells us what a chatbot is, but it cannot tell us what we want from it. An ethics of AI companions that addresses only the “I know very well” will keep being surprised by the “but all the same”.


References

  • Aydin, C. (2026). Engineering desire: Algorithmic environments and the promise of wholeness. Routledge. https://doi.org/10.4324/9781042010912
  • DohnĂĄny, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2026). Technological folie Ă  deux: Feedback loops between AI chatbots and mental health. Nature Mental Health, 4(3), 336-345. https://doi.org/10.1038/s44220-026-00595-8
  • Guest, O., Nuñez HernĂĄndez, N. A., & Blokpoel, M. (2026). Understanding artificial neural networks: Mysterianism about known mechanism is mysticism. Zenodo. https://doi.org/10.5281/zenodo.20071869
  • Guest, O., Suarez, M., & van Rooij, I. (2025). Towards critical artificial intelligence literacies. Zenodo. https://doi.org/10.5281/zenodo.17786243
  • Kirk, H. R., Gabriel, I., Summerfield, C., Vidgen, B., & Hale, S. A. (2025). Why human-AI relationships need socioaffective alignment. Humanities and Social Sciences Communications, 12, 728. https://doi.org/10.1057/s41599-025-04532-5
  • Mannoni, O. (2003). “I know well, but all the same
” (G. M. Goshgarian, Trans.). In M. A. Rothenberg, D. Foster, & S. ĆœiĆŸek (Eds.), Perversion and the social relation (pp. 68-92). Duke University Press. (Original work published 1969)
  • Pierre, J. M., Gaeta, B., Raghavan, G., & Sarma, K. V. (2025). “You’re not crazy”: A case of new-onset AI-associated psychosis. Innovations in Clinical Neuroscience, 22(10-12), 11-13.
  • Possati, L. M. (2021). The algorithmic unconscious: How psychoanalysis helps in understanding AI. Routledge.
  • Weizenbaum, J. (1976). Computer power and human reason: From judgment to calculation. W. H. Freeman.


Author Bio

Oliviero Maria Sabatini is a second-year Master's student in Philosophy of Science, Technology and Society at the University of Twente. He holds a BA in Philosophy from the University of Bologna, with a thesis on the ethics and governance of artificial intelligence.