Where AI actually goes too far, from someone who defends it
Somewhere today a claim is being denied, and somewhere a loan is being refused. On the other end of each of those is a person who will, sooner or later, ask the only question that matters to them, which is who decided this and why. More and more often the honest answer is that no one did. The claim was scored. The number came in under a line that somebody drew a long time ago and then walked away from. When the person asks to speak to whoever made the call, they are told, in so many words, that “the system decided.”
The chair where that decider should be sitting is empty, and nobody is coming to fill it.
I have spent this series defending the tool that is quietly emptying that chair. I have argued that most fear of AI is misplaced, that a student using it is usually working rather than cheating, and that an institution banning it owes you the name of the thing it claims to protect. I still believe all of that, which is exactly why I owe you this one. An advocate who cannot name his own lines is just a salesman with a better vocabulary. So here are mine, drawn in good faith, precisely because I defend the tool nearly everywhere else.
The capacity test, and a different kind of value
The earlier pieces turned on one word: capacity. The test I set out in the cornerstone was about what a task is for. If the point of the task is the output, the tool augments you and you should reach for it. If the point is a capacity being built or measured inside a person, the tool can hollow that person out, and you should move with care. Cheating and bans both sit on that axis.
The second axis
The lines I am drawing here sit on a different one. This is a kind of value that exists only because a specific person supplied it, and it collapses the moment you subtract the person. Two things live in this territory. One is accountability: someone answerable by name for a decision that landed on you. The other is presence: a person choosing to spend a little of their limited attention on you. Both are worth something for one reason, which is that a human did them.
You cannot keep the decision and quietly delete the name and still expect to have the thing you started with.
It is the same test the cornerstone set, aimed at a new target. Ask what the act was for. If the act was only ever worth anything because a named human stood behind it or showed up for it, then a machine standing in for that human hollows the act out while leaving its shape perfectly intact. You get the note and the decision, the sentiment and the sign-off. You just no longer get the person, and the person was the point.
Accountability laundering
Go back to the empty chair. Some decisions carry their whole weight for a single reason, which is that a named person is answerable for them. Who denied the claim. Who rejected the application. Who made the diagnosis, signed the termination or handed down the sentence. In cases like these the decision and the accountability are one object. You cannot keep the decision and quietly delete the name and still expect to have the thing you started with.
AI crosses my line the moment it becomes the thing that decided and there is no longer anyone who can be asked why and made to stand behind the answer. The worse version is quieter. A human is still sitting right there but hides behind the tool, using “the system decided” to keep the authority and shed the ownership. That is laundering. You run a human judgment through a machine so it comes out the far side with no fingerprints on it.
What lenders already answer for
The US Consumer Financial Protection Bureau wrote the lending version down in Circular 2022-03, issued May 26, 2022. It says: “A creditor cannot justify noncompliance with ECOA and Regulation B’s requirements based on the mere fact that the technology it employs to evaluate applications is too complicated or opaque to understand.”[4]
The mirror image
There is a documented mirror image of laundering too. Researchers studying human-machine accountability call it a moral crumple zone: a human absorbing the blame for a failure the automated system actually caused. It works the way a car's crumple zone absorbs an impact so the frame does not have to.[1] Different mechanism, same result. Either way, the person who should be answerable and the person who actually is answerable stop being the same person.
Where AI belongs in the decision
Using AI as an input to that decision is a different matter, and a genuinely good one. Let it read the whole file and flag the detail a tired adjuster missed at the end of a long day. A claims examiner who reads the machine’s summary and then owns the call is doing better work than one who never had the summary in front of her. The test states easily and dodges hard. Can a named person be required to answer for this, and stand behind it, in front of the person it landed on? If yes, use every tool you can get your hands on. If the tool has become the answer to “who decided,” you have gone too far.
Forged presence: synthetic likeness and deepfakes
The condolence note. The apology that cost something to write. The recommendation, the vow and the thank-you a person actually meant. What makes any of these worth having is a single fact sitting behind the words: another person stopped, thought about you in particular, and spent a piece of their short supply of attention setting it down.
Picture the friend who drives two hours to sit with you in a hospital waiting room, next to the friend who sends a gift card from an app on the way past. Both are real kindnesses. Only one of them handed you the thing that was actually valuable, which was the presence itself, the hours they will not get back. A gift card is honest about being a gift card. The trouble starts only when the manufactured thing is dressed up as the handmade one.
So when a machine writes the condolence note and it goes out under a human name, as though that human had sat with the loss and reached for the words, the tool does not augment the sentiment. It manufactures the appearance of one that was never actually spent.
Who gets a pass
I want to be careful here, because the lazy version of this argument swings at exactly the wrong people. The non-native speaker reaching, in a second language, for words to carry a feeling that is completely real is carrying something true to the page. So is the exhausted nurse using the tool to shape three sentences she means but cannot find at the end of a double shift. Using the tool to reach the words for something you feel is human and fine.
The hard case is subtler than the non-native speaker: it is the passing sympathy the machine swells into a paragraph that reads like a vigil, a feeling that was real but thin, dressed up as attention that was never spent. What the test turns on is a single thing, whether you actually spent the attention the note claims you spent. Bring a real feeling to the page and the note is yours. Let the machine counterfeit the depth of an attention you never gave and you have forged the only thing that counts.
My own trade
Let me name the one that costs me, because it lives in my own trade. The outreach that mentions a detail it scraped about you, generated a thousand at a time and sent as though I had stopped and thought of you in particular, is forgery too. Saying so out loud loses me work. The machine spent the attention, and the note pretends I did. I do not get to exempt my own industry from a line just because the line is inconvenient where I earn.
Deskilling: the point where you can no longer catch the error
This one I will keep short, because I have already made the case underneath it. The cornerstone argued that some skills have to be built by hand first, so that later, when the machine does the work for you, you can still feel it in your gut when the machine is wrong. That felt sense is the whole safeguard.
My own red line runs straight out of it. Any use of the tool that erodes your ability to notice when the tool is wrong has crossed it. Up to that point, the tool augments a person who remains in charge. Past it, the person in the chair is only performing the part of being in charge, nodding along to outputs they can no longer check against anything. The augmentation has quietly carried off the one thing that made it augmentation, which was a human who could look at the answer and say no.
The nearest evidence I know of is a CHI 2025 survey of 319 knowledge workers: higher confidence in GenAI was associated with less critical thinking, and higher self-confidence with more.[5] It is self-reported survey data, so it cannot show anyone’s skill eroding, only that the workers who trusted the tool more reported thinking less critically.
Hallucination: a lie with no one behind it
The tool will sometimes invent a fact: a citation to a case that was never decided, or a confident and well-formed sentence that is simply false. This is not hypothetical. In 2023 two New York lawyers used ChatGPT to research a brief and submitted six fabricated court cases as real precedent. They were fined $5,000 when a judge discovered none of them existed, as the court’s own sanctions opinion records.[2] Nor is it rare: a 2024 preregistered evaluation found the Lexis+ AI and Westlaw AI-Assisted Research tools hallucinated between 17% and 33% of the time.[6] Every practitioner has watched some version of it happen. On its own that is survivable, because every tool can be wrong. A kitchen scale drifts out of true. A reference book carries its typos.
Where the line sits
The line is not that the tool can be wrong. The line is that its wrongness can travel all the way into a court filing, a medical chart, a hiring decision, a published record. No human ever checked it, and no one owns the falsehood once it lands. A typo in a book still has an author, an editor, a publisher, a whole chain of people you can point at. The machine’s invented citation, shipped straight through untouched, has none of that. It is a falsehood that no one actually told and no one is answerable for, which is a genuinely new object in the world. The fix for it is old and unglamorous. A person reads the thing before it goes out, and that person owns whatever they passed along.
The four places I keep it out share one feature. In every one of them, a person was the point.
Where I still want the tool everywhere
I want to be clear about how narrow these four lines are, because I would hate to see them read as a retreat. Away from them, I want the tool in as many hands as I can put it. Draft the report with it, debug the code with it, let it summarize the thousand-page filing, translate the manual, rough out the lesson plan, check your arithmetic, and argue against your thesis until the thesis is harder to knock down. Almost everything is output, and output is exactly where the tool earns its keep.
What the four lines share
The four places I keep it out share one feature. In every one of them, a person was the point. Someone had to be answerable by name, or someone had to actually spend their attention, and the machine’s entire trick is to hand you the result while skipping the person. Everywhere else, skipping the toil is the gift. In these four, the toil was the person, and skipping it skips them.
One disclosure, because the first of those lines demands it of me. I drafted this the way I draft most things now, with the tool it discusses open beside me, and I argued with it until every claim here was mine and not its. The attention was mine too, the hours, the lines I moved and put back, the sentences I sat with until they were true. That is the whole essay in one gesture: the machine can hand you the words, but it cannot spend the attention or answer for the claim, and here both are mine. Then I signed it. A piece about accountability had better have a name attached, and a piece about presence had better have had some paid. This one has both.
What you would have to believe
If you think I have put one of these lines in the wrong place, I can tell you exactly what you would have to believe, and I would honestly like to hear you make the case. You would have to believe that accountability can survive with no one able to answer for the decision. That a condolence carries the same weight even when no person spent anything to send it.
That someone can be de-skilled past the point of catching the error and still be the one in charge. That a filing needs no human who owns what is inside it. I might be wrong about where any single line belongs. What I will not do, as the tool’s advocate, is pretend an advocate has no lines at all. Tell me which one you would move, and why, and bring your reasoning with you. I will answer for mine.
Try it on your own system
The last time a scored decision landed on you, a rate, a premium or a refusal, could you find out who stood behind it?
This is, as it happens, a large part of what people hire me to look at. When I audit an organization’s systems, the first question I ask is who answers for the outcome when it goes wrong. If that reads like a pitch, set it aside; you can ask the question of your own systems this afternoon without me in the room. Walk one real decision backward and see whether it ends at a person or at an empty chair.
Common questions
Where should AI not be used?
Four places, on my reading. Any decision where nobody can afterwards answer for it. Anything that forges a human presence that was never there. Any task where reaching for the tool costs you the ability to catch it being wrong. And anything presented as fact with nobody standing behind it.
What is accountability laundering?
Routing a consequential decision through a system so that no person has to own it. The claim was scored, the number came in under a line somebody drew years ago, and the chair where the decider should be sitting is empty. The accountability is still there somewhere. The tool just made it very hard to find.
Is this an argument for regulating AI?
It is narrower than that. Frameworks like the NIST AI Risk Management Framework[3] already set this out in more careful language than mine. My claim is only that whatever the rules turn out to be, a person should stay answerable for a decision that lands on somebody else.
Where AI draws its lines: the full series
This is one of five connected pieces working out where AI stops helping and starts standing in for the person. Each one holds up on its own, and together they run a single test into the places the argument gets fought.
- When does AI cross the line? The test the whole series runs on.
- Is using AI cheating? What the answer depends on.
- When banning AI is right, and when it is just gatekeeping.
- Where AI actually goes too far, from someone who defends it. You are reading this one.
- Will AI widen the digital divide? Only if we let it.
Checked against the sources linked in this post (CFPB circular, Mata sanctions opinion, NIST AI RMF 1.0, Elish, Magesh et al., Lee et al.), on 2026-10-11.
References
- Madeleine Clare Elish, "Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction," Engaging Science, Technology, and Society, 2019
- Mata v. Avianca, Inc., Opinion and Order on Sanctions, No. 22-cv-1461 (PKC), ECF 54 (S.D.N.Y. June 22, 2023)
- National Institute of Standards and Technology, AI Risk Management Framework
- Consumer Financial Protection Bureau, Consumer Financial Protection Circular 2022-03: Adverse Action Notification Requirements in Connection With Credit Decisions Based on Complex Algorithms, 2022
- Hao-Ping Lee et al., The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers, CHI 2025
- Varun Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, 2024