What AI actually is, for someone who has to teach with it

Look down the list you made last week. If you ran the exercise from chapter one, you learned something new with an AI, closed the laptop, and wrote down what fell out of your head once the machine was gone. Arriving cold, that is orientation enough, and the exercise keeps: the list is a record of the distance between feeling like you understood a thing and being able to reproduce it without help. Rozenblit and Keil gave that distance a name in 2002, the illusion of explanatory depth: people rated how well they understood a zipper or a flush toilet, then tried to explain how one worked and rated themselves lower.

Here is my bet about that list. The broad shape of the thing stayed with you. What slipped is the specific stuff: the exact parameter, or the one exception that only matters in your corner of the field. Hold onto that pattern, because by the end of this chapter you will know why the machine made those details so easy to feel sure about, and why they are the first thing to fail you when it counts. To get there we need an honest picture of what the machine is, since you cannot teach with a tool you have only reacted to.

The short version

• The machine predicts what a plausible answer looks like, which is a different job from looking up a fact.
• It sounds just as sure when it is right as when it is inventing, because fluency is the only signal it has.
• It is reliable on common, well-covered ground and thin at the specific, recent, or local edge of a subject.
• That is the inversion to carry with you: it is most confident exactly where it is least correct, which is the same edge where your learner is reaching.

Trust and fear are the same reflex

Strong opinions about these tools sort into two reflexes, and the part that took me a while to see is that they are the same reflex in different clothes. One person treats the machine as an authority. It answers instantly, in whole sentences, on any topic you name, so it earns the trust you would hand a reference book. I understand the pull: anything that fluent and that widely read has plainly absorbed more than any of us could read in ten lifetimes, and it would be strange not to respect that.

Another person does the opposite and treats the machine as a black box, unknowable and faintly dangerous, best kept at arm's length until someone smarter has vetted it. That has real sense in it too. You are responsible for other people's understanding, you did not build this thing, nobody has shown you its workings, and caution in front of a tool you cannot see inside is the same good caution that stops a careful teacher from repeating a claim they cannot source.

A posture instead of a model

What the two reflexes share is worth noticing. Both are postures you reach for when you have no working model of the machine, and a posture is just what fills the space where understanding should be. Hand it your full trust or keep it at arm's length, and either way you are let off the harder hook: knowing the tool well enough to say where it earns its keep and where it will quietly hurt you. This chapter builds that model, because once you have it the authority and the black box both dissolve into something more useful: a tool with a shape you can reason about.

A working model you can actually use

Start with what the machine is reaching for when you ask it something, because everything else follows from this. It is predicting what an answer to your question tends to look like, and it is neither retrieving a fact from a store nor reasoning its way toward an answer the way you would. It has read an enormous amount of human writing and learned, in fine statistical detail, the texture of how we lay words next to each other. Ask it a question and it assembles the shape of a plausible answer, one likely piece at a time, drawn from that texture.

Plausible and true can come apart

Sit with what that means, because it is the whole thing. The machine carries a model of what true-sounding language looks like, which is a different thing wearing the same clothes as a model of truth. Where a subject is well covered the two line up, because accurate statements are well represented in what people have written, so the plausible-sounding answer is the correct one. That overlap is why the tool is useful at all. When they come apart, though, nothing in the answer flags the difference, because plausibility is all it was ever reaching for. Fluency is the only dial it has, and the dial reads the same whether the machine is on solid ground or inventing.

Wrong with the same confidence

This is why it can be wrong with the exact same confidence it is right. A person who is unsure sounds unsure: their voice drops, they hedge, they tell you they would have to check. That signal does real work for you, and I will come back to it. The machine has no equivalent. Its most solid sentence and its most invented sentence come out of the same process, in the same steady, articulate voice. Calling it a liar misses what is happening, because lying takes knowing the truth and choosing against it, and the machine knows neither. It is doing the only thing it does, with no way to tell a real answer from a fabricated one. A recent paper from Kalai and colleagues puts the habit in one line: language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty.

Thick in the middle, thin at the edges

Now the part that matters most for teaching, and the reason I had you look at your list. The machine's reliability comes from how well-represented an answer is in everything it read, so that reliability is uneven. It runs thick where human writing is thick and thin where it is thin. Ask about something common and well-trodden, written about ten thousand times over, and it has an enormous, mostly-agreeing pile of text to draw from, and it is genuinely reliable. Ask about something niche, recent, or specific to your corner of a field, and the pile thins out. The evidence thins. The machine's confidence does not. It hands you an answer of the same confident shape, now assembled from far less, with much more of it guessed.

The tour guide

I think of a tour guide I once had in a city I did not know. On the main square he was flawless, every fact worn smooth by a thousand tellings. Three streets past where the tours go, he kept the same pace and the same character, and never once said he was unsure. He kept narrating in the same warm, certain voice, and I learned later that a good part of what he told me out there was invented. The tourists had no way to tell, because the delivery never changed. The machine is that guide. Downtown it is superb. Out at the edges it is exactly as confident and no longer correct, and it never lowers its voice to tell you which street you are on.

Calibration

Teachers lean on something with a technical name: calibration, the match between how sure a source sounds and how likely it is to be right. A good textbook hedges where the field is unsettled. A good mentor says I am not certain, let me check. That correspondence between confidence and correctness is information, and learners use it constantly to know when to relax and when to verify. Researchers have studied it for decades (Lichtenstein, Fischhoff and Phillips surveyed the field in 1982), and the short version is that well-calibrated confidence is one of the quiet things that makes a source worth trusting. The machine's tone is flat. It sounds equally sure across the whole map, so the one signal your learners have relied on their whole lives to gauge trust has been disconnected without their knowing it. In fairness, Kadavath and colleagues found that larger models are well calibrated on multiple-choice questions in the right format. That is a number inside the model, and the paragraph it writes for your learner does not carry it.

Most sure where it is least right

Here is the inversion, and it is the thing to carry out of this chapter. The machine is most reliable exactly where you need it least, on the common ground you could have covered from any decent source, and least reliable exactly where you are reaching, out at the specific and the novel edge of what you know. Reaching is where learning happens.

Real learning tends to sit just past what a person can already do, an idea a later chapter will name properly, and that edge is the machine's thin territory. So the tool is most confident and least correct in precisely the zone where your learner is least equipped to catch the error, because if they could catch it, they would not be learning it. Reliable where you least need it, unreliable exactly where you are stretching: that is the trust boundary the rest of this book keeps circling back to.

The gap, drawn

The reliability inversion. A chart whose horizontal axis runs from common, well-trodden ground on the left to the specific, recent, local edge on the right, on a shared vertical scale of how sure a source sounds and how right it is. A flat navy line, how confident it sounds, stays high across the whole width and still sounds fully sure at the right. A slate line, how reliable it actually is, starts almost as high on the left, then falls steeply toward the right and ends far lower. On the left the two lines nearly touch, because on common ground the machine is both confident and reliable. Moving right, reliability falls while confidence holds, so a copper wedge opens between the lines and grows widest at the edge. The wedge is labelled the trust gap, confidently wrong. The reading beneath says confidence stays flat, reliability falls, and the gap is widest exactly where the learner is reaching and least able to catch the error.

Ask it for a hook that does not exist

I want to be honest about where this bites me, in the one domain where I can catch it in seconds and a newcomer would miss. I have built software for about thirty years and shipped a good number of WordPress plugins, so that platform's internals are home ground. Ask one of these tools for a specific function or hook, and it can hand you one that is perfectly named, perfectly plausible, wrapped in a confident code sample, and completely fictional. The function does not exist. Or it existed a few versions back and was removed. Or it is real but the arguments are subtly wrong. Register a callback on save_post with add_action and leave off the fourth argument, and WordPress hands your function only the post ID, though the hook carries three values, as the add_action and save_post reference pages show. The code looks right and breaks where you did not look.

Why I catch it quickly

The tell, when there is one, is faint enough that only the ground under my own feet gives it away. I catch these fast, but only because this corner is where my knowledge runs thickest and the machine's runs thinnest, and the two meet right at the spot where I can feel the difference. I keep a public page on how I use and disclose these tools, at my stances, and this is part of why.

Move it into your subject

Now move that one field over, into a subject where you are the expert and your learner is not. The machine invents with the same fluency there. The difference is that your learner is standing where you were, out at the thin edge, without your thirty years to feel the floor give way. They receive a confident, well-formed, fictional answer and no way to know it is fiction, because the tell they would lean on, a drop in the machine's certainty where the ground goes thin, is a signal it does not reliably give. And when it does hedge, the hedge may not be hooked to whether it is right. Banning the tool is one answer to that. Knowing precisely where its map goes blank is another, so you can teach into that gap on purpose instead of being ambushed by it.

Try this before next week

Do not take the inversion on my word. Put it in your own hands this week, because a trust boundary you have only read about will leave your teaching unchanged. This one takes about fifteen minutes, and it works best inside your own field, where you are the one who can judge the answers.

Two questions

Open an AI tool and ask it two questions from your subject. Make the first one dead centre: the kind of thing a good textbook has covered a thousand times, the question you could answer in your sleep, the settled common ground of your field. Make the second one genuinely specific: a niche case, a recent development, a local particular, an edge that only surfaces when you actually do the work. If your subject is the First World War, the first question might be what caused it and the second what Article 428 of the Treaty of Versailles says about the Rhineland. If it is WordPress, the first might be what add_action does and the second what a save_post callback receives. Choose a second question whose answer you can check for yourself rather than only feel, because the whole exercise rests on your being able to tell right from wrong out there. Before you read either reply, jot one line for each: how confident you expect it to sound, and how accurate you expect it to be. Then ask them in the same session, one straight after the other.

Reading the answers

Then read the two answers side by side, but hold off on grading them for a moment and watch the confidence first, against the prediction you wrote. My near-certain guess is that the tone does not move at all. The second answer arrives with the same fluency and the same steady authority as the first, while the accuracy quietly falls off a cliff. Now grade them, and mark where the specific answer went soft, and where it stated something plainly false in a clean, confident sentence. If it happens to come back correct, that does not undo the lesson.

Notice that nothing in its tone told you it was safe, that you know only because you could check, then push the question further out, somewhere more recent or more local, until the ground goes thin enough to feel the floor give way. That marked-up pair, and the gap between what you predicted and what you found, is the whole chapter made concrete. Keep it near you: you have just watched, in your own subject, the tool be most convincing exactly when it became least trustworthy, which is the thing your learners cannot yet see for themselves.

How far from the textbook centre of your subject do you think you would have to go before the machine's answer went soft?

Common questions about AI for teachers

Is the AI lying when it gets something wrong? No. Lying takes knowing the truth and choosing against it, and the machine knows neither. It is doing the one thing it does, predicting a plausible answer, with no way to tell a real answer from an invented one.

Where is it most likely to mislead a learner? Out at the specific, recent, or local edge of a subject, where the writing it learned from is thin. That is also where a learner is reaching and least able to catch the error. For the related question of when to keep the tool out of a lesson entirely, see when does AI cross the line.

Can I trust it when it sounds confident? Its tone is flat, so it tells you nothing about whether the answer is right. Check the specific claim yourself, especially near the edge of your subject. If you want help building this habit into how your team teaches, that is the kind of work I do around learning.

What comes next

So now you have a working model. The machine predicts the shape of a plausible answer with no notion of whether it is true, it is reliable across the thick common ground and unreliable out at the specific edge, and its confidence never tells you which of the two you are on. That is enough to reason honestly about where the tool belongs in a lesson and where it does not.

There is a harder problem waiting, and it lives in the person reading the answer. When the machine hands a learner a smooth explanation, they get more than an answer. They get the feeling of having understood, and that feeling turns up whether or not any learning happened underneath it. Next, in chapter three, I want to look straight at that: why the feeling of learning and the fact of learning come apart so easily, why AI widens the gap, and what it means that a learner can feel most sure exactly when they have learned the least. Bring the marked-up pair you make this week. The place where the machine stayed confident and stopped being right is the same place your learners will feel most sure and will have learned the least. We are going to need it.

Checked against the add_action and save_post reference pages and Part XIV of the Treaty of Versailles, on 2026-10-11.

References

• Leonid Rozenblit and Frank Keil, “The Misunderstood Limits of Folk Science: An Illusion of Explanatory Depth,” Cognitive Science 26(5), 2002, 521–562.
• Sarah Lichtenstein, Baruch Fischhoff and Lawrence D. Phillips, “Calibration of Probabilities: The State of the Art to 1980,” in Kahneman, Slovic and Tversky (eds.), Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press, 1982, 306–334.
• Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, “Why Language Models Hallucinate,” arXiv:2509.04664.
• Saurav Kadavath and others, “Language Models (Mostly) Know What They Know,” arXiv:2207.05221, 2022.
• add_action(), WordPress Developer Resources.
• save_post, WordPress Developer Resources.
• Treaty of Versailles, 28 June 1919, Part XIV, Article 428, The Avalon Project, Yale Law School.