Issue 10: Three systems told me they were done
I spent this week finding out that three of my own automations were reporting success on work they had not done. Nothing was broken. Nothing threw an error. The logs were green and the numbers were wrong.
That gap, between what a system reports and what it did, is the most expensive thing in AI implementation right now, and almost nobody schedules time to look for it.
A note on timing: this issue comes four weeks after Issue 09. I started two teaching jobs in late August and the newsletter waited while I got my footing. It is back on Fridays from here, and Issue 11 is next week.
Reader Question
I do not have a subscriber question in the queue this week, so I am answering the objection I hear most often instead. If you have a real one, reply to this email and it goes in the next issue.
"We ran an AI pilot last year. Nobody can tell me whether it worked."
I hear a version of this in almost every first conversation, and it is usually said with some embarrassment, as if the district did something wrong. It did not do anything unusual. It did the normal thing, which is to launch a pilot without writing down what things cost before the pilot started.
Here is why that one missing step decides everything that comes after. When you get to the end of a pilot, someone asks whether to keep it. Without a baseline, the only evidence in the room is how people feel about the tool. Enthusiastic staff say it saved time. Skeptical staff say it added steps. Both are describing the same weeks. There is no way to settle it, so the decision gets made by whoever is more senior or more tired, and then it gets made again next year.
The fix is smaller than it sounds and it is retroactive. Pick one workflow the pilot touched. Ask the two or three people who do it how long it took before, and how long it takes now. Write both numbers down, with the date, in one place. That is not research. It is thirty minutes and a shared doc, and it turns "nobody can tell me" into a number you can argue with.
I build this into the first thirty days of my 30/60/90 AI Implementation Plan for exactly this reason. The Foundation phase has one deliverable people skip and later wish they had: document the baseline time cost so day 90 has something to compare against. Everything in the Compound phase, the automation, the evaluation, the scale-keep-cut decision, runs on that number. Skip it and day 90 becomes a vote.
If your pilot is already running, you are not too late. A baseline reconstructed from three short conversations beats no baseline by a wide margin.
Worth Your Time
Schools spend billions on AI, but struggle to figure out what's worth buying by Robbie Sequeira
The number that stopped me: districts have access to an average of 3,001 digital tools, and students and educators use four. That is from Instructure's own Canvas data across 12.6 million K-12 users. The piece is a clear-eyed look at how much vetting work has been pushed down onto districts with almost no state or federal support. If you are on the buying side, read the section on what vetting involves, then ask whether your district has ever written down what a tool was supposed to change.
I Catalogued 111 AI Workforce Partnerships. Only 23 Can Tell You What Happened To A Student. by Dr. Aviva Legatt
Legatt built a public catalog of 111 industry-university AI partnerships and found that the column that mattered most was the one almost nobody filled in: what happened to students afterward. Twenty-three out of 111 published anything. This is higher ed, not K-12, and the pattern is identical. Announcements are cheap and follow-up is rare. Worth reading next to anything your own organization has announced this year.
10 Back-to-School AI Updates For Educators by Dan Fitzpatrick
A fast, practical roundup of what changed over the summer across Google Classroom, Copilot, and the major education tools. I am linking it because the start of the year is when tool decisions get made by default, and it helps to see the whole board at once. Skim it, pick the one thing that touches a workflow you already run, and ignore the other nine until December.
Try This
The five-minute read-back check.
This is the habit that caught three failures for me this week, and you can put it on any workflow that writes something: a spreadsheet, a form, a status field, a shared doc.
The problem it solves: most automations, and most AI assistants, report success when the write command returns without an error. That is not the same as the value being there afterward. Something else can overwrite it. A permission can silently block it. A partial read can come back empty and look like a legitimate zero.
Here is the check.
Pick the one field that matters. Not the whole system. The single value that, if it were wrong, would send the work backward. For me it was a status field that keeps contacts out of the send queue.
After the write, read it back separately. Not the confirmation from the write call. A fresh, independent read of that field. If your tool cannot do that, have it print the value it believes is there and check three of them by hand.
Wait, then read it again. This is the step people skip and it is the one that found my bug. My writes were succeeding and then reverting. A read one second later looked fine. A read the next morning did not. Check tomorrow, not just today.
Treat an empty result as a question, not an answer. When a read comes back with nothing, ask whether nothing is plausible. An empty inbox after seven days when the same mailbox held 157 messages in ninety days is not quiet. It is a signal that something upstream stopped.
Write down what you checked and when. One line, in the same place every time. The value of this compounds: the second entry is what turns a strange reading into a pattern.
Five minutes, once a week, on one field. If you run any AI workflow that touches student data or a system of record, run it Friday.
That's it for this one. If something here was useful, share it with someone who'd get something out of it. If it wasn't, that's part of the deal too. You can find more of what I'm working on at evalveconsulting.com, or book a call if you want to talk through what you're dealing with.
Talk soon,
Chris