Founder's Corner

The coach was tutoring blind

For weeks my AI coach was tutoring blind. The moment the whole design aims at — you miss a question you were sure about and ask "what did I misjudge?" — is exactly where it had the least to work with.

The context we passed it carried an opaque item id. No scenario, no stem, no options, no result. The coach had to ask the learner to paste the question back in, or guess. Nobody noticed, because a language model is very good at sounding informed about things it cannot see.

The fix is a server-side item snapshot composed into the context: scenario, stem, and option texts always; the learner's outcome and the correct answer only once a graded attempt exists. That's the same reveal rule the grading endpoint enforces, so an unattempted item never leaks answer-key material to the model. Option letters are left out on purpose: the client shuffles options per request, so "option B" would mean different things to coach and learner.

The same audit caught a sibling bug: the snapshot reported only the latest attempt, so a third miss on an item looked identical to a first miss and the hint ladder restarted at rung one — the same Socratic question the learner had already walked through. It now counts attempts and escalates.

Lesson: the model doesn't know what you assume it knows. Most "AI tutor" quality problems are context problems wearing a trench coat.

A browser crash deleted your exam

Picture it: 90 minutes into a 120-minute mock exam, your browser crashes. You come back to an empty answer form — and the clock is still running, because the server recomputes remaining time from when you started and stores nothing until you submit.

The answers lived in sessionStorage, which survives a refresh and dies with the tab. I had tested refresh. I had not tested crash. A second bug sat on the same path: any error during resume cleared the saved state, so a flaky network could wipe an exam too.

Nobody hit this — a code review did. I ran the branch through two different AI reviewers before merging, and one of them flagged the wipe as data loss. Fix: answers moved to localStorage, a resume pointer so the picker warns "exam in progress — the clock is still running", and only a confirmed client error clears state now.

Lesson: the unhappy path is the product. And a reviewer that isn't the author — human or model — is the cheapest place to learn that.

Sign-in broke and it wasn't my code

Users were looping on sign-in. Every debugging instinct says: something I shipped. This time the auth vendor's hosted account portal was answering all of its own API proxy requests with an HTML 404 page — it could never sync a session, so sign-in never completed.

Before blaming the vendor you have to prove it. Three checks: reproduced with bare curl and no cookies, confirmed the session was ACTIVE via their backend API, and reproduced the same 404s on another company's portal running the same build. Not my code.

You can't fix a vendor, but you can shrink what you rent from them. Sign-in is now self-hosted at /login, talking to the auth API directly; the hosted portal is out of the critical path entirely. Their pages are still broken as I write this — and my users can no longer tell.

Lesson: every vendor page in your critical path is an outage you haven't scheduled yet.

Paying for replies nobody ever saw

The AI coach started returning empty replies — while still logging cost for each one. The gateway logs showed the failing calls stopping at exactly 1024 output tokens, the default budget. All of it "reasoning". Zero visible text.

Root cause: the model provider had quietly moved this model to different serving infrastructure where extended thinking is on by default. Thinking tokens count against the output budget, so a hard-thinking turn spent the entire reply budget before producing a single visible word. Same model name, same API, new behavior.

The fix is one line — request no reasoning effort for coach turns — plus an honest in-band message if a turn ever ends textless again, instead of a silent "(no reply)".

Lesson: a model behind an API can change under you without the API changing. Pin what you can, log token breakdowns, and treat "same slug" as a rumor, not a contract.

The guardrails blocked our own security lesson

I enabled LLM guardrails on the AI gateway in block mode. First thing they blocked: our own curriculum. A quiz question from the prompt-injection lesson reads exactly like a prompt injection — because it teaches one. HTTP 424, "prompt blocked". The same question passes untouched in flag mode.

A safety classifier can't tell teaching an attack from attempting one, and this platform teaches attacks so engineers can defend against them. So the rule here is now flag, never block: full visibility on what would have been stopped, no false positive in front of a learner.

Testing this had its own trap: the gateway caches responses, so re-sending an identical prompt after a config change shows you the cached verdict, not the new one. Vary the prompt or you're testing the cache.

Lesson: security controls that don't understand your domain will protect you from your own product.

Live, with 14 modules and no CI

aiarch.dev went to production today: 14 modules, 85 quiz items, lead capture, and a deliberate absence of CI. Pushing code deploys nothing here — production changes only when I run one deploy command, on purpose, while the platform is small enough that I want a human finger on every release.

First commit was two weeks ago. The spine — spaced repetition, mastery tracking, grading, the bounded coach loop — was working locally on day one; the fortnight went into content, provenance, and the boring parts that make it a product instead of a demo. By the end of launch day, 22 modules were published.

Lesson: the thin end-to-end slice beats the polished fragment. Everything above is already the second version of something that shipped smaller.