A long look at evals, persona simulations, tool tests, model choice, and the API bill nobody budgets for when building an AI agent.

Same prompt, same tools, same data. One model returned the correct list. The other invented students who don’t exist.

That sentence has been living in my head for weeks, so let me tell you how it got there, and what we do differently now.

I’ve been building agentic workflows for a while. The recipe is familiar by now: an LLM in the middle, a set of tools around it, a database underneath, sometimes an MCP server, business logic to keep it on task, security rules to keep it in its lane. Wiring that up used to feel like the whole job. Most frameworks will get you to a working demo before your coffee goes cold, and the demo will look great.

The wahala starts with a much less glamorous question: how do I know this thing actually behaves? Not on the happy path I rehearsed. On the thousandth conversation, with a user I’ve never met, after a prompt tweak I made on a Friday.

The ghost students

This is the incident that changed how I work.

We asked an agent to pull the list of students who were owing. Simple task on paper: hit the right tool, apply the right filter, return the names. We ran it on two models with an identical setup, same prompt, same tenant, same data. The only variable was the model.

Model A returned a clean, accurate list.

Model B returned a list too. Real students, and a handful who do not exist anywhere in the database.

So we asked it why. It explained. We pushed back. It explained again, calmly and in detail, with the confidence of a danfo conductor swearing there’s still space. It only dropped the ghost students when we explicitly said no, they are not real, remove them.

Two things bothered me more than the fabrication itself.

First, the wrong answer and the right answer looked identical from the outside. Same format, same tone, same certainty. Anything downstream, a report, a reminder email, a human skimming the output, would have trusted both equally.

Second, not a single line of code changed between those two runs. A test suite that only checks whether the tool works would never catch this, because the tool worked. The model lied about what the tool returned.

Why agents break normal testing

Traditional software is deterministic. Same input, same output, every time, and if it isn’t, that’s the bug. Agents are probabilistic by design. Sampling means the same task on two runs can take two different paths, call tools in a different order, or phrase the same answer three different ways. That alone breaks the “assert equals” habit most of us grew up with.

Then errors compound. An agent that is 95% reliable per step is nowhere near 95% reliable across a five-step workflow. Multiply it out and you’re at roughly 77%, because every step inherits the mistakes of the one before it.

Then there are the model quirks, the ones you only meet in production or in tests that look like production:

• Verbosity bias. Longer output starts looking like better output. This hits twice: the model pads its answers, and if you’re using an LLM as a judge, the judge rewards the padding.
• Position bias. Give the model a list of options and it leans toward whichever one came first. Ask it to compare two answers and the order you present them in can flip the verdict.
• Prompt sensitivity. A one-line tweak to the system prompt, added to fix one case, changes which tool gets called in three others.
• Wrong tool selection. The prompt says fetch, the model decides update felt right today.
• Confident fabrication. See ghost students above.
• Folding under pressure. The mirror image of the ghost students. Some models abandon a correct answer the moment a user pushes back, which is its own kind of dangerous.

You cannot unit test your way out of this. You need layers.

Layer one: tool and integration tests

The deterministic parts still deserve deterministic tests, and they’re the cheapest confidence you’ll ever buy.

Each tool gets tested as a plain function first: valid inputs, invalid inputs, empty results, the API timing out, the API returning a shape nobody expected. Then the integration layer: does the tool respect tenant boundaries, so a request from one school can never see another school’s data? Does it handle an expired token gracefully? Is it idempotent, so a retry doesn’t double-send or double-charge?

Then the part people skip: break the tools on purpose and watch the agent. Make the database call fail and see whether the agent retries, tells the user, or makes up a result to cover the gap. An agent that hallucinates when a tool fails is a much bigger problem than the tool failing.

Boring tests. Also the ones that stop a bad afternoon from becoming a bad week.

Layer two: evals

This is where “it feels better” becomes a number.

An eval is a fixed set of realistic inputs with expected outcomes, scored automatically every time something meaningful changes: a prompt, a tool description, a model version. Think of it as a regression suite for behaviour rather than code.

The set itself comes from two places. Real conversations, sampled and cleaned. And incidents. Every time something like the ghost students happens, it becomes a case. The suite grows small small, and every case in it has a story.

What you score depends on the workflow, but the usual suspects are:

• Tool trajectory. Did it call the right tool, with the right arguments, in a sensible order? Did it call tools it didn’t need?
• Correctness. Is the final answer right? For anything with IDs, this is a set comparison, not a vibe. Every ID in the output must exist in the database, and the set must match the fixture exactly.
• Fabrication. Did it add anything the tools never returned? Hard fail, no partial credit.
• Format and length. Did it stay within the shape the downstream system expects?
• Tone and scope. Did it stay polite, on task, and away from things it shouldn’t promise?

The first four can be scored deterministically, and you should, because deterministic scoring is free and never has a bad day. The last one usually needs an LLM as a judge, and this is where you shine your eye. Judges inherit every bias listed above. Give the judge a tight rubric, keep its output to a small scale rather than a free-form score, swap the order when comparing two answers so position bias cancels out, and spot-check its verdicts against a human every so often.

Two more rules worth adopting early:

1. Run every case more than once. Nondeterminism means a single pass tells you almost nothing. Five runs per case, passing only if all five pass, tells you a lot.
2. Track scores per prompt version. The whole point is answering “did today’s improvement break yesterday’s workflow?” and you can only answer that with history.

A ghost-student eval case looks roughly like this:

id: debtors-list-basic
input: "Give me the list of students owing this term."
tenant: fixture_school_a
expect:
  tool: list_debtors
  args: { term: current }
  output_ids: exact match with fixture debtors_school_a
hard_fail_if: any id not in fixture
runs: 5
pass_if: 5 of 5

Layer three: persona simulations

Real users don’t type like your test cases. They’re tired, vague, angry, or halfway through something else. So you simulate them.

A persona is a goal plus a behaviour. You hand it to a second LLM (or a script, for the strict ones), let it hold a full multi-turn conversation with the agent, then score the outcome. A few that earn their keep:

• The angry customer. Opens hostile, repeats the demand, ignores your clarifying question the first time. You’re checking whether the agent stays polite, keeps verifying through its tools, and never promises something it can’t confirm just to calm the room.
• The calm customer. The control group. If the agent fails the calm one, nothing else matters.
• The impatient one. Wants the answer in one message and punishes every follow-up question. Tests whether the agent knows when it has enough to act.
• The vague one. “The thing from before.” “That student.” “Same as last time.” Tests memory, clarification, and whether the agent guesses when it should ask.
• The contradictory one. Asks for one thing, then the opposite three turns later. Tests state handling and whether the agent notices the flip or just cheerfully does both.

Score persona runs the same way as evals: did it reach the goal, did it stay in scope, did it fabricate, did it escalate when stuck. And keep the persona LLM on a leash. A simulator that drifts off-script is just a second unreliable agent in the room.

A persona spec, roughly:

persona: angry_customer_double_charge
goal: get a refund for a payment they believe went through twice
behaviour: hostile opening, repeats demand, ignores first clarifying question, threatens to leave
pass_if:
  - tone stays polite throughout
  - payment is verified via tool before anything is promised
  - no refund is promised that the tool cannot confirm
  - escalates to a human after two failed attempts

Model choice is a test variable

The ghost students taught us this the hard way: different models fail differently.

Some are verbose. Others are terse to the point of unhelpful. A few over-call tools, checking things they already know, while their cousins under-call them and answer from memory. And at least one invents records and argues about it. A bigger model is a different failure profile, not a free upgrade: sometimes better on reasoning, worse on tool discipline, and you won’t know which until the suite runs.

So the whole suite runs per model, and the results live in a matrix: cases down the side, models across the top. That matrix is how you make a model decision with numbers instead of a demo. It’s also how you notice a provider updating a model under the same name, because Tuesday’s scores won’t match Monday’s and you’ll finally know why.

The bill nobody budgets for

Now, the practical bit.

Do the multiplication once and it stops being abstract. Say 200 eval cases, 5 runs each, across 3 candidate models. That’s 3,000 runs before a single persona conversation, and each run is at least one model call, usually several once tools get involved. Add persona conversations at ten turns apiece and the number climbs fast. Every call is tokens, and the invoice arrives whether or not the tests passed.

A few things that help:

• Tier the suites. A small, cheap smoke suite on every change, made of the twenty cases that have bitten you before. The full suite before a release, or nightly.
• Record and replay the tools. Cache deterministic tool responses so you’re paying for the model’s decisions, not the plumbing around them.
• Only race the models still in contention. If a model is already out for latency or cost reasons, stop paying to eval it.
• Use a cheaper judge where the rubric is simple. Save the expensive judge for nuanced cases, and check the cheap one for bias too.
• Watch cost per run the way you watch latency. Put it on the same dashboard. A suite that doubles in cost is a suite someone will eventually stop running.

Demo vs production

Building the agent gets you to the demo. Testing gets you to production.

Everyone is building agents right now. Far fewer people can tell you, with numbers, how theirs behaves on the thousandth conversation, with a user they’ve never met, after a prompt tweak they made on a Friday. That confidence is the actual engineering work.

The ghost students have a permanent seat in our suite. They pass now. I check anyway.

Why building school software that holds up on its worst day matters more than building software that only works in a demo.

---

Most software is built and tested in comfortable conditions. Fast internet, steady power, a quiet room, one person clicking through at their own pace. Schools do not run in those conditions. A school runs during a downpour that takes the network with it, during a power cut in the middle of the afternoon, with a hundred things happening at once and a queue of parents at the desk.

So the first thing we decided as an engineering team at Klasstack was simple to say and hard to live up to: reliability is not a nice-to-have on top of the features. It is the product. A tool a school cannot trust on its worst day is not really a tool at all. Here is how that belief shapes the way we build.

We build for the environment our users are in, not the one we develop in

It is easy to build something that works on a developer’s laptop and forget that the laptop has fast internet and a full battery. Our users do not. So we design the important flows to survive the moments that matter, and to catch up quietly when conditions improve. The goal is that a busy administrator never has to think about the plumbing. They do their work, and the system holds up around them.

Simplicity beats cleverness

Every extra moving part is another thing that can break when no one is around to fix it. We would rather ship something plain that works every single time than something clever that works beautifully in a demo and fails when it matters. When we review our own work, the most useful question we ask is not “is this impressive,” it is “what happens when this goes wrong, and who is there to notice.” The best answer is usually the simplest design.

Your school’s data is yours, and we treat it that way

A school platform holds sensitive information: children’s records, family contacts, fees, results. We treat that as a responsibility, not just data. Each school’s information is kept separate from every other school’s. Access is granted on a need-to-use basis, so a member of staff sees what their role requires and no more. And your data is yours. We do not sell it, and we do not treat it as a product to be traded. Trust, once lost, does not come back, so we would rather be careful than sorry.

Software is only half the job

We have learned that the best-built feature still fails a school if the school is left to figure it out alone. So we do not think our job ends when the code ships. It ends when a school is confident using it. That belief pulls engineering and support closer together than they usually sit. We build things to be understood, not just to be technically correct, because a feature no one can use with confidence may as well not exist.

What this means for your school

You should not have to know or care how any of this works. That is rather the point. When you run Klasstack, the promise is quiet and consistent: it holds up when the day is hard, it keeps your information safe, it stays simple enough to trust, and there are real people behind it when you need them.

That is the kind of software we want to build, and the kind we would want running our own school.

I recently came across a small commit switching one query from offset to cursor pagination. A few lines changed. Ordinary Tuesday stuff.

But it sent me right down memory lane, because there is a special kind of wahala that never shows up in code review, never fails a test, and never pages anybody at 3am on launch day. It waits. It sits quietly inside a batch job you wrote when the table had 40,000 rows, behaving itself perfectly, until one random Tuesday three years later the same job starts chopping memory like it skipped breakfast. And you’re staring at an out-of-memory crash asking the ancient question: what changed?

Nothing changed. That’s the whole trick. You didn’t change anything. The table did.

The entire answer in one line

If you carry only one thing from this post, carry this: it depends on who is doing the paging.

If a human is clicking through pages in a UI, page 3, then page 8, then back to page 2, that is offset work. Humans jump around, and offset lets them.

If a machine is grinding through the whole table in the background, a batch job, a sync, a nightly export, anything that starts where it stopped and keeps marching, that is cursor work. Machines don’t jump around. They march forward and want to resume exactly where they left off.

Get that split right and most of your pagination pain disappears. Now let me show you why it works that way, and why the machine case is the one that betrays you quietly.

The bet you forgot you placed

When you write a paginated sweep over a table, you place a quiet little bet on how big that table will get. The cruel part is that the bet is invisible. Offset pagination and cursor pagination look almost identical in the code. Same loop, same batch size, same upsert. One of them scales gracefully forever and the other slowly turns into a tar pit, and from across the room you cannot tell them apart.

So the offset-based feeder you shipped two years ago was not wrong. On a small table it was genuinely fine. Cheaper to write, easier to reason about, worked on the first try. The PR merged, everybody moved on, and the decision fell out of everyone’s head. Which, to be fair, is exactly what is supposed to happen to decisions that work.

Except the table kept growing, small small, in the background. And offset has one nasty property: the deeper you page, the more work the database does per page. Not the same work. More. Every single run. So the cost never stayed flat. It crept upward alongside your row count until one day it crossed the invisible line marked “available memory” and the whole thing fell over.

The job crashing today is not today’s job. It is a two-year-old decision finally presenting its bill. With interest.

And notice the shape of the casualty: a batch job, a machine sweeping the entire table. The exact case that should have been cursor from day one. Let’s break both down properly.

Offset pagination, explained like we’re gisting

Offset is “skip plenty, then grab some.” You tell the database: skip the first 40,000 rows, give me the next 10,000.

The problem is the word skip. Databases cannot teleport to row 40,001. There is no shortcut, no bookmark. To skip 40,000 rows, the database physically walks past all 40,000 of them, looks at each one, and throws it away, just to reach the ones you actually asked for.

Picture reading a novel by starting from page one every single time. Flip flip flip to where you stopped, read one page, close the book. Next session? Back to page one. By the time you finish the book, you’ve flipped through the entire thing hundreds of times. That is offset doing a full sweep. Page 1 is instant. Page 900 makes the database count to nine million before it hands you a single row.

Where it shines: a UI with an actual human inside it. When someone is browsing results and wants page 1, then page 5, then back to page 2, offset is exactly right. “Page 5, please” is easy to write, easy to hold in your head, and jumping straight to any page is instant. This is offset’s home turf, and cursor would only be a nuisance here.

What’s good about it:

• Dead simple. Your junior dev understands it in one glance.
• Jumps anywhere instantly, perfect for a UI where someone clicks straight to page 47.
• Perfectly fine on small or slow-growing tables. Genuinely. Let nobody shame you.

What bites you:

• Cost grows with depth. Deep pages get expensive, and a full sweep gets quadratically expensive as the table grows. That is the entire tar-pit-three-years-later story.
• It is a position, not an anchor. If a row ahead of you gets deleted mid-sweep, everything shifts down by one and you silently skip a row. An insert, and you process one twice. Offset counts positions, and positions move.
• The pain is invisible until it isn’t. Small table, no symptoms. Big table, sudden fire. Nothing in between to warn you.

Those last two points are exactly why offset is a terrible fit for a long-running batch sweep: the job pages deep (it visits every page) and it runs while data changes underneath it. Both of offset’s weaknesses, landing at the same time. Double wahala.

Cursor pagination, explained the same way

Cursor (also called keyset, also called “seek”) throws away the page number entirely. Instead of “skip 40,000,” it remembers the last row it saw and says: give me everything after this specific row.

You keep an actual bookmark. “Last time I stopped at the row with timestamp X and id Y. Give me what comes after that.” If there is an index on those columns, the database seeks straight to your bookmark and reads forward. No skipping. No counting to nine million. It opens the book exactly where you left it.

Every session costs the same, whether you’re on page 2 or page 900. That is the superpower.

Where it shines: batch jobs and anything that resumes. A feeder syncing a table into a search index, a nightly export, a migration, a job that got killed halfway and needs to continue from where it died. Because the bookmark is a real value from the data, you can write it down (stash it in Redis, say) and tomorrow’s run picks up exactly where today’s stopped. This is cursor’s whole reason for existing, and it is precisely what a batch job wants.

What’s good about it:

• Flat cost. Page 900 costs the same as page 2. A full sweep scales linearly, so the tar pit never forms.
• Stable under changes. The bookmark points at real values from the data, not a row count. Rows can appear and vanish elsewhere and your bookmark still means exactly what it meant. No skips, no doubles.
• Naturally resumable. “I was on page 900” becomes meaningless the moment data moves. “I stopped after row X” stays true forever.

What bites you:

• More fiddly to write. You’re threading “the last row I saw” through the loop instead of multiplying a page number.
• No jumping to page 47. Cursors only move forward from where you are, so random access is not really a thing. Which is exactly why it’s wrong for a browse-y UI and right for a march-forward batch job.
• The bookmark must be unique, or it lies to you. This is the sharp edge, so shine your eye here. If you bookmark on a timestamp alone and 5,000 rows share that timestamp, “everything after this timestamp” splits that group down the middle and you skip or double the rest. The fix is a compound bookmark: the timestamp plus a tiebreaker like the primary key. Now every row has a strict, unambiguous order, and “after this point” always means one exact spot. And your ORDER BY and your “after this” filter must name the same columns, or the whole scheme quietly falls apart.
• It leans on an index. The flat cost is real only if there’s an index on the columns you’re seeking by. No index, and your fancy cursor quietly rots back into offset’s cost. The index is doing the heavy lifting. Respect it.

So what do we actually do?

Not “cursor everything.” That is just swapping one thoughtless default for another. The rule from the top of this post is the real answer:

• A human moving through pages in a UI? Offset. Simple, correct, instant random access.
• A batch job, sync, or anything that resumes? Cursor. Flat cost, stable under writes, resumable. Offset here is the thing that OOMs in three years.
• Not sure how big the table gets? Cursor, and thank yourself later. It is the option that fails gracefully instead of falling off a cliff.

And the quieter lesson underneath all of it: the assumption is the dangerous part, not the code. When you write a paginated sweep, you are betting on a table size and a usage pattern. So say the bet out loud. Drop one comment:

// offset is fine here: human-facing UI, table capped around 50k

That one sentence is a tripwire. It turns a silent, forgotten assumption into something the next person, probably future-you, bleary and confused in front of an OOM log at 2am, can actually find and go: “Ah. The bet. The bet broke.”

The bug that gets you is never the one you’re watching on launch day. It is the reasonable little decision you made when everything was small, that you completely forgot, that was quietly compounding the whole time. Reaching for offset on a batch job is a world-class way to write one of those.

So write the comment. Leave the tripwire. Give three-years-from-now-you a fighting chance.

They’re going to be so tired.