AI evals in practice: why changing the model wasn’t enough

How do you tell whether an AI feature is helping someone think, or simply rearranging what they’ve told it?

I kept coming back to that question while building the demo. The responses could be neatly structured, accepted by the application and broadly relevant, yet still leave me wondering what I was supposed to do next.

We eventually changed models, moving from Kimi K2.6 to GLM 5.3 Flash. But all three configurations in our controlled comparison failed the initial qualification criteria. Getting from that result to a beta I was willing to put in front of people took more than choosing the most promising candidate.

What I’m building, and where it is today

The AI Guided Roadmap is one of the experiments in my website’s Playground. I’m building it to help people work through customer pain points, gather evidence and decide which opportunities deserve a place on a product roadmap.

That starts with understanding who is struggling and what they need to achieve. I want the AI to help examine the evidence and challenge assumptions before we settle on something to build.

The first stage is now a public discovery beta. You describe a business problem, the guide offers its reading of the situation and an approach worth exploring, then asks a question intended to sharpen the next decision. A working brief sits alongside the conversation for you to review and correct.

The wider prioritisation and roadmap stages are still ahead. Today’s demo is about whether that initial discovery conversation can help you reach a clearer, more useful starting point. It does not connect to your CRM or carry out the workflow it suggests.

For example, if sales proposals feel generic, a useful guide should help establish whether the difficulty lies in customer discovery, access to relevant material, or how the proposal is assembled. Those possibilities lead to different interventions. Jumping straight to an AI proposal writer would leave the most important assumption untested.

So checking whether the model returned valid output would only get us so far.

What would a useful conversation actually do?

For this application, I wanted the guide to interpret the problem, suggest a plausible direction and ask a question whose answer could change that direction. It needed to keep supplied information separate from hypotheses and adapt when the visitor corrected something.

An invented benefit, assumed integration or unsupported permission could make an otherwise convincing answer misleading. Equally, repeating the visitor’s words and marking everything else as unknown would offer little help.

I used Codex to help implement the application and run the evaluation workflow, then reviewed outputs and tested the interface myself. The cases were synthetic and separate from private customer or employment material. These were development tests with my review as the product owner. They weren’t an independent benchmark or an external user study.

The AI evals were structured tests of the behaviour we needed: a sound answer and a conversation that helped the visitor make progress.

Trying a different model was an obvious next step

Earlier development runs had already shown how valid responses could contain generic advice or unsupported assumptions. Kimi became a working candidate and stayed in place through several revisions to the instructions and interaction design. It had not passed the full product requirements at that point.

For the controlled comparison, we froze the configurations, synthetic cases and scoring rules before making inference calls. The models weren’t given the expected answers. We kept their original responses alongside the processed output, usage and reported reasons for stopping. That let us follow an answer through the application when something went wrong.

We compared three configurations:

By “contract”, I mean what the application asks the model to return and the rules it uses to accept, reject or transform that response.

Because the settings differed, we couldn’t attribute every difference to the model alone. We also changed too much at once in the revised Kimi configuration to isolate the effect of removing a schema constraint. The comparison could help us choose where to spend development effort, but it couldn’t settle those questions.

The cases included follow-up questions, corrections, uncertain information and instructions to stop. A conversation could fail because the system mishandled a later correction even if its first answer sounded convincing.

All three configurations failed the first screening

The run made 17 screening calls and one compatibility probe. Some failed responses blocked dependent turns, so the candidates did not complete equal numbers of cases. No candidate qualified to advance to the additional cases reserved for a later validation stage.

The screening results were:

Full compliance covered meaning, language and adherence to the interaction rules. It was more demanding than producing valid JSON. The counts describe these cases under these configurations; they are not estimates of general model accuracy.

GLM produced complete language, supported one useful three-turn journey and needed less output filtering in the recorded sample. That was enough to make further work worthwhile. It still mishandled a correction, made a misleading association about permission and asked questions that broke our interaction rules.

The revised Kimi contract produced complete English in all four calls, but every response exceeded local length limits and was rejected. An application that discards everything may prevent a bad suggestion from reaching the screen, while also providing no useful assistance. We needed to count the lost usefulness as well as the rejected output.

At this point, the conclusion was still “none qualifies”.

Sometimes the application made a useful answer worse

The incomplete wording from the Kimi baseline needed a closer look. The fragments were already present in stored provider output. They had not been introduced by the interface cutting a string short.

Fresh responses reported a normal stop, with no indication that they had exhausted their output-token allowance. We could narrow the diagnosis, but we still couldn’t establish the provider’s internal cause. Several fields were tightly constrained, and changing the surrounding contract changed the symptoms without producing an acceptable configuration.

We also found the reverse problem: application logic could take a useful model question and make the experience worse. Grounding rules and conservative filtering sometimes removed guidance, leaving the visitor with copied facts and little progress.

I found it useful to separate the checks:

• Provider response: did the request succeed and return usable output?
• Raw meaning: were facts supported, uncertainty preserved and suggestions relevant?
• Application processing: what was rejected, filtered or replaced, and why?
• Saved state: did corrections and restrictions survive the update?
• Visitor progress: did the interaction leave the person better able to make the next decision?

For example, preserving a visitor’s work after a failed response is useful recovery, but it tells us little about the model’s answer. We also had to check whether a cited source actually supported the claim, rather than just checking that the source existed.

If those outcomes are compressed into one pass rate, it becomes difficult to know what to fix.

Then I tried it with a problem I recognised

After further GLM work, I tried a business problem about sales proposals: they were too generic, did not reflect the prospect’s challenges and made poor use of existing service and value material.

The response largely moved fragments into planning fields. I had already named the people involved, yet some of that information was still treated as unknown. Technical labels and unresolved fields dominated the interface, and the questions did little to advance the problem.

It felt more like filling in a form with an assistant than working through the problem with one. GLM looked more promising in the tests, but the experience still wasn’t doing what I wanted.

We brought the conversation to the foreground: here is my reading of the problem, here is an approach worth exploring, and here is the question that would help decide whether it makes sense. The working brief remained available for inspection and correction, with hypotheses clearly distinguished from supplied facts.

For the proposal scenario, a useful question was where prospect challenges and desired outcomes were currently recorded. If good discovery existed in a CRM, the next step could explore matching that evidence to relevant service material. If discovery was sparse or mainly verbal, the more immediate need could be improving information capture before generating a proposal.

Changing that answer gave us a practical way to test whether the guidance adapted. Broadly relevant sales advice would have sounded plausible in either case. A separate repair-café scenario checked that the approach extended beyond the proposal example.

The same GLM model remained in place through this redesign. We also adjusted rules that were rejecting useful questions simply because they contained closely related supporting parts. We still needed to distinguish unknown permission from an explicit prohibition.

The later refinement recorded seven useful controlled turns, with five raw full-compliance passes and remaining caveats disclosed. When I tried the preview, it was good enough for the initial beta I had in mind. That didn’t mean it had passed every earlier qualification gate.

What about cost and speed?

The controlled comparison reported roughly $0.035 in estimated incremental inference cost across all 18 attempts, including the compatibility probe and failed or rejected outputs. That is a historical estimate for this small run. It excludes development effort, coding-assistant usage, other experiments and hosting, and was not independently reconciled to an invoice.

GLM’s estimated inference cost looked attractive for this workload. But with different call counts and response lengths, and some conversations blocked before completion, I couldn’t claim a meaningful percentage saving for equivalent successful journeys.

The same caution applies to speed. Provider-request timings did not measure the full browser experience or the visitor’s effort. A quick response that requires a correction, repeated explanation and another generation may create more work overall.

For the next evaluation, I would measure cost and elapsed time across complete journeys, alongside useful completion, repair effort and failure severity. Token cost is one input to that decision.

What I would carry into the next evaluation

I would begin with realistic two- or three-turn conversations and define what useful progress looks like before committing to a detailed output structure. The proposal example should have appeared earlier in development.

I would also distinguish material failures from interaction preferences. Unsupported factual claims, implied authority and lost corrections have different consequences from a focused question containing two related parts. When the criteria change, record why and evaluate the revised product against them. Keep the original results intact.

For a subsequent comparison, I would hold the application path as stable as possible, document unavoidable provider differences and advance viable candidates to comparable cases kept aside from tuning. Our original held-out stage remained unrun. Codex also saw the test cases while designing and reviewing the experiment, so they were not independently blinded test data.

I’d keep checking the answer before and after application processing, particularly when changing prompts, filters or the way corrections are saved.

In When “we need AI” lands on the roadmap, I argued for making the evidence required for the next commitment explicit. Here, the initial comparison gave us enough evidence to continue investigating GLM. Integration work and trying the revised experience myself came before the beta decision. Broader reliability still needs more testing.

The next time an AI response disappoints me, I’ll still ask whether another model might help. I’ll also look at the question we asked, the constraints we imposed and what the application did with the reply.

Evidence note: this account draws on the project’s recorded September 2026 evaluation reports and model-selection retrospective. Results use synthetic development scenarios and owner review. The original request/response artifacts have not been independently replayed for this article, and the figures should not be read as a general model benchmark.

Give the discovery beta a go

I’d like to hear how the conversation works for you. Bring a fictional or non-confidential version of a workflow or product problem, answer the guide’s question, then try correcting one of its assumptions.

Does the next question help you think? Does the approach change when you provide different information? Do you leave with a clearer brief, or find yourself doing most of the work?

Try the AI Guided Roadmap discovery beta in the Playground.

It’s an early beta, so please tell me where you get stuck as well as what works. Leave a comment on my LinkedIn post or send me feedback directly. I’m particularly interested in any point where you had to repeat yourself, correct the same assumption twice, or found a question genuinely useful.