Testing

A managed eval tool adopted but not relied on, with a single item-018 score delta flagged as noise

The Eval Tool I Adopted but Won't Rely On

I migrated my eval harness to a managed tool, got it working, and then decided not to trust it as canonical. A single per-item delta on identical code is why.

The same eval scored two ways: a buggy run averaging 3.40 and the corrected run averaging 4.95

A Green Eval Was Lying to Me

Adding strict typing to my eval harness exposed an intermittent judge failure, and fixing that exposed a second bug a passing eval had been hiding. The lesson: a green eval is not a correct eval.