https://jackmonte.com/posts/Recent content in Posts on Jack Monte — AI EngineerHugoen-usThu, 21 May 2026 00:00:00 +0000https://jackmonte.com/posts/eval-harness-rag-pipeline/Thu, 21 May 2026 00:00:00 +0000https://jackmonte.com/posts/eval-harness-rag-pipeline/<p>Most RAG pipelines ship without any systematic way to measure whether they&rsquo;re actually working. You run a few manual queries, the answers look reasonable, and you move on. The problem is that &ldquo;looks reasonable&rdquo; doesn&rsquo;t tell you where the system is failing, how often it fails, or whether it would fail on the queries your users actually send.</p> <p>That was the state of my rag-starter pipeline at the end of Week 4. It could answer questions grounded in a document corpus, and it correctly said &ldquo;I don&rsquo;t know&rdquo; when context was insufficient. But I had no numbers. I didn&rsquo;t know if the retrieval was finding the right chunks, whether Claude was staying faithful to the context, or whether the answers were actually addressing the questions asked.</p>