I Built an Eval Harness for My RAG Pipeline. Here's What the Numbers Revealed.
Most RAG pipelines ship without any systematic way to measure whether they’re actually working. You run a few manual queries, the answers look reasonable, and you move on. The problem is that “looks reasonable” doesn’t tell you where the system is failing, how often it fails, or whether it would fail on the queries your users actually send. That was the state of my rag-starter pipeline at the end of Week 4. It could answer questions grounded in a document corpus, and it correctly said “I don’t know” when context was insufficient. But I had no numbers. I didn’t know if the retrieval was finding the right chunks, whether Claude was staying faithful to the context, or whether the answers were actually addressing the questions asked. ...