Every customer-facing agent runs a loop. Call the model with the user's message, decide which tool to use and with what input, run it, read the output, decide the next step, and keep going until the request is fulfilled. The tool that decides answer quality most often is knowledge retrieval, so evaluating an agent starts with evaluating what it retrieves. That is the subject here.

Good retrieval gets defined on three axes: accuracy, relevance, and completeness. Accuracy is not really the variable, because a retriever returns text taken directly from your own documents, so it is as correct as they are. Relevance and completeness are the variables, and they formalize into precision and recall by comparing what the retriever returned against the ground truth snippets that actually hold the answer. The problem is how ground truth is usually stored. Label it as a list of chunk IDs and it dies the moment you change chunking strategy, because chunk three now points at completely different text, and testing each new setup means rebuilding the whole dataset by hand. Vinit walks through the alternative, drawn from published research on evaluating chunking strategies: anchor ground truth to character spans in the document itself, a start index and an end index, like a highlighter dragged across the page. Any retriever that reports where its chunks start and end can then be scored against that same dataset.

Build the golden dataset once, then test as many retrieval setups as you want and let the numbers pick the winner. Watch this if you are choosing between retrieval configurations and want a measurement instead of an impression, and particularly if you work somewhere that an incomplete answer is a compliance problem rather than just a bad experience.