[D] How to evaluate RAG - both retrieval and generation, when all I have is a set of PDF documents?

Say I have 1000 PDF docs that I use as input to a RAG Pipeline.

I want to to evaluate different steps of the RAG pipeline so that I can measure:
- Which embedding models work better for my data?
- Which rerankers work and are they required?
- Which LLMs give the most factual and coherent answers?

How do I evaluate these steps of the pipeline?

Based on my research, I found that most frameworks require labels for both …

data documents embedding embedding models machinelearning pdf pipeline rag retrieval set work