Evaluation & Testing of LLM Outputs

15 lessons in AI Engineering / Working with LLMs

• Evaluating RAG QualitySlides / Video
• Evaluating RAG QualityQuiz
• Evaluating RAG QualityFlashcard
• Human Evaluation and AnnotationFlashcard
• Assertion-Based Unit Testing of OutputsFlashcard
• Building a Golden Evaluation SetFlashcard
• Pairwise vs Pointwise JudgingFlashcard
• Rubric-Based ScoringFlashcard
• LLM-as-a-Judge EvaluationFlashcard
• Overlap Metrics: BLEU, ROUGE, METEORFlashcard
• Exact Match and String MatchingFlashcard
• Reference-Based vs Reference-FreeFlashcard
• Why LLM Evaluation Is HardFlashcard
• Inter-Annotator AgreementFlashcard
• Semantic Similarity via EmbeddingsFlashcard