Evaluation & Testing of LLM Outputs

15 lessons in AI Engineering / Working with LLMs

• Evaluating RAG QualitySlides / Video
• Evaluating RAG QualityQuiz
• Evaluating RAG QualityFlashcard
• Human Evaluation and AnnotationFlashcard
• Inter-Annotator AgreementFlashcard
• Reference-Based vs Reference-FreeFlashcard
• Why LLM Evaluation Is HardFlashcard
• Rubric-Based ScoringFlashcard
• Pairwise vs Pointwise JudgingFlashcard
• Assertion-Based Unit Testing of OutputsFlashcard
• LLM-as-a-Judge EvaluationFlashcard
• Semantic Similarity via EmbeddingsFlashcard
• Exact Match and String MatchingFlashcard
• Building a Golden Evaluation SetFlashcard
• Overlap Metrics: BLEU, ROUGE, METEORFlashcard