AI Engineering / Working with LLMs
Observability, Monitoring & LLMOps
76 lessons in AI Engineering / Working with LLMs
- Counting API Error RatesCode Quiz
- Debugging Failed LLM GenerationsSlides / Video
- Why LLM Observability Is HardSlides / Video
- What LLMOps Is vs MLOpsSlides / Video
- Session and Conversation-Level ObservabilitySlides / Video
- Privacy and PII in LLM LogsSlides / Video
- Prompt-to-Production Feedback LoopSlides / Video
- Eval Datasets from Production TracesSlides / Video
- A/B Testing Prompts and ModelsSlides / Video
- Prompt Versioning and Experiment TrackingSlides / Video
- LLM Observability ToolsQuiz
- Spans and Trace StructureQuiz
- Correlating Traces with FeedbackQuiz
- Online vs Offline EvaluationQuiz
- Observability vs MonitoringQuiz
- Dashboards for LLM MetricsQuiz
- Monitoring Hallucination RateQuiz
- CI/CD for LLM ApplicationsQuiz
- Golden Datasets for EvaluationQuiz
- Prompt Versioning and ManagementQuiz
- Cost AttributionQuiz
- Capturing User Feedback SignalsQuiz
- What LLMOps IsQuiz
- Sampling and Trace RetentionQuiz
- PII Handling in LogsQuiz
- A/B Testing Prompts and ModelsQuiz
- Production Quality MonitoringQuiz
- Alerting on AnomaliesQuiz
- Detecting Model or Prompt DriftQuiz
- LLM Observability Platforms and ToolsSlides / Video
- Dashboards for LLM MetricsSlides / Video
- Alerting and Anomaly Detection for LLMsSlides / Video
- Detecting Drift in LLM SystemsSlides / Video
- Detecting Hallucinations in ProductionSlides / Video
- Capturing User Feedback SignalsSlides / Video
- LLM-as-a-Judge Quality ScoringSlides / Video
- Online vs Offline EvaluationSlides / Video
- Quality and Output Monitoring in ProductionSlides / Video
- Token Usage and Cost TrackingSlides / Video
- Logging Prompts, Completions, and MetadataSlides / Video
- Three Pillars of LLM ObservabilitySlides / Video
- Logging Prompt Template VersionCode Quiz
- Measuring Token ThroughputCode Quiz
- Logging Cache Hit vs MissCode Quiz
- Counter and Histogram for LLM MetricsCode Quiz
- Logging Thumbs Up/Down FeedbackCode Quiz
- Automated Eval Scoring on Logged OutputsCode Quiz
- Sampling a Fraction of TrafficCode Quiz
- Alerting on Latency/Cost ThresholdsCode Quiz
- Attaching Session ID to RequestsCode Quiz
- Capturing Tool Call Events in TracesCode Quiz
- Recording Model MetadataCode Quiz
- Wrapping Calls with a DecoratorCode Quiz
- Structured Logging with JSON FieldsCode Quiz
- Logging LLM Request and ResponseCode Quiz
- Measuring End-to-End LatencyCode Quiz
- Correlation ID Across StepsCode Quiz
- Instrumenting Spans for a ChainCode Quiz
- A/B Testing Prompts and ModelsFlashcard
- Evaluation-Driven Feedback LoopsFlashcard
- CI/CD for Prompts and LLM AppsFlashcard
- Alerting, Dashboards, and SLOsFlashcard
- Prompt and Model VersioningFlashcard
- Sampling and PII in LogsFlashcard
- Detecting Hallucinations and RegressionsFlashcard
- Tracing Requests and SpansFlashcard
- Logging Prompts, Completions, MetadataFlashcard
- Online vs Offline EvaluationFlashcard
- LLM Observability LifecycleFlashcard
- Error Rate and Failure MonitoringFlashcard
- Capturing User Feedback SignalsFlashcard
- LLM Observability Platforms and ToolsFlashcard
- What LLMOps Is vs MLOpsFlashcard
- Distributed Tracing for Chains and AgentsFlashcard
- Cost Tracking and Budget AlertingFlashcard
- Detecting Model, Data, and Prompt DriftFlashcard