AI Engineering / Working with LLMs
Observability, Monitoring & LLMOps
76 lessons in AI Engineering / Working with LLMs
- Counting API Error RatesCode Quiz
- Debugging Failed LLM GenerationsSlides / Video
- Why LLM Observability Is HardSlides / Video
- What LLMOps Is vs MLOpsSlides / Video
- Session and Conversation-Level ObservabilitySlides / Video
- Privacy and PII in LLM LogsSlides / Video
- Prompt-to-Production Feedback LoopSlides / Video
- Eval Datasets from Production TracesSlides / Video
- A/B Testing Prompts and ModelsSlides / Video
- Prompt Versioning and Experiment TrackingSlides / Video
- Detecting Model or Prompt DriftQuiz
- Capturing User Feedback SignalsQuiz
- Correlating Traces with FeedbackQuiz
- PII Handling in LogsQuiz
- A/B Testing Prompts and ModelsQuiz
- Online vs Offline EvaluationQuiz
- Observability vs MonitoringQuiz
- Spans and Trace StructureQuiz
- LLM Observability ToolsQuiz
- Production Quality MonitoringQuiz
- Prompt Versioning and ManagementQuiz
- Sampling and Trace RetentionQuiz
- Cost AttributionQuiz
- Golden Datasets for EvaluationQuiz
- CI/CD for LLM ApplicationsQuiz
- Monitoring Hallucination RateQuiz
- Dashboards for LLM MetricsQuiz
- Alerting on AnomaliesQuiz
- What LLMOps IsQuiz
- LLM Observability Platforms and ToolsSlides / Video
- Dashboards for LLM MetricsSlides / Video
- Alerting and Anomaly Detection for LLMsSlides / Video
- Detecting Drift in LLM SystemsSlides / Video
- Detecting Hallucinations in ProductionSlides / Video
- Capturing User Feedback SignalsSlides / Video
- LLM-as-a-Judge Quality ScoringSlides / Video
- Online vs Offline EvaluationSlides / Video
- Quality and Output Monitoring in ProductionSlides / Video
- Token Usage and Cost TrackingSlides / Video
- Logging Prompts, Completions, and MetadataSlides / Video
- Three Pillars of LLM ObservabilitySlides / Video
- Logging Prompt Template VersionCode Quiz
- Measuring Token ThroughputCode Quiz
- Automated Eval Scoring on Logged OutputsCode Quiz
- Counter and Histogram for LLM MetricsCode Quiz
- Sampling a Fraction of TrafficCode Quiz
- Alerting on Latency/Cost ThresholdsCode Quiz
- Attaching Session ID to RequestsCode Quiz
- Capturing Tool Call Events in TracesCode Quiz
- Logging Cache Hit vs MissCode Quiz
- Logging Thumbs Up/Down FeedbackCode Quiz
- Instrumenting Spans for a ChainCode Quiz
- Recording Model MetadataCode Quiz
- Wrapping Calls with a DecoratorCode Quiz
- Structured Logging with JSON FieldsCode Quiz
- Logging LLM Request and ResponseCode Quiz
- Measuring End-to-End LatencyCode Quiz
- Correlation ID Across StepsCode Quiz
- A/B Testing Prompts and ModelsFlashcard
- Alerting, Dashboards, and SLOsFlashcard
- CI/CD for Prompts and LLM AppsFlashcard
- Prompt and Model VersioningFlashcard
- Sampling and PII in LogsFlashcard
- Evaluation-Driven Feedback LoopsFlashcard
- Tracing Requests and SpansFlashcard
- Cost Tracking and Budget AlertingFlashcard
- Detecting Model, Data, and Prompt DriftFlashcard
- Online vs Offline EvaluationFlashcard
- LLM Observability Platforms and ToolsFlashcard
- What LLMOps Is vs MLOpsFlashcard
- Distributed Tracing for Chains and AgentsFlashcard
- Detecting Hallucinations and RegressionsFlashcard
- Capturing User Feedback SignalsFlashcard
- Logging Prompts, Completions, MetadataFlashcard
- Error Rate and Failure MonitoringFlashcard
- LLM Observability LifecycleFlashcard