AI Engineering / Working with LLMs
Deployment, Scaling & Cost/Latency Optimization
84 lessons in AI Engineering / Working with LLMs
- Measuring Time-to-First-TokenCode Quiz
- Cost Estimation for LLM WorkloadsSlides / Video
- Setting max_tokens to Control CostCode Quiz
- Deploying an LLM EndpointCode Quiz
- Concurrent Async Requests for ThroughputCode Quiz
- Fallback to Alternate ModelCode Quiz
- Tracking Per-Request Cost MetricsCode Quiz
- Semantic Cache with Embedding SimilarityCode Quiz
- Choosing Embedding Batch SizeCode Quiz
- Using Prompt Caching for ContextCode Quiz
- Model Routing for Simpler TasksCode Quiz
- Handling 429 Rate Limit ErrorsCode Quiz
- Truncating Conversation HistoryCode Quiz
- Load Balancing Across API KeysCode Quiz
- Batching Requests to Reduce CostCode Quiz
- Estimating Cost from Token UsageCode Quiz
- Retry with Exponential BackoffCode Quiz
- Token Bucket Rate LimitingCode Quiz
- Reducing Latency with StreamingCode Quiz
- Response Cache for Repeated PromptsCode Quiz
- Prompt Compression to Reduce TokensCode Quiz
- Reusing the HTTP ClientCode Quiz
- Adding Request TimeoutsCode Quiz
- Cost/Latency ObservabilityQuiz
- Throughput and Tokens-per-SecondQuiz
- Prompt Length ImpactQuiz
- GPU Requirements for Self-HostingQuiz
- Output Token Length ImpactQuiz
- Model Size vs Cost TradeoffQuiz
- Handling Rate LimitsQuiz
- Prompt CachingQuiz
- Semantic Response CachingQuiz
- Timeouts and Graceful DegradationQuiz
- Provisioned vs On-Demand PricingQuiz
- Choosing a Cheaper ModelQuiz
- TTFT vs Total LatencyQuiz
- Fallback and Model RoutingQuiz
- Load Balancing Across EndpointsQuiz
- Autoscaling LLM WorkloadsQuiz
- Inference Servers OverviewQuiz
- Quantization for Cheaper InferenceQuiz
- Self-Hosting vs Managed APIQuiz
- Concurrency and Parallel RequestsQuiz
- Estimating Token Costs at ScaleQuiz
- Batching for ThroughputQuiz
- Reducing Cost via Prompt CompressionQuiz
- Monitoring and Observability for LLM SystemsSlides / Video
- Retry, Timeout & Fallback StrategiesSlides / Video
- Quantization for Faster, Cheaper InferenceSlides / Video
- Model Cascading and RoutingSlides / Video
- Semantic Caching for Similar QueriesSlides / Video
- Caching Responses and Prompt CachingSlides / Video
- Prompt Compression to Cut TokensSlides / Video
- Choosing Smaller Models to Cut CostsSlides / Video
- Autoscaling Based on Traffic DemandSlides / Video
- Horizontal Scaling & Load Balancing ReplicasSlides / Video
- Batching Requests to Improve ThroughputSlides / Video
- Throughput vs Latency Trade-offsSlides / Video
- KV CacheFlashcard
- Token Optimization for CostFlashcard
- Inference ServersFlashcard
- Monitoring and ObservabilityFlashcard
- Model Cascading and RoutingFlashcard
- Cold Starts and Warm PoolsFlashcard
- Speculative DecodingFlashcard
- Provisioned Throughput vs Pay-Per-TokenFlashcard
- Output Length ControlFlashcard
- Horizontal Scaling and Load BalancingFlashcard
- GPU SelectionFlashcard
- Prompt CachingFlashcard
- Throughput MetricsFlashcard
- Key Latency MetricsFlashcard
- QuantizationFlashcard
- Self-Hosting vs Managed APIFlashcard
- Autoscaling for Traffic SpikesFlashcard
- Cost/Latency/Quality TriangleFlashcard
- Model Selection for Cost/LatencyFlashcard
- Semantic CachingFlashcard
- Context Window Size ImpactFlashcard
- Batching and Continuous BatchingFlashcard
- TTFT and Tokens Per SecondSlides / Video
- GPU Selection for LLM ServingSlides / Video
- Model Serving Infrastructure & Inference ServersSlides / Video
- Hosted API vs Self-Hosted ModelsSlides / Video