Deployment, Scaling & Cost/Latency Optimization

84 lessons in AI Engineering / Working with LLMs

• Measuring Time-to-First-TokenCode Quiz
• Cost Estimation for LLM WorkloadsSlides / Video
• Estimating Cost from Token UsageCode Quiz
• Tracking Per-Request Cost MetricsCode Quiz
• Using Prompt Caching for ContextCode Quiz
• Handling 429 Rate Limit ErrorsCode Quiz
• Truncating Conversation HistoryCode Quiz
• Prompt Compression to Reduce TokensCode Quiz
• Response Cache for Repeated PromptsCode Quiz
• Reducing Latency with StreamingCode Quiz
• Semantic Cache with Embedding SimilarityCode Quiz
• Choosing Embedding Batch SizeCode Quiz
• Load Balancing Across API KeysCode Quiz
• Model Routing for Simpler TasksCode Quiz
• Retry with Exponential BackoffCode Quiz
• Reusing the HTTP ClientCode Quiz
• Setting max_tokens to Control CostCode Quiz
• Deploying an LLM EndpointCode Quiz
• Concurrent Async Requests for ThroughputCode Quiz
• Fallback to Alternate ModelCode Quiz
• Token Bucket Rate LimitingCode Quiz
• Batching Requests to Reduce CostCode Quiz
• Adding Request TimeoutsCode Quiz
• Reducing Cost via Prompt CompressionQuiz
• Load Balancing Across EndpointsQuiz
• Cost/Latency ObservabilityQuiz
• Model Size vs Cost TradeoffQuiz
• Output Token Length ImpactQuiz
• GPU Requirements for Self-HostingQuiz
• Prompt Length ImpactQuiz
• Throughput and Tokens-per-SecondQuiz
• Choosing a Cheaper ModelQuiz
• TTFT vs Total LatencyQuiz
• Estimating Token Costs at ScaleQuiz
• Handling Rate LimitsQuiz
• Timeouts and Graceful DegradationQuiz
• Provisioned vs On-Demand PricingQuiz
• Fallback and Model RoutingQuiz
• Autoscaling LLM WorkloadsQuiz
• Inference Servers OverviewQuiz
• Quantization for Cheaper InferenceQuiz
• Self-Hosting vs Managed APIQuiz
• Concurrency and Parallel RequestsQuiz
• Batching for ThroughputQuiz
• Semantic Response CachingQuiz
• Prompt CachingQuiz
• Monitoring and Observability for LLM SystemsSlides / Video
• Retry, Timeout & Fallback StrategiesSlides / Video
• Quantization for Faster, Cheaper InferenceSlides / Video
• Model Cascading and RoutingSlides / Video
• Semantic Caching for Similar QueriesSlides / Video
• Caching Responses and Prompt CachingSlides / Video
• Prompt Compression to Cut TokensSlides / Video
• Choosing Smaller Models to Cut CostsSlides / Video
• Autoscaling Based on Traffic DemandSlides / Video
• Horizontal Scaling & Load Balancing ReplicasSlides / Video
• Batching Requests to Improve ThroughputSlides / Video
• Throughput vs Latency Trade-offsSlides / Video
• Token Optimization for CostFlashcard
• Speculative DecodingFlashcard
• Provisioned Throughput vs Pay-Per-TokenFlashcard
• Inference ServersFlashcard
• Batching and Continuous BatchingFlashcard
• Cost/Latency/Quality TriangleFlashcard
• Autoscaling for Traffic SpikesFlashcard
• Self-Hosting vs Managed APIFlashcard
• Context Window Size ImpactFlashcard
• QuantizationFlashcard
• Monitoring and ObservabilityFlashcard
• Output Length ControlFlashcard
• Horizontal Scaling and Load BalancingFlashcard
• GPU SelectionFlashcard
• KV CacheFlashcard
• Prompt CachingFlashcard
• Semantic CachingFlashcard
• Throughput MetricsFlashcard
• Model Selection for Cost/LatencyFlashcard
• Cold Starts and Warm PoolsFlashcard
• Model Cascading and RoutingFlashcard
• Key Latency MetricsFlashcard
• TTFT and Tokens Per SecondSlides / Video
• GPU Selection for LLM ServingSlides / Video
• Model Serving Infrastructure & Inference ServersSlides / Video
• Hosted API vs Self-Hosted ModelsSlides / Video