Local & Open-Source Model Deployment

67 lessons in AI Engineering / Working with LLMs

• Streaming Without a Background ThreadCode Quiz
• 4-bit Loading with BitsAndBytesConfigCode Quiz
• Generating and Decoding OutputCode Quiz
• AutoTokenizer and AutoModel LoadingCode Quiz
• Deploying Open Models on Cloud GPUsSlides / Video
• Securing Self-Hosted Model ServersSlides / Video
• Configuring Sampling ParametersCode Quiz
• Half-Precision Loading with torch_dtypeCode Quiz
• Device Placement with device_mapCode Quiz
• Applying a Chat TemplateCode Quiz
• Loading a Model with the PipelineCode Quiz
• Where trust_remote_code BelongsCode Quiz
• Slicing Off the Prompt TokensCode Quiz
• Stop Sequence with stop_stringsCode Quiz
• Local Embedding and Reranker ModelsSlides / Video
• Benchmarking Local Inference SpeedSlides / Video
• Serving LoRA Adapters LocallySlides / Video
• Containerizing Local Model DeploymentsSlides / Video
• Serving Quantized Models: Speed vs QualitySlides / Video
• Chat Templates Per ModelSlides / Video
• OpenAI-Compatible Local API EndpointsSlides / Video
• CPU Offloading and Mixed InferenceSlides / Video
• Model Sharding and ParallelismSlides / Video
• Estimating VRAM for a ModelSlides / Video
• PagedAttention and KV Cache ManagementSlides / Video
• Production Serving Engines for LLMsSlides / Video
• Local Inference Runtimes ComparedSlides / Video
• Model Weight Formats ExplainedSlides / Video
• Landscape of Open ModelsSlides / Video
• Local Embeddings with sentence-transformersCode Quiz
• Estimating VRAM from Parameter CountCode Quiz
• KV Cache During GenerationCode Quiz
• GPU Offloading with n_gpu_layersCode Quiz
• Authenticating for Gated HF ModelsCode Quiz
• Loading a Model from a Local PathCode Quiz
• Merging a LoRA AdapterCode Quiz
• Batched Generation Padding SideCode Quiz
• max_new_tokens vs max_lengthCode Quiz
• Missing pad token for GPT-2Code Quiz
• OpenAI client base_url pathCode Quiz
• Default context window too smallCode Quiz
• Reading an Ollama responseCode Quiz
• Loading a GGUF with llama-cppCode Quiz
• vLLM model name mismatchCode Quiz
• Open Model Licensing and Commercial UseSlides / Video
• Open-Source vs Open-Weight vs ProprietarySlides / Video
• Why Run LLMs LocallySlides / Video
• Containerizing Model DeploymentsFlashcard
• Security of Self-Hosted InferenceFlashcard
• Serving LoRA Adapters LocallyFlashcard
• Evaluating Quantization Quality LossFlashcard
• Cloud GPU Providers for Self-HostingFlashcard
• Chat Templates per ModelFlashcard
• Edge and On-Device ConstraintsFlashcard
• Context Length and RoPE ScalingFlashcard
• VRAM EstimationFlashcard
• PagedAttention and vLLMFlashcard
• Open Weights vs Open SourceFlashcard
• GGUF Quantization FormatsFlashcard
• Multi-GPU InferenceFlashcard
• Model Weight FormatsFlashcard
• OpenAI-Compatible Local APIFlashcard
• Memory OffloadingFlashcard
• Open-Source Model LandscapeFlashcard
• GPU vs CPU vs Apple SiliconFlashcard
• Local Inference RuntimesFlashcard
• Hugging Face Hub DownloadingFlashcard