Streaming Without a Background Thread

4-bit Loading with BitsAndBytesConfig

Generating and Decoding Output

AutoTokenizer and AutoModel Loading

Deploying Open Models on Cloud GPUs

Securing Self-Hosted Model Servers

Configuring Sampling Parameters

Half-Precision Loading with torch_dtype

Device Placement with device_map

Applying a Chat Template

Loading a Model with the Pipeline

Where trust_remote_code Belongs

Slicing Off the Prompt Tokens

Stop Sequence with stop_strings

Local Embedding and Reranker Models

Benchmarking Local Inference Speed

Serving LoRA Adapters Locally

Containerizing Local Model Deployments

Serving Quantized Models: Speed vs Quality

Chat Templates Per Model

OpenAI-Compatible Local API Endpoints

CPU Offloading and Mixed Inference

Model Sharding and Parallelism

Estimating VRAM for a Model

PagedAttention and KV Cache Management

Production Serving Engines for LLMs

Local Inference Runtimes Compared

Model Weight Formats Explained

Landscape of Open Models

Local Embeddings with sentence-transformers

Estimating VRAM from Parameter Count

KV Cache During Generation

GPU Offloading with n_gpu_layers

Authenticating for Gated HF Models

Loading a Model from a Local Path

Merging a LoRA Adapter

Batched Generation Padding Side

max_new_tokens vs max_length

Missing pad token for GPT-2

OpenAI client base_url path

Default context window too small

Reading an Ollama response

Loading a GGUF with llama-cpp

vLLM model name mismatch

Open Model Licensing and Commercial Use

Open-Source vs Open-Weight vs Proprietary

Why Run LLMs Locally

Containerizing Model Deployments

Security of Self-Hosted Inference

Serving LoRA Adapters Locally

Evaluating Quantization Quality Loss

Cloud GPU Providers for Self-Hosting

Chat Templates per Model

Edge and On-Device Constraints

Context Length and RoPE Scaling

VRAM Estimation

PagedAttention and vLLM

Open Weights vs Open Source

GGUF Quantization Formats

Multi-GPU Inference

Model Weight Formats

OpenAI-Compatible Local API

Memory Offloading

Open-Source Model Landscape

GPU vs CPU vs Apple Silicon

Local Inference Runtimes

Hugging Face Hub Downloading