Streaming Without a Background Thread

AutoTokenizer and AutoModel Loading

4-bit Loading with BitsAndBytesConfig

Generating and Decoding Output

Deploying Open Models on Cloud GPUs

Securing Self-Hosted Model Servers

Device Placement with device_map

Half-Precision Loading with torch_dtype

Configuring Sampling Parameters

Loading a Model with the Pipeline

Applying a Chat Template

Slicing Off the Prompt Tokens

Where trust_remote_code Belongs

Stop Sequence with stop_strings

Local Embedding and Reranker Models

Benchmarking Local Inference Speed

Serving LoRA Adapters Locally

Containerizing Local Model Deployments

Serving Quantized Models: Speed vs Quality

Chat Templates Per Model

OpenAI-Compatible Local API Endpoints

CPU Offloading and Mixed Inference

Model Sharding and Parallelism

Estimating VRAM for a Model

PagedAttention and KV Cache Management

Production Serving Engines for LLMs

Local Inference Runtimes Compared

Model Weight Formats Explained

Landscape of Open Models

Estimating VRAM from Parameter Count

Batched Generation Padding Side

Merging a LoRA Adapter

Loading a Model from a Local Path

Authenticating for Gated HF Models

GPU Offloading with n_gpu_layers

KV Cache During Generation

Local Embeddings with sentence-transformers

max_new_tokens vs max_length

Default context window too small

Loading a GGUF with llama-cpp

vLLM model name mismatch

OpenAI client base_url path

Reading an Ollama response

Missing pad token for GPT-2

Open Model Licensing and Commercial Use

Open-Source vs Open-Weight vs Proprietary

Why Run LLMs Locally

Edge and On-Device Constraints

Chat Templates per Model

Cloud GPU Providers for Self-Hosting

Serving LoRA Adapters Locally

Evaluating Quantization Quality Loss

Containerizing Model Deployments

Security of Self-Hosted Inference

Context Length and RoPE Scaling

PagedAttention and vLLM

Hugging Face Hub Downloading

Local Inference Runtimes

Memory Offloading

Open-Source Model Landscape

Open Weights vs Open Source

GGUF Quantization Formats

Multi-GPU Inference

OpenAI-Compatible Local API

GPU vs CPU vs Apple Silicon

VRAM Estimation

Model Weight Formats