Enabling Flash Attention 2

Why Run Models Locally

Quantization Formats: GPTQ, AWQ, GGUF

Streaming Without a Background Thread

4-bit Loading with BitsAndBytesConfig

AutoTokenizer and AutoModel Loading

Generating and Decoding Output

Deploying Open Models on Cloud GPUs

Securing Self-Hosted Model Servers

Loading a Model with the Pipeline

Half-Precision Loading with torch_dtype

Device Placement with device_map

Applying a Chat Template

Configuring Sampling Parameters

Stop Sequence with stop_strings

Where trust_remote_code Belongs

Slicing Off the Prompt Tokens

Local Embedding and Reranker Models

Benchmarking Local Inference Speed

Serving LoRA Adapters Locally

Containerizing Local Model Deployments

Serving Quantized Models: Speed vs Quality

Chat Templates Per Model

OpenAI-Compatible Local API Endpoints

CPU Offloading and Mixed Inference

Model Sharding and Parallelism

Estimating VRAM for a Model

PagedAttention and KV Cache Management

Production Serving Engines for LLMs

Local Inference Runtimes Compared

Model Weight Formats Explained

Landscape of Open Models

Authenticating for Gated HF Models

Batched Generation Padding Side

Estimating VRAM from Parameter Count

GPU Offloading with n_gpu_layers

Loading a Model from a Local Path

KV Cache During Generation

Merging a LoRA Adapter

Local Embeddings with sentence-transformers

Missing pad token for GPT-2

max_new_tokens vs max_length

OpenAI client base_url path

vLLM model name mismatch

Default context window too small

Reading an Ollama response

Loading a GGUF with llama-cpp

Open Model Licensing and Commercial Use

Open-Source vs Open-Weight vs Proprietary

Why Run LLMs Locally

Chat Templates per Model

Containerizing Model Deployments

Security of Self-Hosted Inference

Edge and On-Device Constraints

Evaluating Quantization Quality Loss

Serving LoRA Adapters Locally

Cloud GPU Providers for Self-Hosting

Context Length and RoPE Scaling

GGUF Quantization Formats

OpenAI-Compatible Local API

Multi-GPU Inference

Model Weight Formats

Open-Source Model Landscape

VRAM Estimation

Hugging Face Hub Downloading

Memory Offloading

GPU vs CPU vs Apple Silicon

Open Weights vs Open Source

Local Inference Runtimes

PagedAttention and vLLM