Code QuizAdvanced

vLLM model name mismatch

Calling a vLLM endpoint with an alias instead of the served model id.

Codepython

# Server started with:
#   python -m vllm.entrypoints.openai.api_server \
#     --model mistralai/Mistral-7B-Instruct-v0.2 --port 8000

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="mistral-7b",
    messages=[{"role": "user", "content": "Hello"}]
)

Why does this request get rejected with a model-not-found error?

← PreviousLoading a GGUF with llama-cpp Next →Open Model Licensing and Commercial Use