OpenVINO™ Model Server
High-performance model serving for Generative AI and classic deep learning — powered by OpenVINO and optimized for Intel hardware.
What is OVMS?
OpenVINO Model Server (OVMS) is a production-grade, C++ inference server that exposes ML models over standard network APIs. It serves both Generative AI, Agentic workloads (LLMs, VLMs, image generation, audio) and classic deep learning models (object detection, classification, OCR, and more).
- OpenAI-compatible API for text generation, embeddings, image generation, and audio
- KServe APIs for classic model inference
- Runs anywhere — Docker, bare metal, Kubernetes/OpenShift, Windows
- Intel-optimized — CPU, GPU, NPU acceleration via OpenVINO

Quick Start
Serve an LLM with OpenAI-compatible API
On Linux (Docker):
mkdir -p ${HOME}/models
# Model is downloaded automatically from HuggingFace to models folder
docker run --rm -p 8000:8000 \
--user $(id -u):$(id -g) -v ${HOME}/models:/models:rw \
openvino/model_server:latest \
--source_model OpenVINO/Qwen3-4B-int4-ov \
--model_repository_path /models \
--rest_port 8000
For GPU acceleration, use the latest-gpu image tag and pass --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1) to expose the Intel GPU device.
On Windows (binary package):
mkdir c:\models
ovms.exe --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path c:\models --rest_port 8000
Query the model:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
stream = client.chat.completions.create(
model="OpenVINO/Qwen3-4B-int4-ov",
messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}],
stream=True,
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
LLM QuickStart
Serve a Classic Model with KServe API
Download the model:
curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.bin -O
curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.xml -O
On Linux (Docker):
docker run --rm -d -u $(id -u) -v ${PWD}:/models -p 9000:9000 \
openvino/model_server:latest \
--model_name resnet --model_path /models/resnet50.xml \
--mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" \
--port 9000
For GPU acceleration, use the latest-gpu image tag and pass --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1) to expose the Intel GPU device.
Windows (binary package):
ovms --model_name resnet --model_path resnet50.xml --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" --port 9000
Run inference with a sample client
pip install numpy tritonclient[grpc]
curl -L -o image.jpeg https://github.com/openvinotoolkit/model_server/blob/main/demos/common/static/images/bee.jpeg?raw=true
import numpy as np
import tritonclient.grpc as grpcclient
with open("image.jpeg", "rb") as f:
image_bytes = f.read()
client = grpcclient.InferenceServerClient(url="localhost:9000")
inputs = [grpcclient.InferInput("image", [1], "BYTES")]
inputs[0].set_data_from_numpy(np.array([image_bytes], dtype=object))
outputs = [grpcclient.InferRequestedOutput("output")]
result = client.infer(model_name="resnet", inputs=inputs, outputs=outputs)
output = result.as_numpy("output") # (1, 1000) FP32
print("Top-1 class index:", int(np.argmax(output[0])))
Vision model QuickStart
Features
Generative AI
Classic Models & Pipelines
Deployment & Integration
Hardware Acceleration
→ Full feature list
Documentation
Get the Server
Docker images:
docker pull openvino/model_server:latest # Intel CPU
docker pull openvino/model_server:latest-gpu # Intel CPU,GPU,NPU
docker pull openvino/model_server:weekly # pre-production version with all accelerators enabled
Binary official packages (Linux & Windows): GitHub Releases
Binary pre-production packages (Linux & Windows): storage.openvinotoolkit.org
Contributing
Contributions are welcome! Please open an issue or pull request on GitHub.
See the contributing guide for contribution guidelines. For responsible disclosure, see the security policy.
References
* Other names and brands may be claimed as the property of others.