TAPPaaS

vllm-amd

Primary audience: developers and services needing a local, OpenAI-compatible LLM endpoint.

Local LLM inference on the AMD Ryzen AI MAX+ 395 integrated GPU (Radeon 8060S, Strix Halo) — an OpenAI-compatible API for LiteLLM or direct clients, with no data leaving your network.

What you get

Capability Access from How
OpenAI-compatible inference API Consumers of vllm-amd:inference http://vllm-amd.<zone>.internal:8000, with the API key vllm-amd:inference hands each consumer (TCP 8000 pinhole for cross-zone consumers)
Large model support API Up to 120B parameters on 128 GB unified memory
Fast inference API ~50 tok/s (7B FP16), ~20 tok/s (30B GPTQ-4bit)

Tested models (reference):

Model Size Quantization Notes
Qwen2.5-7B-Instruct 7B FP16 Default — fast, good quality
Qwen3-14B-AWQ 14B AWQ Good balance speed/quality
Qwen3-30B-A3B-GPTQ-Int4 30B GPTQ-4bit Mixture-of-experts, low active params
Qwen3-Coder-30B-GPTQ-Int4 30B GPTQ-4bit Code generation
openai/gpt-oss-120b 120B AWQ Largest tested

FP8 is not supported on gfx1151. Use AWQ or GPTQ for large models.

What is not included

Requirements

Dependencies

Depends on Purpose
cluster:lxc LXC container provisioning
backup:vm Container snapshots

For installation steps see INSTALL.md.

External references