Qwen/Qwen3.5-4B
Qwen3.5 compact dense multimodal model (4B) — fits on 16 GB consumer GPUs with full 262K context or one Xeon 6 NUMA node or Intel Arc Pro B60/B70
Consumer-GPU-friendly Qwen3.5 dense with MTP support
export VLLM_USE_RUST_FRONTEND=1 vllm serve Qwen/Qwen3.5-4B \ --trust-remote-code \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3
The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /Qwen/Qwen3.5-4B.md.
Overview
Qwen3.5-4B is the compact dense entry in the Qwen3.5 family — same gated delta networks architecture, vision encoder, 262K context, and MTP decoding as the larger siblings, sized for 16 GB consumer GPUs.
Prerequisites
- vLLM version: >= 0.17.0
- Hardware: single 16 GB GPU (RTX 4080 / L4 / A10 / T4-24GB) or one Xeon 6 NUMA node or one Intel Arc Pro B60/B70
Client Usage
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
# The server's model name is the checkpoint it runs: the selected variant's.
model = client.models.list().data[0].id
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Hello!"}],
max_tokens=128,
)
print(resp.choices[0].message.content)
Qwen3.5 thinks before it answers. To turn thinking off for one request, pass
extra_body={"chat_template_kwargs": {"enable_thinking": False}}; for every
request, start the server with
--default-chat-template-kwargs '{"enable_thinking": false}'.
References
assert num_cache_lines >= batchNVIDIAAMD
The CUDA graph capture size is larger than the Mamba cache. Reduce
--max-cudagraph-capture-size (default 512); see
vllm#34571.