Qwen/Qwen3.5-122B-A10B
Mid-size Qwen3.5 multimodal MoE (122B total / 10B active) with gated delta networks, 256 experts, and 262K context
Qwen3.5 mid-tier MoE — fits on 4x H200 BF16 or 2x H200 FP8
export VLLM_USE_RUST_FRONTEND=1 vllm serve Qwen/Qwen3.5-122B-A10B \ --trust-remote-code \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 \ --mm-encoder-tp-mode data
The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /Qwen/Qwen3.5-122B-A10B.md.
Overview
Qwen3.5-122B-A10B is a mid-tier member of the Qwen3.5 family, sharing the gated delta networks MoE architecture with 122B total parameters and 10B activated per token (256 experts). It is multimodal (vision + text) and natively supports 262K context.
Prerequisites
- vLLM version: >= 0.17.0
- Hardware (BF16): 4x H200 or 8x H100
- Hardware (FP8): 2x H200 or 4x H100
- Hardware (Int4): single 80 GB GPU
AMD (MI325X / MI355X)
Three settings matter on ROCm and none is on by default: VLLM_ROCM_USE_AITER=1
switches the MoE to the AITER FP8 kernels, --attention-backend ROCM_AITER_FA
keeps decode flat across context (the auto-selected ROCM_ATTN decays with
prompt length on this hybrid GDN model), and --kv-cache-dtype fp8 keeps MTP
flat across context too. The AMD commands above set all three (the FP8 KV cache
on the FP8 variant).
The FP8 checkpoint also fits one MI300X-class GPU; there, add
--max-num-seqs 64, because the default exceeds the Mamba block budget on
192 GB. Single-stream decode of the FP8 checkpoint on one GPU (tokens/s at
~200 / ~2k / ~12k / ~30k prompt tokens, 128 output tokens, prefix caching off),
measured 2026-10-07 on vllm/vllm-openai-rocm:v0.31.0:
| MI325X | MI355X | |
|---|---|---|
AITER + ROCM_AITER_FA | 128 / 127 / 127 / 127 | 143 / 143 / 143 / 143 |
| + MTP | 184 / 169 / 172 / 164 | 204 / 193 / 195 / 190 |
This recipe used ROCM_AITER_UNIFIED_ATTN before v0.31.0. Avoid it on MI350X/MI355X with
v0.31.0: there it runs AITER's new gfx950 Gluon kernel, which decodes at 110 tokens/s and,
with MTP, falls from 200 to 72 tokens/s between ~200 and ~30k prompt tokens
(vllm-project/vllm#60408). MI325X is
unaffected (126 tokens/s; 182 to 162 with MTP).
Client Usage
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
# The server's model name is the checkpoint it runs: the selected variant's.
model = client.models.list().data[0].id
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Summarize the gated delta networks paper."}],
max_tokens=512,
)
print(resp.choices[0].message.content)
Qwen3.5 thinks before it answers. To turn thinking off for one request, pass
extra_body={"chat_template_kwargs": {"enable_thinking": False}}; for every
request, start the server with
--default-chat-template-kwargs '{"enable_thinking": false}'.
References
assert num_cache_lines >= batchNVIDIAAMD
The CUDA graph capture size is larger than the Mamba cache. Reduce
--max-cudagraph-capture-size (default 512); see
vllm#34571.
Prefix caching is experimental
Prefix caching for the Mamba cache runs in the experimental "align" mode.