Qwen/Qwen3.5-397B-A17B
Multimodal MoE model with gated delta networks architecture, 397B total / 17B active parameters, up to 262K context
Verified on 8x H200, 8x MI300X/MI355X, a single MI350X/MI355X (MXFP4), GB200 nodes, and Ascend 950DT
export VLLM_DEEP_GEMM_WARMUP=skip export VLLM_USE_DEEP_GEMM=0 export VLLM_USE_RUST_FRONTEND=1 vllm serve Qwen/Qwen3.5-397B-A17B \ --trust-remote-code \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 \ --mm-encoder-tp-mode data
The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /Qwen/Qwen3.5-397B-A17B.md.
Overview
Qwen3.5 is a multimodal mixture-of-experts model featuring a gated delta networks architecture with 397B total parameters and 17B active parameters. This guide covers how to efficiently deploy and serve the model across different hardware configurations and workload profiles using vLLM.
Verified configurations
These configurations were verified on 8x H200, 8x MI300X/MI355X and GB200. On H200 and MI300X/MI355X they used the official FP8 checkpoint Qwen/Qwen3.5-397B-A17B-FP8.
| Workload | Hardware | Checkpoint | Parallelism | Other settings |
|---|---|---|---|---|
| Throughput, text only | 8x H200, 8x MI300X/MI355X | FP8 | DP8 + EP | --language-model-only, --enable-prefix-caching |
| Throughput, multimodal | 8x H200, 8x MI300X/MI355X | FP8 | DP8 + EP | --mm-encoder-tp-mode data, --mm-processor-cache-type shm, --enable-prefix-caching |
| Latency | 8x H200, 8x MI300X/MI355X | FP8 | TP8 | MTP-1, prefix caching off |
| Throughput, text only | GB200 (4 GPUs) | NVFP4 | DP4 + EP | --language-model-only, --enable-prefix-caching |
| Throughput, text only | MI355X | FP8 | TP2 + EP | --language-model-only, --enable-prefix-caching |
--language-model-only skips loading the vision encoder and frees memory for KV
cache. For multimodal workloads, --mm-encoder-tp-mode data runs the vision
encoder data-parallel and --mm-processor-cache-type shm caches preprocessed
multimodal inputs in shared memory. MTP-1 lowers time per output token with a
high acceptance rate, but speculative tokens take KV cache capacity, so
throughput drops under high concurrency; tune num_speculative_tokens (1-5) to
the workload.
Single MI350X / MI355X (MXFP4)
AMD's MXFP4 checkpoint amd/Qwen3.5-397B-A17B-MXFP4 (239 GB)
fits one 288 GB MI350X or MI355X: pick the MXFP4 (W4A4) variant above. It sets three things vLLM does not
pick on its own: VLLM_ROCM_USE_AITER=1 turns on the AITER kernels; --attention-backend ROCM_AITER_FA replaces
the auto-selected ROCM_ATTN, whose decode slows down as the context grows on this hybrid model; and
--max-num-seqs 64, because the default (1024) is more than the ~296 Mamba cache blocks a single GPU leaves,
and the server would refuse to start.
Single-stream decode in tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens (128 output tokens), one GPU,
vllm/vllm-openai-rocm:v0.31.0:
| ~200 | ~2k | ~12k | ~30k | |
|---|---|---|---|---|
| MXFP4, this variant (MI355X) | 119 | 119 | 119 | 117 |
+ MTP (3 tokens) + --kv-cache-dtype fp8 (MI355X) | 219 | 201 | 209 | 188 |
| + MTP (3 tokens), default KV cache (MI350X) | 198 | 164 | 125 | 80 |
auto-selected ROCM_ATTN instead of ROCM_AITER_FA (MI355X) | 116 | 89 | 39 | 19 |
GPTQ-Int4, AITER + ROCM_AITER_FA (MI350X) | 75 | 74 | 74 | 74 |
With MTP, add --kv-cache-dtype fp8: on ROCm, MTP's gain otherwise fades as the prompt grows. The MXFP4
checkpoint quantizes activations as well as weights (W4A4): on the same wikitext-2 tokens vLLM scored PPL 3.43
with it and 3.39 with GPTQ-Int4 (W4A16), so GPTQ-Int4 is the higher-fidelity option at about two thirds of the speed.
Processing Ultra-Long Texts
Qwen3.5-397B-A17B natively supports 262,144 tokens. To serve longer inputs, turn on the
Long Context feature above: it applies the model card's YaRN RoPE
configuration via --hf-overrides and raises --max-model-len to ~1,010,000 tokens.
Static YaRN holds the scaling factor constant regardless of input length, so
leave it off unless you actually serve long prompts; for a ~524K workload,
halve the card's factor to 2.0.
See the model card for the full parameter reference.
Client Usage
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1",
timeout=3600
)
# The server's model name is the checkpoint it runs: the selected variant's.
model = client.models.list().data[0].id
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "Read all the text in the image."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model=model,
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
Qwen3.5 thinks before it answers. To turn thinking off for one request, pass
extra_body={"chat_template_kwargs": {"enable_thinking": False}}; for every
request, start the server with
--default-chat-template-kwargs '{"enable_thinking": false}'.
References
- Model card: https://huggingface.co/Qwen/Qwen3.5-397B-A17B
- FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8
- NVFP4 checkpoint: https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4
- GPTQ-Int4 checkpoint: https://huggingface.co/Qwen/Qwen3.5-397B-A17B-GPTQ-Int4
- AMD MXFP4 checkpoint: https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4
- Ascend W8A8 checkpoint: https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp
- Ascend W4A4 checkpoint: https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a4-mxfp4
- vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.5-397B-A17B.html
- vLLM documentation: https://docs.vllm.ai/
Configurations qualified on specific hardware, layered on the baseline command. Unless marked Opt-in, a preset applies automatically on its hardware and strategy; pick Baseline in the builder to drop it.
One TP2 engine with expert parallelism on two of the node's eight B200 GPUs: MTP with three draft tokens, FP8 KV cache, float16 Mamba SSM cache, async scheduling, and a 160 GiB host KV tier so prefixes survive across turns; admission is capped at 48 sequences to hold mean ITL under 20 ms.
image vllm/vllm-openai:v0.32.0
Runbook
Measured by NVIDIA on vLLM v0.32.0 (vllm/vllm-openai:v0.32.0) on two B200 SXM 180 GB with a
942-request multi-turn agentic coding trace (about 64k mean input tokens, most prefix tokens
reused across turns, outputs capped at 12k tokens), replayed at fixed concurrency with real
MTP acceptance (mean acceptance length 3.2 to 3.3). Host: 512 GiB RAM and 200 GiB /dev/shm per
server for the 160 GiB offload tier.
| Concurrency | Output tok/s/GPU | Mean ITL | Mean TTFT |
|---|---|---|---|
| 16 | 936 | 8.7 ms | 0.70 s |
Further concurrencies and the comparison against this recipe's baseline command on the same image and trace are being collected and will follow in a separate update.
The 48-sequence cap trades time to first token for inter-token latency at 64 clients; raise
--max-num-seqs if TTFT matters more than ITL for your traffic. The offload tier only helps
workloads that revisit long prefixes; drop the --kv-offloading-* flags for single-turn traffic.
assert num_cache_lines >= batchNVIDIAAMD
The CUDA graph capture size is larger than the Mamba cache. Reduce
--max-cudagraph-capture-size (default 512); see
vllm#34571.
Prefix caching is experimental
Prefix caching for the Mamba cache runs in the experimental "align" mode.
Out of memory with Encoder ParallelEncoder parallel
--mm-encoder-tp-mode data places a copy of the vision encoder on every
GPU, which takes extra memory. If the server runs out of memory, lower
--max-num-seqs or --max-num-batched-tokens, or drop
--mm-encoder-tp-mode data.