vLLM/Recipes
Search
Qwen

Qwen/Qwen3.5-35B-A3B

Compact Qwen3.5 multimodal MoE (35B total / 3B active) with gated delta networks, 256 experts, and 262K context

Compact Qwen3.5 MoE — single-GPU FP8, 2x GPU or 2x Xeon 6 NUMA nodes or 4x Intel Arc Pro B60/B70 BF16 serving

moe35B / 3B262,144 ctxvLLM 0.17.0+Multimodal
Default configuration: 8× H200 · Tensor Parallel · BF16
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.5-35B-A3B \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data

The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /Qwen/Qwen3.5-35B-A3B.md.

Overview

Qwen3.5-35B-A3B is the smallest MoE in the Qwen3.5 family, sharing the gated delta networks architecture with 35B total parameters and 3B activated per token (256 experts). With FP8 weights it fits on a single 80 GB GPU and supports the full 262K context.

Prerequisites

  • vLLM version: >= 0.17.0
  • Hardware (BF16): 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, 4x Intel Arc Pro B60 / B70
  • Hardware (FP8): single H100/H200
  • Hardware (Int4): single 24 GB GPU

Client Usage

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
# The server's model name is the checkpoint it runs: the selected variant's.
model = client.models.list().data[0].id
resp = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)

Qwen3.5 thinks before it answers. To turn thinking off for one request, pass extra_body={"chat_template_kwargs": {"enable_thinking": False}}; for every request, start the server with --default-chat-template-kwargs '{"enable_thinking": false}'.

References