# Qwen/Qwen3.5-35B-A3B — vLLM serving recipe

> Compact Qwen3.5 multimodal MoE (35B total / 3B active) with gated delta networks, 256 experts, and 262K context

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`, `encoder_parallel` · frontend `rust`

- Interactive page: https://vllm-recipes-hm812xejs-inferact-inc.vercel.app/Qwen/Qwen3.5-35B-A3B
- This configuration as JSON: https://vllm-recipes-hm812xejs-inferact-inc.vercel.app/Qwen/Qwen3.5-35B-A3B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `xeon6`, `arc_pro_b60`, `arc_pro_b70`
- `variant`: `default`, `fp8`, `gptq_int4`, `redhat_fp8`, `mlx_int4`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_dep`, `multi_node_tep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `encoder_parallel`, `long_context` (default: `tool_calling`, `reasoning`, `encoder_parallel`). Use `features=none` to turn them all off.

## Install

Requires vLLM >= 0.17.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.5-35B-A3B \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest Qwen/Qwen3.5-35B-A3B \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

## Known issues

### assert num_cache_lines >= batch

The CUDA graph capture size is larger than the Mamba cache. Reduce
`--max-cudagraph-capture-size` (default 512); see
[vllm#34571](https://github.com/vllm-project/vllm/pull/34571).

### Prefix caching is experimental

Prefix caching for the Mamba cache runs in the experimental "align" mode.

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) is the smallest
MoE in the Qwen3.5 family, sharing the gated delta networks architecture with
35B total parameters and 3B activated per token (256 experts). With FP8 weights
it fits on a single 80 GB GPU and supports the full 262K context.

### Prerequisites

- **vLLM version:** >= 0.17.0
- **Hardware (BF16):** 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, 4x Intel Arc Pro B60 / B70
- **Hardware (FP8):** single H100/H200
- **Hardware (Int4):** single 24 GB GPU

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
# The server's model name is the checkpoint it runs: the selected variant's.
model = client.models.list().data[0].id
resp = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

Qwen3.5 thinks before it answers. To turn thinking off for one request, pass
`extra_body={"chat_template_kwargs": {"enable_thinking": False}}`; for every
request, start the server with
`--default-chat-template-kwargs '{"enable_thinking": false}'`.

### References

- [Model card](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8)
- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4)
- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)
