{
  "hf_id": "Qwen/Qwen3.5-35B-A3B",
  "meta": {
    "title": "Qwen3.5-35B-A3B",
    "slug": "qwen3.5-35b-a3b",
    "provider": "Qwen",
    "description": "Compact Qwen3.5 multimodal MoE (35B total / 3B active) with gated delta networks, 256 experts, and 262K context",
    "date_added": "2026-04-22",
    "date_updated": "2026-10-27",
    "tasks": [
      "multimodal",
      "text"
    ],
    "performance_headline": "Compact Qwen3.5 MoE — single-GPU FP8, 2x GPU or 2x Xeon 6 NUMA nodes or 4x Intel Arc Pro B60/B70 BF16 serving",
    "related_recipes": [
      "Qwen/Qwen3.5-397B-A17B",
      "Qwen/Qwen3.5-122B-A10B"
    ],
    "hardware": {
      "xeon6": "verified",
      "arc_pro_b60": "verified",
      "arc_pro_b70": "verified",
      "apple_64gb": "supported"
    }
  },
  "recommended_command": {
    "hardware": "h200",
    "strategy": "single_node_tp",
    "variant": "default",
    "hub": "huggingface",
    "checkpoint_url": "https://huggingface.co/Qwen/Qwen3.5-35B-A3B",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {
      "VLLM_USE_RUST_FRONTEND": "1"
    },
    "docker_image": "vllm/vllm-openai:latest",
    "command": "vllm serve Qwen/Qwen3.5-35B-A3B \\\n  --trust-remote-code \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder \\\n  --reasoning-parser qwen3 \\\n  --mm-encoder-tp-mode data",
    "argv": [
      "vllm",
      "serve",
      "Qwen/Qwen3.5-35B-A3B",
      "--trust-remote-code",
      "--tensor-parallel-size",
      "1",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "qwen3_coder",
      "--reasoning-parser",
      "qwen3",
      "--mm-encoder-tp-mode",
      "data"
    ],
    "docker_command": "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  -e VLLM_USE_RUST_FRONTEND=1 \\\n  vllm/vllm-openai:latest Qwen/Qwen3.5-35B-A3B \\\n  --trust-remote-code \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder \\\n  --reasoning-parser qwen3 \\\n  --mm-encoder-tp-mode data",
    "docker_argv": [
      "docker",
      "run",
      "--gpus",
      "all",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "-e",
      "VLLM_USE_RUST_FRONTEND=1",
      "vllm/vllm-openai:latest",
      "Qwen/Qwen3.5-35B-A3B",
      "--trust-remote-code",
      "--tensor-parallel-size",
      "1",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "qwen3_coder",
      "--reasoning-parser",
      "qwen3",
      "--mm-encoder-tp-mode",
      "data"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "NVIDIA",
      "generation": "hopper",
      "compute_arch": "sm90",
      "display_name": "H200",
      "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node",
      "gpu_count": 8,
      "vram_gb": 1128,
      "multi_node": false
    },
    "alternatives": {
      "single_node_tep": "/Qwen/Qwen3.5-35B-A3B/strategies/single_node_tep.json",
      "single_node_dep": "/Qwen/Qwen3.5-35B-A3B/strategies/single_node_dep.json",
      "multi_node_tp": "/Qwen/Qwen3.5-35B-A3B/strategies/multi_node_tp.json",
      "multi_node_dep": "/Qwen/Qwen3.5-35B-A3B/strategies/multi_node_dep.json",
      "multi_node_tep": "/Qwen/Qwen3.5-35B-A3B/strategies/multi_node_tep.json",
      "kv_store_centralized_mooncake": "/Qwen/Qwen3.5-35B-A3B/strategies/kv_store_centralized_mooncake.json",
      "kv_store_distributed_mooncake": "/Qwen/Qwen3.5-35B-A3B/strategies/kv_store_distributed_mooncake.json"
    },
    "by_hardware": {
      "h200": "/Qwen/Qwen3.5-35B-A3B/hw/h200.json",
      "h100": "/Qwen/Qwen3.5-35B-A3B/hw/h100.json",
      "b200": "/Qwen/Qwen3.5-35B-A3B/hw/b200.json",
      "gb200": "/Qwen/Qwen3.5-35B-A3B/hw/gb200.json",
      "b300": "/Qwen/Qwen3.5-35B-A3B/hw/b300.json",
      "gb300": "/Qwen/Qwen3.5-35B-A3B/hw/gb300.json",
      "mi300x": "/Qwen/Qwen3.5-35B-A3B/hw/mi300x.json",
      "mi325x": "/Qwen/Qwen3.5-35B-A3B/hw/mi325x.json",
      "mi350x": "/Qwen/Qwen3.5-35B-A3B/hw/mi350x.json",
      "mi355x": "/Qwen/Qwen3.5-35B-A3B/hw/mi355x.json",
      "xeon6": "/Qwen/Qwen3.5-35B-A3B/hw/xeon6.json",
      "arc_pro_b60": "/Qwen/Qwen3.5-35B-A3B/hw/arc_pro_b60.json",
      "arc_pro_b70": "/Qwen/Qwen3.5-35B-A3B/hw/arc_pro_b70.json"
    }
  },
  "model": {
    "model_id": "Qwen/Qwen3.5-35B-A3B",
    "min_vllm_version": "0.17.0",
    "default_frontend": "rust",
    "architecture": "moe",
    "parameter_count": "35B",
    "active_parameters": "3B",
    "context_length": 262144,
    "base_args": [
      "--trust-remote-code"
    ],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:latest"
      }
    }
  },
  "features": {
    "tool_calling": {
      "description": "Enable automatic tool choice with Qwen3 Coder parser",
      "args": [
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "qwen3_coder"
      ],
      "hardware_overrides": {
        "metal": {
          "args": [
            "--enable-auto-tool-choice",
            "--tool-call-parser",
            "qwen3_coder"
          ]
        }
      }
    },
    "reasoning": {
      "description": "Enable chain-of-thought reasoning with Qwen3 parser",
      "args": [
        "--reasoning-parser",
        "qwen3"
      ],
      "hardware_overrides": {
        "metal": {
          "args": [
            "--reasoning-parser",
            "qwen3"
          ]
        }
      }
    },
    "spec_decoding": {
      "description": "Multi-token prediction speculative decoding for lower latency",
      "args": [
        "--speculative-config",
        "{\"method\":\"mtp\",\"num_speculative_tokens\":1}"
      ]
    },
    "text_only": {
      "description": "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel.",
      "args": [
        "--language-model-only"
      ]
    },
    "encoder_parallel": {
      "description": "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only.",
      "args": [
        "--mm-encoder-tp-mode",
        "data"
      ]
    },
    "long_context": {
      "description": "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0.",
      "args": [
        "--hf-overrides",
        "{\"text_config\": {\"rope_parameters\": {\"mrope_interleaved\": true, \"mrope_section\": [11, 11, 10], \"rope_type\": \"yarn\", \"rope_theta\": 10000000, \"partial_rotary_factor\": 0.25, \"factor\": 4.0, \"original_max_position_embeddings\": 262144}}}",
        "--max-model-len",
        "1010000"
      ],
      "env": {
        "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1"
      }
    }
  },
  "opt_in_features": [
    "spec_decoding",
    "text_only",
    "long_context"
  ],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 84,
      "description": "Full precision BF16 — fits on 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, or 4x Intel Arc Pro B60 / B70"
    },
    "fp8": {
      "model_id": "Qwen/Qwen3.5-35B-A3B-FP8",
      "precision": "fp8",
      "vram_minimum_gb": 42,
      "description": "Qwen official FP8 checkpoint — single-GPU serving",
      "supported_hardware": [
        "arch:sm90",
        "arch:sm100",
        "arch:sm103",
        "arch:gfx942",
        "arch:gfx950",
        "arc_pro_b60",
        "arc_pro_b70"
      ],
      "json": "/Qwen/Qwen3.5-35B-A3B-FP8.json"
    },
    "gptq_int4": {
      "model_id": "Qwen/Qwen3.5-35B-A3B-GPTQ-Int4",
      "precision": "int4",
      "vram_minimum_gb": 21,
      "description": "GPTQ Int4 checkpoint — fits on a single 24GB GPU",
      "supported_hardware": [
        "arch:sm90",
        "arch:sm100",
        "arch:sm103",
        "arch:gfx942",
        "arch:gfx950",
        "arc_pro_b60",
        "arc_pro_b70"
      ],
      "json": "/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4.json"
    },
    "redhat_fp8": {
      "model_id": "RedHatAI/Qwen3.5-35B-A3B-FP8-dynamic",
      "precision": "fp8",
      "label": "FP8 Dynamic (RedHat)",
      "vram_minimum_gb": 42,
      "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization",
      "supported_hardware": [
        "arch:sm90",
        "arch:sm100",
        "arch:sm103",
        "arch:gfx942",
        "arch:gfx950",
        "arc_pro_b60",
        "arc_pro_b70"
      ],
      "extra_args": [
        "--kv-cache-dtype",
        "fp8"
      ],
      "json": "/RedHatAI/Qwen3.5-35B-A3B-FP8-dynamic.json"
    },
    "mlx_int4": {
      "label": "MLX 4-bit",
      "model_id": "mlx-community/Qwen3.5-35B-A3B-4bit",
      "description": "Qwen3.5-35B-A3B, with room for many conversations of about 24,000 words\nat once.",
      "precision": "int4",
      "vram_minimum_gb": 25,
      "supported_hardware": [
        "apple_64gb"
      ],
      "min_vllm_version": "0.31.0",
      "install": {
        "pip": {
          "note": "Requires Apple Silicon and macOS 15 or later. Install vLLM Metal.",
          "command": "curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash\nsource ~/.venv-vllm-metal/bin/activate\n"
        },
        "docker": false
      },
      "supported_frontends": [
        "python"
      ],
      "supported_features": [
        "tool_calling",
        "reasoning"
      ],
      "base_args": [
        "--host",
        "127.0.0.1",
        "--max-model-len",
        "32768"
      ],
      "json": "/mlx-community/Qwen3.5-35B-A3B-4bit.json"
    }
  },
  "hardware_overrides": {
    "cpu": {
      "extra_env": {
        "VLLM_ENGINE_READY_TIMEOUT_S": "1800"
      }
    },
    "xpu": {
      "extra_args": [
        "--enforce-eager"
      ]
    }
  },
  "notes": [
    {
      "title": "Intel Arc Pro validation",
      "when": {
        "hardware": [
          "arc_pro_b60",
          "arc_pro_b70"
        ]
      },
      "body": "Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official\nvLLM XPU image `vllm/vllm-openai-xpu:latest` with `--max-model-len 8192`.\n"
    },
    {
      "title": "Model download",
      "when": {
        "hardware": [
          "apple_64gb"
        ],
        "variants": [
          "mlx_int4"
        ]
      },
      "body": "- The first run downloads 20 GB.\n"
    }
  ],
  "troubleshooting": [
    {
      "title": "Engine startup times out on CPU",
      "when": {
        "hardware": [
          "cpu"
        ]
      },
      "body": "A cold weight download followed by CPU compilation can exceed the default\n600-second engine readiness timeout. The CPU command sets\n`VLLM_ENGINE_READY_TIMEOUT_S=1800`; set any external health-check deadline\nhigher (for example, 2100 seconds). Increasing only the external deadline\ndoes not prevent the frontend from timing out.\n"
    },
    {
      "title": "assert num_cache_lines >= batch",
      "when": {
        "hardware": [
          "nvidia",
          "amd"
        ]
      },
      "body": "The CUDA graph capture size is larger than the Mamba cache. Reduce\n`--max-cudagraph-capture-size` (default 512); see\n[vllm#34571](https://github.com/vllm-project/vllm/pull/34571).\n"
    },
    {
      "title": "Prefix caching is experimental",
      "body": "Prefix caching for the Mamba cache runs in the experimental \"align\" mode.\n"
    }
  ],
  "guide": "## Overview\n\n[Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) is the smallest\nMoE in the Qwen3.5 family, sharing the gated delta networks architecture with\n35B total parameters and 3B activated per token (256 experts). With FP8 weights\nit fits on a single 80 GB GPU and supports the full 262K context.\n\n## Prerequisites\n\n- **vLLM version:** >= 0.17.0\n- **Hardware (BF16):** 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, 4x Intel Arc Pro B60 / B70\n- **Hardware (FP8):** single H100/H200\n- **Hardware (Int4):** single 24 GB GPU\n\n## Client Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\n# The server's model name is the checkpoint it runs: the selected variant's.\nmodel = client.models.list().data[0].id\nresp = client.chat.completions.create(\n    model=model,\n    messages=[{\"role\": \"user\", \"content\": \"Explain gated delta networks in one paragraph.\"}],\n    max_tokens=512,\n)\nprint(resp.choices[0].message.content)\n```\n\nQwen3.5 thinks before it answers. To turn thinking off for one request, pass\n`extra_body={\"chat_template_kwargs\": {\"enable_thinking\": False}}`; for every\nrequest, start the server with\n`--default-chat-template-kwargs '{\"enable_thinking\": false}'`.\n\n## References\n\n- [Model card](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)\n- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8)\n- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4)\n- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)\n",
  "strategy_hardware": {
    "single_node_tep": {
      "metal": "unsupported"
    },
    "single_node_dep": {
      "metal": "unsupported"
    }
  },
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "1bf414e97acf7563297726531f702cdbea6b2921",
    "path": "recipes/models/Qwen/Qwen3.5-35B-A3B.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/Qwen3.5-35B-A3B.yaml",
    "overlays": [
      {
        "source": "vllm-metal",
        "repo": "vllm-project/vllm-metal",
        "ref": "main",
        "sha": "3451a0149531e4d2889488ed1a80b5a74f6bb2c5",
        "path": "recipes/overlays/Qwen/Qwen3.5-35B-A3B.yaml",
        "edit_url": "https://github.com/vllm-project/vllm-metal/edit/main/recipes/overlays/Qwen/Qwen3.5-35B-A3B.yaml"
      }
    ]
  }
}