{
  "hf_id": "meta-models/Muse-Spark-Open-816B-A42B",
  "meta": {
    "title": "Muse Spark Open",
    "slug": "muse-spark-open-816b-a42b",
    "provider": "Muse (Meta)",
    "description": "Sparse MoE reasoning model with 816B total / 42B active parameters and a 1M-token context, built for long-horizon agentic coding. Native image and video perception, channel-scoped reasoning and tool calls via the `muse` parsers, and an optional DSpark draft model for speculative decoding.",
    "date_added": "2026-10-15",
    "date_updated": "2026-11-13",
    "tasks": [
      "text",
      "multimodal"
    ],
    "performance_headline": "MXFP4 and BF16 on one 8-GPU node: TP + expert parallel with DSpark for interactive serving, TP + expert parallel alone for peak MXFP4 throughput",
    "related_recipes": [
      "meta-models/Muse-Glimmer-30B"
    ],
    "default_hardware": "mi355x",
    "default_variant": {
      "amd": "mxfp4",
      "nvidia": "default"
    },
    "hardware": {
      "h200": "verified",
      "gb200": "verified",
      "gb300": "verified",
      "mi350x": "verified",
      "mi355x": "verified"
    }
  },
  "recommended_command": {
    "hardware": "b200",
    "strategy": "single_node_tep",
    "variant": "default",
    "hub": "huggingface",
    "checkpoint_url": "https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {
      "VLLM_USE_RUST_FRONTEND": "1"
    },
    "docker_image": "vllm/vllm-openai:muse-spark-open",
    "command": "vllm serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \\\n  --enable-expert-parallel \\\n  --tensor-parallel-size 8 \\\n  --moe-backend deep_gemm_mega_moe \\\n  --reasoning-parser muse \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser muse \\\n  --speculative-config '{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}'",
    "argv": [
      "vllm",
      "serve",
      "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
      "--enable-expert-parallel",
      "--tensor-parallel-size",
      "8",
      "--moe-backend",
      "deep_gemm_mega_moe",
      "--reasoning-parser",
      "muse",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "muse",
      "--speculative-config",
      "{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}"
    ],
    "docker_command": "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  -e VLLM_USE_RUST_FRONTEND=1 \\\n  vllm/vllm-openai:muse-spark-open meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \\\n  --enable-expert-parallel \\\n  --tensor-parallel-size 8 \\\n  --moe-backend deep_gemm_mega_moe \\\n  --reasoning-parser muse \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser muse \\\n  --speculative-config '{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}'",
    "docker_argv": [
      "docker",
      "run",
      "--gpus",
      "all",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "-e",
      "VLLM_USE_RUST_FRONTEND=1",
      "vllm/vllm-openai:muse-spark-open",
      "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
      "--enable-expert-parallel",
      "--tensor-parallel-size",
      "8",
      "--moe-backend",
      "deep_gemm_mega_moe",
      "--reasoning-parser",
      "muse",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "muse",
      "--speculative-config",
      "{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}"
    ],
    "strategy_spec": {
      "name": "single_node_tep",
      "deploy_type": "single_node",
      "display_name": "Tensor + Expert Parallel",
      "short_label": "TEP",
      "orientation": "balanced",
      "description": "Single-node TEP. TP splits dense layers and EP splits expert layers across local GPUs. TP must be set to GPU count to avoid OOM from replicated dense layers. For MoE models only.\n",
      "hardware_match": {
        "min_gpus": 2,
        "max_gpus": 8,
        "multi_node": false,
        "architecture": "moe"
      },
      "vllm_args": [
        "--enable-expert-parallel"
      ],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "NVIDIA",
      "generation": "blackwell",
      "compute_arch": "sm100",
      "display_name": "B200",
      "description": "NVIDIA B200 SXM 180 GB HBM3e · 8-GPU HGX B200 node",
      "gpu_count": 8,
      "vram_gb": 1440,
      "multi_node": false
    },
    "alternatives": {
      "single_node_tp": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/single_node_tp.json",
      "single_node_dep": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/single_node_dep.json",
      "multi_node_tp": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/multi_node_tp.json",
      "multi_node_tp_pp": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/multi_node_tp_pp.json",
      "multi_node_tep": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/multi_node_tep.json",
      "multi_node_dep": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/multi_node_dep.json",
      "pd_cluster": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/pd_cluster.json",
      "kv_store_centralized_mooncake": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/kv_store_centralized_mooncake.json",
      "kv_store_distributed_mooncake": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/kv_store_distributed_mooncake.json"
    },
    "by_hardware": {
      "b200": "/meta-models/Muse-Spark-Open-816B-A42B/hw/b200.json",
      "gb200": "/meta-models/Muse-Spark-Open-816B-A42B/hw/gb200.json",
      "b300": "/meta-models/Muse-Spark-Open-816B-A42B/hw/b300.json",
      "gb300": "/meta-models/Muse-Spark-Open-816B-A42B/hw/gb300.json"
    }
  },
  "model": {
    "model_id": "meta-models/Muse-Spark-Open-816B-A42B",
    "min_vllm_version": "0.33.0",
    "nightly_required": true,
    "docker_image": {
      "nvidia": "vllm/vllm-openai:muse-spark-open",
      "amd": "vllm/vllm-openai-rocm:muse-spark-open"
    },
    "default_frontend": "rust",
    "architecture": "moe",
    "parameter_count": "816B",
    "active_parameters": "42B",
    "context_length": 1048576,
    "base_args": [],
    "base_env": {},
    "install": {
      "docker": {
        "command": "docker pull vllm/vllm-openai:muse-spark-open",
        "note": "Serve from the image of your GPU vendor — no pip wheel is available now."
      }
    }
  },
  "features": {
    "reasoning": {
      "description": "Channel-scoped chain-of-thought. Strength is set per request with the top-level `reasoning_effort` field — low/medium/high/xhigh map to strength hints 8/32/256/512, defaulting to xhigh when the field is omitted. Legacy `disable_reasoning` and `enable_thinking` kwargs are ignored.",
      "args": [
        "--reasoning-parser",
        "muse"
      ]
    },
    "tool_calling": {
      "description": "Channel-scoped tool calls. The parser is scoped to the tool channel, so an invocation echoed back inside ordinary assistant text is not re-parsed as a call. The template always offers `self` as a valid recipient.",
      "args": [
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "muse"
      ]
    },
    "spec_decoding": {
      "description": "DSpark speculative decoding, drafting up to seven tokens per pass. On NVIDIA, adaptive verification keeps only the draft slots worth verifying, so it stays on at large batch sizes; on AMD it is a fixed seven-token profile, best for latency-bound serving.",
      "default_mode": "dspark",
      "modes": {
        "dspark": {
          "label": "DSpark",
          "description": "Seven-token DSpark profile used for BF16, MXFP4, and NVFP4.",
          "variants": [
            "bf16",
            "mxfp4",
            "default"
          ],
          "args": [
            "--speculative-config",
            "{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}"
          ],
          "hardware_overrides": {
            "amd": {
              "args": [
                "--speculative-config",
                "{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":false,\"attention_backend\":\"ROCM_AITER_FA\"}"
              ],
              "env": {
                "AITER_UNIFIED_ATTN_SHORT_QUERY_3D": "1",
                "VLLM_KV_CACHE_LAYOUT": "LBHNC"
              }
            }
          }
        },
        "dspark_specialized": {
          "label": "DSpark + specialized decode",
          "description": "Low-concurrency decode path: DSpark verify steps run on fused persistent kernels, other steps on the regular path. Needs the Tensor + Expert Parallel strategy (TP4/8 on B300, TP4/8/16 on GB300, TP8 on MI355X) and an image built from vLLM main at or after 4fab5a0; vLLM refuses other deployments at startup.",
          "variants": [
            "default",
            "mxfp4"
          ],
          "hardware": {
            "default": "unsupported",
            "b300": "supported",
            "gb300": "supported",
            "mi355x": "supported"
          },
          "args": [
            "--speculative-config",
            "{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":false}",
            "--additional-config",
            "{\"decode_backend\":\"latency_specialized\"}",
            "--moe-backend",
            "auto"
          ],
          "hardware_overrides": {
            "amd": {
              "args": [
                "--speculative-config",
                "{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":false,\"attention_backend\":\"ROCM_AITER_FA\"}",
                "--monokernel"
              ],
              "env": {
                "AITER_UNIFIED_ATTN_SHORT_QUERY_3D": "1",
                "VLLM_KV_CACHE_LAYOUT": "LBHNC"
              }
            }
          }
        }
      }
    }
  },
  "opt_in_features": [
    "spec_decoding"
  ],
  "variants": {
    "bf16": {
      "model_id": "meta-models/Muse-Spark-Open-816B-A42B",
      "precision": "bf16",
      "vram_minimum_gb": 1959,
      "description": "BF16 reference weights for NVIDIA and AMD; the ROCm path is validated on gfx950 (MI350X/MI355X).",
      "hardware_overrides": {
        "amd": {
          "extra_args": [
            "--moe-backend",
            "triton",
            "--attention-backend",
            "ROCM_AITER_FA",
            "--max-model-len",
            "32768",
            "--max-num-seqs",
            "32",
            "--max-num-batched-tokens",
            "4096"
          ]
        }
      }
    },
    "mxfp4": {
      "model_id": "amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4",
      "precision": "mxfp4",
      "vram_minimum_gb": 611,
      "supported_hardware": [
        "mi350x",
        "mi355x"
      ],
      "description": "AMD Quark OCP MXFP4 W4A4 weights for gfx950 on the AITER MXFP4 MoE kernels.",
      "hardware_overrides": {
        "amd": {
          "extra_args": [
            "--attention-backend",
            "ROCM_AITER_FA",
            "--max-model-len",
            "32768",
            "--max-num-seqs",
            "128",
            "--max-num-batched-tokens",
            "8192"
          ]
        }
      },
      "json": "/amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4.json"
    },
    "default": {
      "model_id": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
      "precision": "nvfp4",
      "vram_minimum_gb": 640,
      "description": "NVFP4 weights for Blackwell. The primary NVIDIA serving path.",
      "default_modes": {
        "spec_decoding": "dspark"
      },
      "json": "/meta-models/Muse-Spark-Open-816B-A42B-NVFP4.json"
    }
  },
  "strategy_hardware": {
    "single_node_dep": {
      "amd": "unsupported"
    },
    "multi_node_dep": {
      "amd": "unsupported"
    }
  },
  "hardware_overrides": {
    "amd": {
      "extra_args": [
        "--dtype",
        "bfloat16",
        "--distributed-executor-backend",
        "mp",
        "--language-model-only",
        "--compilation-config",
        "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}",
        "--no-enable-prefix-caching",
        "--enable-chunked-prefill"
      ],
      "extra_env": {
        "VLLM_ROCM_USE_AITER": "1",
        "VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE": "bf16",
        "VLLM_USE_BREAKABLE_CUDAGRAPH": "1"
      }
    }
  },
  "notes": [
    {
      "title": "Specialized decode needs a newer image",
      "when": {
        "hardware": [
          "b300",
          "gb300",
          "mi355x"
        ],
        "features": [
          "spec_decoding"
        ]
      },
      "body": "The **DSpark + specialized decode** method is on vLLM main since 2026-11-03 and is not in the `muse-spark-open` images. Serve it from `vllm/vllm-openai:nightly-4fab5a07475822c6eaa971dceb2e2220d0446808` (B300/GB300) or `vllm/vllm-openai-rocm:nightly-4fab5a07475822c6eaa971dceb2e2220d0446808` (MI355X), or a later nightly."
    }
  ],
  "guide": "## Overview\n\nMuse Spark Open is a sparse-MoE reasoning model — 816B total parameters, 42B\nactive — with a 1M-token context window, aimed at long-horizon agentic coding.\nIt is the model Muse Glimmer 30B was distilled from.\n\n## Architecture\n\n62 layers, hidden size 8192, and **every layer is a mixture-of-experts layer** —\nthere are no dense layers. Each holds 256 experts and uses 8 per token. The\nexperts are 799B of the 816B; everything else is small.\n\n**The experts work narrow.** A learned matrix squeezes each token from 8192 to\n4096, all 256 experts run at that width, and a second matrix widens the result\nback. Both stay in full precision even when the experts are 4-bit. The router\nreads the token *before* the squeeze, and in fp32 — picking the wrong expert\ncosts more than running a smaller one. There is no shared expert.\n\n**Attention is 128 query heads sharing 16 key-value heads**, head dim 64, so the\nKV cache is eight times smaller than with one key-value head per query head.\n\n**Layers alternate between a narrow and a wide view**, on a cycle of four:\n\n```\nsliding_window: 2048\nlayer_types: [sliding_attention, full_attention, sliding_attention, sliding_attention, ...]\n```\n\nThree layers in each cycle look back 2048 tokens; the other (layers 1, 5, …,\n61) sees the whole context and drops rotary position (`layer_rope_theta: 0`),\nbecause rotary becomes unreliable over very long distances.\nAcross 62 layers that is **46 narrow layers and 16 wide ones**, and it is what\nmakes a 1M-token context affordable — only the 16 wide layers hold KV for the\nwhole sequence.\n\n**Normalization appears in more places than usual**, each doing a job: a norm\nbefore *and* after each block (the ones after carry no learnable weight), on\nqueries and keys before rotary position, on each attention head before its\nsigmoid gate, and on every expert's output before the weighted blend. The\nresidual stream is carried in fp32.\n\nLogits are scaled and then squashed through tanh (`final_logit_softcapping: 20.0`)\nbefore the softmax.\n\n## Prerequisites\n\n- **vLLM**: Use Docker image for now until the code is in vLLM main repo.\n- **Hardware**: the checkpoints hold ~1.64 TB of weights in BF16 and\n  ~509 GB in Quark MXFP4, before KV cache. NVFP4 (~533 GB) is the primary\n  NVIDIA Blackwell path and fits one 8-GPU node. On AMD, all three AMD\n  variants are validated on gfx950 (MI350X/MI355X).\n\n## NVIDIA\n\nMeta ships BF16 and NVFP4 checkpoints. On Blackwell, serve\n`meta-models/Muse-Spark-Open-816B-A42B-NVFP4` on one 8-GPU node with\n**Tensor + Expert Parallel** (TP8 + `--enable-expert-parallel`), the default\nstrategy here. The command builder turns DSpark on by default for NVFP4; see\n[Speculative decoding](#speculative-decoding-dspark).\n\nNVIDIA supports every parallelism this recipe offers: TP, TP + PP, TP + expert\nparallel, data + expert parallel, and prefill/decode disaggregation. AMD\nsupports only TP and TP + expert parallel.\n\n## AMD ROCm variants\n\nQuark MXFP4 is the default AMD variant.\n\n| Variant | Checkpoint | AMD hardware | Experts / attention |\n| --- | --- | --- | --- |\n| MXFP4 | `amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4` | MI350X, MI355X | AITER MXFP4 MoE / AITER FA |\n| BF16 → MXFP4 | `meta-models/Muse-Spark-Open-816B-A42B` | MI350X, MI355X | experts of layers 1-60 rounded to MXFP4 at load, AITER MXFP4 MoE / AITER FA |\n| BF16 | `meta-models/Muse-Spark-Open-816B-A42B` | MI350X, MI355X | Triton BF16 MoE / AITER FA |\n\nIn both MXFP4 variants the experts of the first and last layers stay BF16 and\nrun on Triton BF16 MoE.\n\nOn AMD, the recipe's commands are text-only (`--language-model-only`), use the\nV2 model runner, disable prefix caching, enable chunked prefill, and request\nbreakable `FULL_AND_PIECEWISE` graphs. The ROCm MoE path supports the Tensor\nParallel and Tensor + Expert Parallel strategies; data-parallel strategies\nare not available on AMD.\n\n### Choosing TP or TP + expert parallel (MXFP4 variants)\n\nUse plain **TP** for latency-bound serving (a handful of concurrent requests)\nand the **Tensor + Expert Parallel** strategy (TP8 + `--enable-expert-parallel`,\n32 of 256 experts per GPU) for throughput at 32 or more concurrent requests,\nwhere it also lowers TTFT. Accuracy is the same under both. Keep **TP** for\nBF16: its Triton experts are slower with expert parallelism. With DSpark on,\nexpert parallelism pays off from a few concurrent requests; see the DSpark\nsection.\n\nThe AMD commands use `--max-model-len 32768`. For the full\n1M-token context, the MXFP4 variant is validated on TP with\n`--max-model-len 1048576` and `--max-num-seqs 8`, with either the default or\n`--kv-cache-dtype fp8` KV cache.\n\n## Launching the Server\n\nOn Blackwell, single-node TEP explicitly uses `deep_gemm_mega_moe`.\nFor NVFP4 TEP4/TEP8 with DP1 and PP1, this enables hybrid TRT-LLM/MegaMoE\nexecution: eligible replicated batches of up to 128 tokens at TEP4 or\n256 tokens at TEP8 use TRT-LLM, while sequence-sharded batches use MegaMoE.\nThe threshold counts tokens entering MoE, not concurrent requests.\nOn 8-GPU B200 and B300 nodes, DEP8 also explicitly uses MegaMoE, without\nthis hybrid switch.\n\nThese backend settings apply with or without DSpark. Specialized decode is\na separate path that currently rejects MegaMoE, so its mode explicitly selects\nthe `auto` MoE backend.\n\n## Frontends\n\nCommands use the Rust frontend by default on NVIDIA and AMD ROCm.\n\nTo use the Python frontend, select Python in the command builder or replace\n`VLLM_USE_RUST_FRONTEND=1` with `VLLM_USE_RUST_FRONTEND=0` in the generated\ncommand.\n\n## Reasoning effort\n\nSet the strength per request with the top-level `reasoning_effort` field:\n\n| `reasoning_effort` | Strength hint |\n| --- | --- |\n| `low` | 8 |\n| `medium` | 32 |\n| `high` | 256 |\n| `xhigh` | 512 (default) |\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"model\": \"meta-models/Muse-Spark-Open-816B-A42B-NVFP4\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Refactor this module.\"}],\n    \"reasoning_effort\": \"xhigh\"\n  }'\n```\n\n### Multimodal\n\nMuse Spark is a native multimodal model that understands image, audio and video. Video is\nprocessed through a series of sampled video frames at load time. On AMD,\nthe recipe serves text only.\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"model\": \"meta-models/Muse-Spark-Open-816B-A42B-NVFP4\",\n    \"messages\": [{\"role\": \"user\", \"content\": [\n      {\"type\": \"video_url\", \"video_url\": {\"url\": \"https://example.com/demo.mp4\"}},\n      {\"type\": \"text\", \"text\": \"What happens, and when?\"}\n    ]}]\n  }'\n```\n\nThe audio track of a video is not processed. To include it, extract the\naudio and send it as a separate audio part in the same request.\n\nVideo sampling rate can be changed when starting the server.\n\n```bash\n--media-io-kwargs '{\"video\": {\"fps\": 1}}'\n```\n\n## Speculative decoding (DSpark)\n\nInferact trained a DSpark speculator for this model. The draft is a\nsmall sliding-window model that reads the target's residual stream at selected\nlayers (`target_layer_ids`) and proposes a block per forward.\n\nOn NVIDIA, DSpark is on by default for NVFP4. It drafts up to seven tokens and\nuses **adaptive verification** (`enable_adaptive_verification: true`), which\nuses draft confidence to verify only the slots likely to be accepted. Because\nof that it keeps paying off at large batch sizes, not just for single-request\nlatency. Turn off **Spec Decoding** in the builder to serve without the\ndraft.\n\nOn AMD, selecting **DSpark** emits the fixed seven-token profile with AITER FA\nattention on every variant. Its causal multi-token decode is handled by\nAITER unified attention, with the gfx950 short-query 3D schedule enabled.\nDSpark gains the most when only a few requests are in flight, and still\nstays ahead of plain TP at 32 concurrent requests. It also runs under the\n**Tensor + Expert Parallel** strategy on the MXFP4 variants (TP8 +\n`--enable-expert-parallel`), which is the faster combination from a few\nconcurrent requests up to about 64 at the same accuracy; plain TP + DSpark\nremains the lowest-latency choice for a single request. With expert\nparallelism, DSpark is validated up to `--max-num-seqs 128`; with plain TP,\nkeep `--max-num-seqs 32`. For throughput beyond about 64 concurrent\nrequests, use expert parallelism without DSpark.\n\n### Specialized decode (B300, GB300, MI355X)\n\nThe **DSpark + specialized decode** method runs DSpark verify steps on fused\npersistent kernels and leaves prefill and other steps on the regular path.\nIt targets latency-bound serving with a few requests in flight. Use it with\nthe **Tensor + Expert Parallel** strategy; vLLM refuses other deployments at\nstartup. It is on vLLM main since 2026-11-03 and not in the `muse-spark-open`\nimages, so serve it from a nightly image built at or after `4fab5a0`.\n\n- **B300/GB300, NVFP4** (`--additional-config '{\"decode_backend\":\"latency_specialized\"}'`):\n  TP4 or TP8 on B300; TP4, TP8 or TP16 on GB300, with expert parallelism.\n  It serves the verify step of one request (1 + 7 draft tokens) under FULL CUDA graphs and needs adaptive\n  verification off, which the method sets. 1.53x single-stream decode at\n  TP8 + EP8 ([#64077](https://github.com/vllm-project/vllm/pull/64077)).\n- **MI355X, MXFP4** (`--monokernel`): TP8 with expert parallelism and a bf16\n  KV cache. It runs decoder layers 1-60 of the DSpark verify steps of 1 to\n  16 requests as persistent FlyDSL launches: one launch a layer for one or\n  two requests, three for 3-16 requests, with the verify attention fused\n  into the launches for up to four requests. The launches keep the expert\n  activations in bf16 (MXFP4 weights x bf16 activations) instead of the\n  checkpoint's W4A4, so outputs are close to, but not bit-identical with,\n  the regular path.\n\n### Adaptive DSpark (manual opt-in, Triton attention)\n\nAdaptive verification captures variable-length decode CUDA graphs, which\nthe AITER FA attention backend does not support; with the AMD MXFP4\ncommand above it fails at startup. To use it on MI350X/MI355X, switch the\ntarget to Triton attention and use the adaptive config:\n\n```bash\nexport VLLM_ADAPTIVE_VERIFICATION_PROFILE_CONTEXT_LEN=8192\n\n--attention-backend TRITON_ATTN \\\n--speculative-config '{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"standard\",\"enable_adaptive_verification\":true,\"attention_backend\":\"TRITON_ATTN\"}'\n```\n\nSeven tokens remain the maximum draft depth. At startup, adaptive verification\nprofiles the DSpark and target costs, then uses draft confidence to keep only\nthe slots whose expected accepted-token benefit justifies their verification\ncost. There is no hand-tuned acceptance threshold. Keep the profile context\nrepresentative of production traffic.\n\n\n## References\n\n- [Muse Glimmer 30B recipe](/meta-models/Muse-Glimmer-30B) — the distilled 30B sibling\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "1bf414e97acf7563297726531f702cdbea6b2921",
    "path": "recipes/models/meta-models/Muse-Spark-Open-816B-A42B.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/meta-models/Muse-Spark-Open-816B-A42B.yaml",
    "overlays": []
  }
}