{
  "hf_id": "moonshotai/Kimi-K3.1-Pro",
  "meta": {
    "title": "Kimi-K3.1-Pro",
    "slug": "kimi-k3.1-pro",
    "provider": "Moonshot AI",
    "description": "2.8T-parameter native multimodal MoE with Kimi Delta Attention, sparse MLA, and a 1M-token context window",
    "date_added": "2026-11-02",
    "date_updated": "2026-11-14",
    "tasks": [
      "multimodal",
      "text"
    ],
    "performance_headline": "TP8 on two GB300 trays or one B300 node, or one MI355X node, for the MXFP4 checkpoint; TP32 on a 4-host TPU slice",
    "related_recipes": [
      "moonshotai/Kimi-K3"
    ],
    "default_hardware": "gb300",
    "hardware": {
      "gb300": "verified",
      "vr200": "supported",
      "810e": "verified",
      "m890": "verified",
      "ironwood": "supported"
    }
  },
  "recommended_command": {
    "hardware": "gb300",
    "strategy": "multi_node_tp",
    "variant": "default",
    "hub": "huggingface",
    "checkpoint_url": "https://huggingface.co/moonshotai/Kimi-K3.1-Pro",
    "node_count": 2,
    "deploy_type": "multi_node",
    "env": {
      "GLOO_SOCKET_IFNAME": "$IFACE_NAME",
      "NCCL_SOCKET_IFNAME": "$IFACE_NAME",
      "VLLM_ENGINE_READY_TIMEOUT_S": "3600"
    },
    "docker_image": "vllm/vllm-openai:kimi-k31",
    "head_command": "vllm serve moonshotai/Kimi-K3.1-Pro \\\n  --trust-remote-code \\\n  --tensor-parallel-size 8 \\\n  --nnodes 2 \\\n  --node-rank 0 \\\n  --master-addr $HEAD_IP \\\n  --load-format fastsafetensors \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser kimi_k3 \\\n  --reasoning-parser kimi_k3",
    "worker_command": "vllm serve moonshotai/Kimi-K3.1-Pro \\\n  --trust-remote-code \\\n  --tensor-parallel-size 8 \\\n  --nnodes 2 \\\n  --node-rank 1 \\\n  --master-addr $HEAD_IP \\\n  --headless \\\n  --load-format fastsafetensors \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser kimi_k3 \\\n  --reasoning-parser kimi_k3",
    "worker_commands": [
      "vllm serve moonshotai/Kimi-K3.1-Pro \\\n  --trust-remote-code \\\n  --tensor-parallel-size 8 \\\n  --nnodes 2 \\\n  --node-rank 1 \\\n  --master-addr $HEAD_IP \\\n  --headless \\\n  --load-format fastsafetensors \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser kimi_k3 \\\n  --reasoning-parser kimi_k3"
    ],
    "head_argv": [
      "vllm",
      "serve",
      "moonshotai/Kimi-K3.1-Pro",
      "--trust-remote-code",
      "--tensor-parallel-size",
      "8",
      "--nnodes",
      "2",
      "--node-rank",
      "0",
      "--master-addr",
      "$HEAD_IP",
      "--load-format",
      "fastsafetensors",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "kimi_k3",
      "--reasoning-parser",
      "kimi_k3"
    ],
    "worker_argv": [
      "vllm",
      "serve",
      "moonshotai/Kimi-K3.1-Pro",
      "--trust-remote-code",
      "--tensor-parallel-size",
      "8",
      "--nnodes",
      "2",
      "--node-rank",
      "1",
      "--master-addr",
      "$HEAD_IP",
      "--headless",
      "--load-format",
      "fastsafetensors",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "kimi_k3",
      "--reasoning-parser",
      "kimi_k3"
    ],
    "worker_argvs": [
      [
        "vllm",
        "serve",
        "moonshotai/Kimi-K3.1-Pro",
        "--trust-remote-code",
        "--tensor-parallel-size",
        "8",
        "--nnodes",
        "2",
        "--node-rank",
        "1",
        "--master-addr",
        "$HEAD_IP",
        "--headless",
        "--load-format",
        "fastsafetensors",
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "kimi_k3",
        "--reasoning-parser",
        "kimi_k3"
      ]
    ],
    "head_docker_command": "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n  vllm/vllm-openai:kimi-k31 moonshotai/Kimi-K3.1-Pro \\\n  --trust-remote-code \\\n  --tensor-parallel-size 8 \\\n  --nnodes 2 \\\n  --node-rank 0 \\\n  --master-addr $HEAD_IP \\\n  --load-format fastsafetensors \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser kimi_k3 \\\n  --reasoning-parser kimi_k3",
    "worker_docker_command": "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n  vllm/vllm-openai:kimi-k31 moonshotai/Kimi-K3.1-Pro \\\n  --trust-remote-code \\\n  --tensor-parallel-size 8 \\\n  --nnodes 2 \\\n  --node-rank 1 \\\n  --master-addr $HEAD_IP \\\n  --headless \\\n  --load-format fastsafetensors \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser kimi_k3 \\\n  --reasoning-parser kimi_k3",
    "worker_docker_commands": [
      "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n  vllm/vllm-openai:kimi-k31 moonshotai/Kimi-K3.1-Pro \\\n  --trust-remote-code \\\n  --tensor-parallel-size 8 \\\n  --nnodes 2 \\\n  --node-rank 1 \\\n  --master-addr $HEAD_IP \\\n  --headless \\\n  --load-format fastsafetensors \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser kimi_k3 \\\n  --reasoning-parser kimi_k3"
    ],
    "head_docker_argv": [
      "docker",
      "run",
      "--gpus",
      "all",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "-e",
      "GLOO_SOCKET_IFNAME=$IFACE_NAME",
      "-e",
      "NCCL_SOCKET_IFNAME=$IFACE_NAME",
      "-e",
      "VLLM_ENGINE_READY_TIMEOUT_S=3600",
      "vllm/vllm-openai:kimi-k31",
      "moonshotai/Kimi-K3.1-Pro",
      "--trust-remote-code",
      "--tensor-parallel-size",
      "8",
      "--nnodes",
      "2",
      "--node-rank",
      "0",
      "--master-addr",
      "$HEAD_IP",
      "--load-format",
      "fastsafetensors",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "kimi_k3",
      "--reasoning-parser",
      "kimi_k3"
    ],
    "worker_docker_argv": [
      "docker",
      "run",
      "--gpus",
      "all",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "-e",
      "GLOO_SOCKET_IFNAME=$IFACE_NAME",
      "-e",
      "NCCL_SOCKET_IFNAME=$IFACE_NAME",
      "-e",
      "VLLM_ENGINE_READY_TIMEOUT_S=3600",
      "vllm/vllm-openai:kimi-k31",
      "moonshotai/Kimi-K3.1-Pro",
      "--trust-remote-code",
      "--tensor-parallel-size",
      "8",
      "--nnodes",
      "2",
      "--node-rank",
      "1",
      "--master-addr",
      "$HEAD_IP",
      "--headless",
      "--load-format",
      "fastsafetensors",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "kimi_k3",
      "--reasoning-parser",
      "kimi_k3"
    ],
    "worker_docker_argvs": [
      [
        "docker",
        "run",
        "--gpus",
        "all",
        "--privileged",
        "--ipc=host",
        "-p",
        "8000:8000",
        "-v",
        "~/.cache/huggingface:/root/.cache/huggingface",
        "-e",
        "GLOO_SOCKET_IFNAME=$IFACE_NAME",
        "-e",
        "NCCL_SOCKET_IFNAME=$IFACE_NAME",
        "-e",
        "VLLM_ENGINE_READY_TIMEOUT_S=3600",
        "vllm/vllm-openai:kimi-k31",
        "moonshotai/Kimi-K3.1-Pro",
        "--trust-remote-code",
        "--tensor-parallel-size",
        "8",
        "--nnodes",
        "2",
        "--node-rank",
        "1",
        "--master-addr",
        "$HEAD_IP",
        "--headless",
        "--load-format",
        "fastsafetensors",
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "kimi_k3",
        "--reasoning-parser",
        "kimi_k3"
      ]
    ],
    "strategy_spec": {
      "name": "multi_node_tp",
      "deploy_type": "multi_node",
      "display_name": "Multi-Node Tensor Parallel",
      "short_label": "Multi-node TP",
      "orientation": "latency",
      "description": "Multi-node TP. Splits the model across GPUs spanning multiple nodes. TP is set to total GPU count across all nodes. Head node serves requests, worker nodes run with --headless. Requires InfiniBand interconnect.\n",
      "hardware_match": {
        "min_gpus": 2,
        "multi_node": true
      },
      "parallel_flag": "--tensor-parallel-size",
      "env": {
        "GLOO_SOCKET_IFNAME": "$IFACE_NAME",
        "NCCL_SOCKET_IFNAME": "$IFACE_NAME"
      }
    },
    "hardware_profile": {
      "brand": "NVIDIA",
      "generation": "blackwell",
      "compute_arch": "sm103",
      "display_name": "GB300 NVL4",
      "description": "NVIDIA GB300 Grace-Blackwell Ultra compute tray · 288 GB HBM3e/GPU · 4 GPUs (NVL72 tray unit)",
      "gpu_count": 4,
      "vram_gb": 1152,
      "multi_node": false
    },
    "alternatives": {
      "multi_node_dep": "/moonshotai/Kimi-K3.1-Pro/strategies/multi_node_dep.json",
      "pd_cluster": "/moonshotai/Kimi-K3.1-Pro/strategies/pd_cluster.json",
      "kv_store_centralized_mooncake": "/moonshotai/Kimi-K3.1-Pro/strategies/kv_store_centralized_mooncake.json",
      "kv_store_distributed_mooncake": "/moonshotai/Kimi-K3.1-Pro/strategies/kv_store_distributed_mooncake.json"
    },
    "by_hardware": {
      "gb300": "/moonshotai/Kimi-K3.1-Pro/hw/gb300.json",
      "b300": "/moonshotai/Kimi-K3.1-Pro/hw/b300.json",
      "vr200": "/moonshotai/Kimi-K3.1-Pro/hw/vr200.json",
      "mi355x": "/moonshotai/Kimi-K3.1-Pro/hw/mi355x.json",
      "ironwood": "/moonshotai/Kimi-K3.1-Pro/hw/ironwood.json",
      "810e": "/moonshotai/Kimi-K3.1-Pro/hw/810e.json",
      "m890": "/moonshotai/Kimi-K3.1-Pro/hw/m890.json"
    }
  },
  "model": {
    "model_id": "moonshotai/Kimi-K3.1-Pro",
    "min_vllm_version": "0.34.0",
    "nightly_required": true,
    "docker_image": {
      "nvidia": "vllm/vllm-openai:kimi-k31",
      "amd": "vllm/vllm-openai-rocm:kimi-k31-5c14fe8"
    },
    "architecture": "moe",
    "parameter_count": "2.8T",
    "active_parameters": "16 experts/token + shared (of 896 routed)",
    "context_length": 1048576,
    "base_args": [
      "--trust-remote-code"
    ],
    "install": {
      "docker": {
        "command": "docker pull vllm/vllm-openai:kimi-k31"
      }
    }
  },
  "features": {
    "tool_calling": {
      "description": "Enable automatic tool choice with the Kimi K3 tool-call parser.",
      "args": [
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "kimi_k3"
      ]
    },
    "reasoning": {
      "description": "Parse Kimi K3 reasoning output separately from the final answer.",
      "args": [
        "--reasoning-parser",
        "kimi_k3"
      ]
    },
    "spec_decoding": {
      "description": "Use DSpark speculative decoding with the Kimi-K3.1 draft.",
      "args": [
        "--speculative-config",
        "{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":5,\"method\":\"dspark\",\"draft_sample_method\":\"probabilistic\"}"
      ],
      "hardware_overrides": {
        "mi355x": {
          "args": [
            "--kv-cache-dtype",
            "fp8",
            "--speculative-config",
            "{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":5,\"method\":\"dspark\",\"attention_backend\":\"ROCM_AITER_MLA\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\"}"
          ]
        }
      },
      "hardware": {
        "ironwood": "unsupported"
      }
    },
    "text_only": {
      "description": "Skip the vision encoder for text-only workloads; frees its memory for KV cache.",
      "hardware": {
        "ironwood": "unsupported"
      },
      "args": [
        "--language-model-only"
      ]
    },
    "low_latency_decode": {
      "label": "Low-latency decode",
      "description": "Run decode steps on the fused latency_specialized decode path (CuTe DSL kernels on NVIDIA, FlyDSL kernels on MI355X). Fastest at low concurrency; needs FULL decode CUDA graphs. On MI355X it needs speculative decoding and at most 16 decode rows per step; see the AgentX TP8 preset's concurrency-1 and 4 points.",
      "strategies": [
        "single_node_tp",
        "multi_node_tp"
      ],
      "hardware": {
        "ppu": "unsupported",
        "ironwood": "unsupported"
      },
      "args": [
        "--additional-config",
        "{\"decode_backend\":\"latency_specialized\"}"
      ]
    },
    "nvfp4_kv_cache": {
      "label": "NVFP4 KV cache",
      "description": "Store the MLA latent in NVFP4 instead of FP8 for about 1.5x the KV cache tokens; the indexer keys stay FP8.",
      "hardware": {
        "ppu": "unsupported",
        "ironwood": "unsupported"
      },
      "args": [
        "--kv-cache-dtype",
        "nvfp4_ds_mla"
      ]
    }
  },
  "opt_in_features": [
    "spec_decoding",
    "text_only",
    "low_latency_decode",
    "nvfp4_kv_cache"
  ],
  "variants": {
    "default": {
      "precision": "mxfp4",
      "vram_minimum_gb": 1874,
      "supported_hardware": [
        "b300",
        "gb300",
        "vr200",
        "mi355x",
        "810e",
        "m890",
        "ironwood"
      ],
      "description": "MXFP4 routed-expert weights (~1.56 TB on disk); attention, shared experts and the vision tower stay in BF16.",
      "hardware_overrides": {
        "810e": {
          "docker_image": "egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun-tmp/inference-xpu-pytorch:kimi-k3.1-pro-vllm0.32.0-cuda13.0-pytorch2.13.0-20261103",
          "docker_serve_command": "vllm serve",
          "install": {
            "pip": false
          }
        },
        "m890": {
          "docker_image": "egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun-tmp/inference-xpu-pytorch:kimi-k3.1-pro-vllm0.32.0-cuda13.0-pytorch2.13.0-20261103",
          "docker_serve_command": "vllm serve",
          "install": {
            "pip": false
          }
        },
        "vr200": {
          "docker_image": "vllm/vllm-openai:kimi-k31-cu134",
          "install": {
            "pip": false
          }
        },
        "ironwood": {
          "install": {
            "docker": false,
            "pip": {
              "command": "git clone https://github.com/vllm-project/vllm-torchtpu.git && cd vllm-torchtpu && git checkout kimi-k31pro && VLLM_TARGET_DEVICE=empty uv sync --group vllm && source .venv/bin/activate",
              "note": "Source build of vLLM-TorchTPU from the kimi-k31pro branch (Python 3.12, uv; its vllm group builds the pinned vLLM 0.32.0). Tracks vllm-torchtpu#2775 until it merges into main."
            }
          },
          "extra_env": {
            "TPU_VLLM_ENABLE_UNIFIED_BLOCK_POOL": "1",
            "TPU_MOE_HIERARCHICAL_EP": "1"
          },
          "extra_args": [
            "--language-model-only",
            "--limit-mm-per-prompt",
            "{\"image\":0,\"video\":0}",
            "--enable-ep-weight-filter",
            "--kv-cache-dtype",
            "fp8",
            "--async-scheduling",
            "--enable-prefix-caching"
          ]
        }
      }
    }
  },
  "strategy_hardware": {
    "multi_node_tep": {
      "810e": "supported",
      "m890": "supported",
      "ironwood": "supported"
    },
    "multi_node_dep": {
      "gb300": "supported"
    },
    "single_node_tp": {
      "810e": "unsupported",
      "m890": "unsupported"
    },
    "single_node_tep": {
      "810e": "unsupported",
      "m890": "unsupported"
    },
    "multi_node_tp": {
      "810e": "unsupported",
      "m890": "unsupported",
      "ironwood": "unsupported"
    },
    "pd_cluster": {
      "810e": "unsupported",
      "m890": "unsupported",
      "ironwood": "unsupported"
    }
  },
  "strategy_min_gpus": {
    "single_node_tp": 8,
    "single_node_tep": 8,
    "multi_node_tp": 8,
    "multi_node_dep": 16,
    "810e": 32,
    "m890": 32,
    "ironwood": 32
  },
  "hardware_overrides": {
    "blackwell": {
      "extra_args": [
        "--load-format",
        "fastsafetensors"
      ],
      "extra_env": {
        "VLLM_ENGINE_READY_TIMEOUT_S": "3600"
      }
    },
    "rubin": {
      "extra_args": [
        "--load-format",
        "fastsafetensors"
      ],
      "extra_env": {
        "VLLM_ENGINE_READY_TIMEOUT_S": "3600"
      }
    },
    "amd": {
      "extra_env": {
        "VLLM_ROCM_USE_AITER": "1",
        "SAFETENSORS_FAST_GPU": "1",
        "VLLM_ROCM_USE_AITER_MOE_SITUV2": "1",
        "VLLM_USE_BREAKABLE_CUDAGRAPH": "1",
        "VLLM_ENGINE_READY_TIMEOUT_S": "7200"
      }
    }
  },
  "presets": {
    "mi355x_agentx": {
      "label": "AgentX TP8",
      "hardware": "mi355x",
      "strategy": "single_node_tp",
      "workload": "Agentic (InferenceX AgentX)",
      "description": "One MI355X node at TP8: plain TP8 with the low-latency decode backend and 7 draft tokens at concurrency 1, DCP8 with the low-latency decode backend and 5 draft tokens at 4, DCP8 with host KV offload and prefill every 32 engine steps at 16 to 32 (4, 4 and 3 draft tokens), and DCP8 with host KV offload and 3 draft tokens at 48, with linear layers, MoE combine and all-reduce in BF16.",
      "source": {
        "label": "InferenceX #5089",
        "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/5089",
        "key": "kimik3.1-fp4-mi355x-vllm-agentic-dspark",
        "revision": "75d74552457a3f44438c6b9bb9906062fc8f82c6"
      },
      "image": "vllm/vllm-openai-rocm:kimi-k31-5c14fe8@sha256:a61996b4c270ec09111802f62b44f5c20ad4cb73068be022b2804a287b924b9a",
      "select": {
        "frontend": "python",
        "features": {
          "spec_decoding": true,
          "text_only": true,
          "low_latency_decode": false
        },
        "kv_offload": "simple"
      },
      "args": [
        "--load-format",
        "fastsafetensors",
        "--max-model-len",
        "1048576",
        "--enable-prefix-caching",
        "--kv-cache-dtype",
        "fp8",
        "--mamba-ssm-cache-dtype",
        "float32",
        "--stream-interval",
        "10",
        "--decode-context-parallel-size",
        "8",
        "--dcp-comm-backend",
        "a2a",
        "--cp-kv-cache-interleave-size",
        "1",
        "--prefix-match-unit",
        "64",
        "--moe-backend",
        "auto",
        "--seed",
        "7",
        "--disable-uvicorn-access-log",
        "--max-num-batched-tokens",
        "16384",
        "--gpu-memory-utilization",
        "0.90",
        "--attention-config",
        "{\"indexer_kv_dtype\":\"mxfp4\"}",
        "--additional-config",
        "{\"kda_prefill_backend\":\"fused\"}"
      ],
      "env": {
        "VLLM_USE_V2_MODEL_RUNNER": "1",
        "VLLM_FASTSAFETENSORS_QUEUE_SIZE": "-1",
        "VLLM_ROCM_AITER_MLA_ASM_PADDING": "asm",
        "VLLM_ROCM_AITER_MLA_DCP_VERIFY": "asm",
        "VLLM_ROCM_DCP_AITER_ALLGATHER": "1",
        "VLLM_ROCM_DCP_VUP_RS": "1",
        "VLLM_ROCM_USE_AITER_FLYDSL_SPARSE_MLA": "1",
        "AITER_FLYDSL_SPARSE_MLA": "1",
        "VLLM_ROCM_DSA_DCP_GLOBAL_KEY_PREFILL": "1",
        "VLLM_ROCM_DSA_DCP_ROWSHARD_PREFILL": "1",
        "VLLM_ROCM_DSA_MULTISTREAM": "1",
        "VLLM_USE_DIRECT_DCP_A2A": "0",
        "VLLM_USE_DIRECT_DCP_Q_GATHER": "0",
        "VLLM_USE_DIRECT_DCP_KV_GATHER": "0",
        "AITER_FLYDSL_STAGE2_FP8": "0",
        "VLLM_ROCM_QUICK_REDUCE_QUANTIZATION": "NONE",
        "AITER_BF16_FP8_MOE_BOUND": "0",
        "HSA_NO_SCRATCH_RECLAIM": "1",
        "VLLM_ROCM_MEM_HEADROOM_GIB": "8",
        "VLLM_K3_SP_PREFILL": "1",
        "THP_MEM_ALLOC_ENABLE": "1",
        "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1200",
        "PYTHONNOUSERSITE": "1",
        "PYTHONHASHSEED": "42"
      },
      "json_args": {
        "--speculative-config": {
          "rejection_sample_method": "block"
        },
        "--kv-transfer-config": {
          "kv_connector_extra_config": {
            "cpu_bytes_to_use_per_rank": 224875000000
          }
        },
        "--additional-config": {
          "kda_prefill_backend": "fused"
        },
        "--compilation-config": {
          "mode": 3,
          "cudagraph_mode": "FULL_AND_PIECEWISE",
          "custom_ops": [
            "+fused_rms_norm_gated"
          ]
        }
      },
      "points": {
        "c1": {
          "label": "1 (no DCP, low-latency decode)",
          "concurrency": 1,
          "select": {
            "features": {
              "low_latency_decode": true
            },
            "kv_offload": "off"
          },
          "remove_args": [
            "--decode-context-parallel-size",
            "--dcp-comm-backend",
            "--cp-kv-cache-interleave-size"
          ],
          "args": [
            "--max-num-seqs",
            "1"
          ],
          "env": {
            "VLLM_USE_BREAKABLE_CUDAGRAPH": "0",
            "VLLM_KIMI_SD_DSA_PROLOGUE": "1"
          },
          "json_args": {
            "--speculative-config": {
              "num_speculative_tokens": 7
            },
            "--compilation-config": {
              "max_cudagraph_capture_size": 16,
              "cudagraph_capture_sizes": [
                2,
                4,
                6,
                8,
                10,
                12,
                14,
                16
              ]
            }
          }
        },
        "c4": {
          "label": "4 (low-latency decode)",
          "concurrency": 4,
          "select": {
            "features": {
              "low_latency_decode": true
            },
            "kv_offload": "off"
          },
          "args": [
            "--max-num-seqs",
            "2"
          ],
          "env": {
            "VLLM_USE_BREAKABLE_CUDAGRAPH": "0"
          },
          "json_args": {
            "--speculative-config": {
              "num_speculative_tokens": 5
            },
            "--compilation-config": {
              "max_cudagraph_capture_size": 12,
              "cudagraph_capture_sizes": [
                6,
                12
              ]
            }
          }
        },
        "c16": {
          "label": "16 (KV offload, prefill every 32 steps)",
          "concurrency": 16,
          "args": [
            "--max-num-seqs",
            "8",
            "--prefill-schedule-interval",
            "32"
          ],
          "env": {
            "VLLM_PREFILL_CADENCE_NO_ESCAPE": "1"
          },
          "json_args": {
            "--speculative-config": {
              "num_speculative_tokens": 4
            },
            "--compilation-config": {
              "max_cudagraph_capture_size": 16384,
              "cudagraph_capture_sizes": [
                5,
                10,
                15,
                20,
                25,
                30,
                35,
                40,
                45,
                50,
                64,
                96,
                128,
                160,
                192,
                224,
                256,
                288,
                320,
                352,
                384,
                416,
                448,
                480,
                512,
                640,
                768,
                896,
                1024,
                1280,
                1536,
                1792,
                2048,
                2560,
                3072,
                3584,
                4096,
                4608,
                5120,
                6144,
                6400,
                7168,
                8192,
                8704,
                10240,
                12288,
                14336,
                16384
              ]
            }
          }
        },
        "c24": {
          "label": "24 (KV offload, prefill every 32 steps)",
          "concurrency": 24,
          "args": [
            "--max-num-seqs",
            "13",
            "--prefill-schedule-interval",
            "32"
          ],
          "env": {
            "VLLM_PREFILL_CADENCE_NO_ESCAPE": "1"
          },
          "json_args": {
            "--speculative-config": {
              "num_speculative_tokens": 4
            },
            "--compilation-config": {
              "max_cudagraph_capture_size": 16384,
              "cudagraph_capture_sizes": [
                5,
                10,
                15,
                20,
                25,
                30,
                35,
                40,
                45,
                50,
                55,
                60,
                65,
                70,
                75,
                80,
                96,
                128,
                160,
                192,
                224,
                256,
                288,
                320,
                352,
                384,
                416,
                448,
                480,
                512,
                640,
                768,
                896,
                1024,
                1280,
                1536,
                1792,
                2048,
                2560,
                3072,
                3584,
                4096,
                4608,
                5120,
                6144,
                6400,
                7168,
                8192,
                8704,
                10240,
                12288,
                14336,
                16384
              ]
            }
          }
        },
        "c32": {
          "label": "32 (KV offload, prefill every 32 steps)",
          "concurrency": 32,
          "args": [
            "--max-num-seqs",
            "20",
            "--max-num-batched-tokens",
            "10240",
            "--gpu-memory-utilization",
            "0.91",
            "--prefill-schedule-interval",
            "32"
          ],
          "env": {
            "VLLM_PREFILL_CADENCE_NO_ESCAPE": "1"
          },
          "json_args": {
            "--speculative-config": {
              "num_speculative_tokens": 3
            },
            "--compilation-config": {
              "max_cudagraph_capture_size": 10240,
              "cudagraph_capture_sizes": [
                4,
                8,
                12,
                16,
                20,
                24,
                28,
                32,
                36,
                40,
                44,
                48,
                52,
                56,
                60,
                64,
                68,
                72,
                76,
                80,
                84,
                88,
                92,
                96,
                128,
                160,
                192,
                224,
                256,
                288,
                320,
                352,
                384,
                416,
                448,
                480,
                512,
                640,
                768,
                896,
                1024,
                1280,
                1536,
                1792,
                2048,
                2560,
                3072,
                3584,
                4096,
                4608,
                5120,
                6144,
                6400,
                7168,
                8192,
                8704,
                10240
              ]
            }
          }
        },
        "c48": {
          "label": "48 (KV offload)",
          "concurrency": 48,
          "default": true,
          "args": [
            "--max-num-seqs",
            "48",
            "--max-num-batched-tokens",
            "10240",
            "--gpu-memory-utilization",
            "0.91"
          ],
          "json_args": {
            "--speculative-config": {
              "num_speculative_tokens": 3
            },
            "--compilation-config": {
              "max_cudagraph_capture_size": 10240,
              "cudagraph_capture_sizes": [
                4,
                8,
                12,
                16,
                20,
                24,
                28,
                32,
                36,
                40,
                44,
                48,
                52,
                56,
                60,
                64,
                68,
                72,
                76,
                80,
                84,
                88,
                92,
                96,
                100,
                104,
                108,
                112,
                116,
                120,
                124,
                128,
                132,
                136,
                140,
                144,
                148,
                152,
                156,
                160,
                164,
                168,
                172,
                176,
                180,
                184,
                188,
                192,
                224,
                256,
                288,
                320,
                352,
                384,
                416,
                448,
                480,
                512,
                640,
                768,
                896,
                1024,
                1280,
                1536,
                1792,
                2048,
                2560,
                3072,
                3584,
                4096,
                4608,
                5120,
                6144,
                6400,
                7168,
                8192,
                8704,
                10240
              ]
            }
          }
        }
      },
      "runbook": "The MI355X AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`kimik3.1-fp4-mi355x-vllm-agentic-dspark`, [SemiAnalysisAI/InferenceX#5089](https://github.com/SemiAnalysisAI/InferenceX/pull/5089)), one TP8 engine on the node's eight GPUs. It keeps linear layers, MoE combine and all-reduce in BF16 and has been measured only on the image digest the preset pins.\n\n**Points.**\n- **Concurrency 1: plain TP8, no DCP, KV cache on the GPU.** At one request DCP's extra KV capacity isn't needed: the server log reports 2.8M KV tokens, 2.7x a 1M-token request. The DSpark draft runs 7 tokens, `--max-num-seqs` is 1, and the capture sizes 2 to 16 cover the 8-row decode step (1 + 7). `VLLM_KIMI_SD_DSA_PROLOGUE=1` runs each DSA layer's decode prologue as one launch; it needs DCP off.\n- **Concurrency 4: DCP8, KV cache on the GPU.** 5 draft tokens at `--max-num-seqs 2`, so a decode step is at most 12 rows (2 × (1 + 5)) and the capture sizes are 6 and 12. DCP8 holds 19.1M KV tokens on the GPUs.\n- **Concurrency 16, 24 and 32: DCP8 with host KV offload, prefill every 32 engine steps.** `--max-num-seqs` is 8, 13 and 20, below the concurrency, with 4, 4 and 3 draft tokens. `--prefill-schedule-interval 32` admits prefill only once every 32 engine steps, so decode runs in long uninterrupted stretches. With fewer slots than clients, requests always wait, and vLLM would otherwise drop the hold on any step that leaves requests waiting; `VLLM_PREFILL_CADENCE_NO_ESCAPE=1` keeps it. Concurrency 32 also runs `--max-num-batched-tokens 10240` at 0.91 memory utilization.\n- **Concurrency 48: DCP8 with host KV offload, prefill on every step.** 3 draft tokens at `--max-num-seqs 48`. Decode batches are at most 48 × 4 rows, so the sizes 4 to 192 in steps of 4 cover every decode step, and the list continues to 10240, the value of `--max-num-batched-tokens`.\n\n**Serving config.**\n- **Low-latency decode** (the Low-latency decode feature, on at concurrency 1 and 4) sets `decode_backend: latency_specialized`. On MI355X it runs pure decode steps of up to 16 rows on fused FlyDSL kernels; prefill and mixed steps run the default path. Among other startup checks, it needs: TP8 with no expert, data or pipeline parallelism; AITER on, the `ROCM_AITER_MLA` attention backend and an fp8 (E4M3) KV cache; speculative decoding on; no KV offload connector; `--max-num-seqs` 1, 2, 4 or 8 with `--max-num-seqs` × (1 + draft tokens) ≤ 16; a BF16 KDA `in_proj` (no `--quantization-config`); and an FP32 KDA recurrent state. Without DCP it serves only 3 or 7 draft tokens with `--max-num-seqs` 1 or 2. At concurrency 1 the server log shows the fused path running the sparse MLA attention itself (the one-launch prologue, the indexer's scoring and DSA split-K, 16 splits); under DCP8 at concurrency 4 the MLA layers run vLLM's attention ops and the rest of the layer is fused. The pinned image also turns on this path's low-concurrency speedups by default: the `VLLM_K31_BEAT_*` variables and `VLLM_K3_INDEXER_SAMPLED_TOPK`, plus `VLLM_KIMI_SD_DSA_PROLOGUE_INDEXER` where `VLLM_KIMI_SD_DSA_PROLOGUE` is on (concurrency 1). All of them default to 1, so the preset sets none. Some act only at one request: at concurrency 4 the server log reports the fused Markov draft step off with more than one request. The decode backend used from concurrency 16 ignores them.\n- **CUDA graphs.** K3.1 has no `torch.compile` submodule, so vLLM resolves `FULL_AND_PIECEWISE` to `FULL_DECODE_ONLY` and captures decode only. Breakable graphs (`VLLM_USE_BREAKABLE_CUDAGRAPH=1`, from the AMD profile) are what let the points from concurrency 16 capture the prefill sizes too. They cost graph memory, which comes out of the KV cache, so keep the capture list to the sizes the workload uses. Concurrency 1 and 4 capture decode sizes only and set it to 0.\n- **`--max-num-batched-tokens` 16384 (10240 at concurrency 32 and 48).** With prefix caching, the hybrid KV manager ends every non-final prefill chunk on a block boundary; the server log reports a KV block of 8192 tokens under DCP8 and 1024 without DCP (`kv lcm block sizes`).\n- **`--gpu-memory-utilization` 0.90 (0.91 at concurrency 32 and 48) and `VLLM_ROCM_MEM_HEADROOM_GIB=8`.** Kernel scratch is allocated outside this budget, so leave headroom. The memory profile also misses some DCP prefill transients, so the headroom variable caps torch's allocator 8 GiB below free memory (`Allocator headroom: ... headroom 8.0 GiB` in the server log).\n- **`VLLM_K3_SP_PREFILL=1`** (set on every point, active under DCP8). In batches of at least 4096 tokens (prefill and large mixed steps), the residual stream stays token-sharded across the TP ranks, so each MoE layer's three all-reduces (attention output, routed latent and MoE output) become reduce-scatters and all-gathers over symmetric memory. These sum the TP partials in FP32 in a fixed order. Decode steps and smaller batches are unchanged. The server log reports `sequence-parallel prefill enabled (batches of >= 4096 tokens)` at concurrency 4 to 48; at concurrency 1 (no DCP) it reports the configuration unsupported and the default all-reduce path runs, and the variable is kept there as InferenceX ran it.\n- **DCP8** shards the MLA latent and indexer-key caches across the TP group, with the A2A DCP backend and `--cp-kv-cache-interleave-size 1`. `VLLM_ROCM_DCP_VUP_RS=1` applies `v_up` first and then sums the ranks with AITER's LSE all-gather and reduce-scatter, for the target's MLA layers and the draft's; it needs `VLLM_ROCM_DCP_AITER_ALLGATHER=1`. `VLLM_ROCM_DSA_MULTISTREAM=1` runs the DSA indexer, the DCP score all-gather and merge, and the gate on a second stream in decode. Concurrency 1 keeps these variables as InferenceX ran it; with DCP off they are expected to have no effect.\n- **The indexer key cache is MXFP4** (`indexer_kv_dtype`). AITER needs padded FP4 indexer pages for it. Stock AITER asserts on the page stride at startup. `kda_prefill_backend: fused` prefills the KDA layers with the fused gfx950 kernel.\n- **Spec decoding** (the Spec decoding feature, on in this preset) uses the K3.1 DSpark draft [`Inferact/Kimi-K3.1-Pro-DSpark-preview`](https://huggingface.co/Inferact/Kimi-K3.1-Pro-DSpark-preview) (5 layers, not the K3 draft) with `ROCM_AITER_MLA` attention, FP8 KV, probabilistic drafting and block rejection; each point sets its draft token count.\n- The **Simple** KV-offload option (concurrency 16 to 48) gets 224,875,000,000 bytes of host DRAM per rank, InferenceX's 1799 GB split over 8 ranks. Size `cpu_bytes_to_use_per_rank` to your host.\n\n**Precision.** As the K3.1 precision policy requires: BF16 all-reduce (`VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE`), BF16 MoE combine (`AITER_FLYDSL_STAGE2_FP8=0`), BF16 KDA input projections (no `--quantization-config`) and an FP32 KDA recurrent state (`--mamba-ssm-cache-dtype float32`).\n\n**Image.** The pinned image carries kernels and switches that are not in vLLM main yet. On main, the DCP and DSA variables (`VLLM_ROCM_AITER_MLA_DCP_VERIFY`, `VLLM_ROCM_DCP_*`, `VLLM_ROCM_DSA_*`), `VLLM_ROCM_MEM_HEADROOM_GIB`, `VLLM_K3_SP_PREFILL`, `VLLM_KIMI_SD_DSA_PROLOGUE`, `VLLM_PREFILL_CADENCE_NO_ESCAPE` and the low-latency speedup variables above are not read, and the low-latency decode backend does not serve K3.1 without DCP: there it also requires `VLLM_ROCM_USE_AITER_FP4BMM=1` and still refuses K3.1's sparse (indexer) MLA layers, so the concurrency-1 point fails at startup on main or nightly images.\n\n[InferenceX run 43717568009](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/43717568009) passed all six performance points and GSM8K at concurrency 1, 4, 32 and 48 at `75d74552` (single run per point, synthetic acceptance as below):\n\n| Concurrency | Interactivity p50 / p90 (tokens/s/user) | Total tokens/s per GPU |\n| --- | ---: | ---: |\n| 1 | 263.6 / 255.1 | 2,164 |\n| 4 | 189.1 / 169.7 | 3,137 |\n| 16 | 103.0 / 85.3 | 9,089 |\n| 24 | 83.2 / 69.6 | 13,387 |\n| 32 | 68.1 / 59.5 | 15,850 |\n| 48 | 40.6 / 26.9 | 20,319 |\n\nGSM8K exact match (strict) over 1319 questions, with block verification, was 0.980 at concurrency 1, 0.982 at 4, 0.980 at 32 and 0.979 at 48.\n\n### Throughput measurement only (synthetic acceptance)\n\nTo measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for the point's draft token count: 3.69 for 7 tokens, 3.41 for 5, 3.19 for 4 and 2.88 for 3. For example, at concurrency 48:\n\n```bash\n--speculative-config '{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":3,\"method\":\"dspark\",\"attention_backend\":\"ROCM_AITER_MLA\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":2.88}'\n```\n\nInferenceX also turns the tool-call and reasoning parsers off for these runs, because the Kimi parsers stop streaming on the unverified draft tokens. Synthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.\n",
      "json": "/moonshotai/Kimi-K3.1-Pro/presets/mi355x_agentx.json"
    },
    "b300_agentx": {
      "label": "AgentX TP8 + DCP8",
      "hardware": "b300",
      "strategy": "single_node_tp",
      "workload": "Agentic (InferenceX AgentX)",
      "description": "Concurrency 1 and 6 on one B300 node: TP8 with DCP8, the low-latency decode backend on FULL decode CUDA graphs, the K3.1 DSpark draft with 4 tokens on TOKENSPEED_MLA, and host KV offload.",
      "source": {
        "label": "InferenceX #4871",
        "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4871",
        "key": "kimik3.1-fp4-b300-vllm-agentic-dspark",
        "revision": "75b1a3c7b745ddf77f79d00fc3a4c939bbcf8774"
      },
      "image": "vllm/vllm-openai:kimi-k31-mc@sha256:f81998764e8594d71b40788c5285ae112acf9230129d5d0d238b5df83da5acf6",
      "select": {
        "frontend": "python",
        "features": {
          "tool_calling": true,
          "reasoning": true,
          "spec_decoding": true,
          "text_only": true,
          "low_latency_decode": true,
          "nvfp4_kv_cache": false
        },
        "dcp": 8,
        "kv_offload": "simple"
      },
      "args": [
        "--decode-context-parallel-size",
        "8",
        "--dcp-comm-backend",
        "a2a",
        "--cp-kv-cache-interleave-size",
        "1",
        "--enable-prefix-caching",
        "--kv-cache-dtype",
        "fp8",
        "--mamba-ssm-cache-dtype",
        "float32",
        "--use-replayssm",
        "--no-enable-flashinfer-autotune",
        "--max-num-batched-tokens",
        "16384",
        "--gpu-memory-utilization",
        "0.9",
        "--stream-interval",
        "10",
        "--disable-uvicorn-access-log",
        "--attention-config",
        "{\"indexer_kv_dtype\":\"mxfp4\",\"tokenspeed_mla_min_split_kv\":32}"
      ],
      "env": {
        "VLLM_ENGINE_READY_TIMEOUT_S": "7200",
        "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "900",
        "VLLM_USE_V2_MODEL_RUNNER": "1",
        "VLLM_USE_DIRECT_DCP_A2A": "1",
        "VLLM_USE_DIRECT_DCP_Q_GATHER": "1",
        "VLLM_USE_DIRECT_DCP_KV_GATHER": "1",
        "VLLM_ALLREDUCE_USE_FLASHINFER": "1",
        "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0",
        "VLLM_USE_NCCL_SYMM_MEM": "0",
        "VLLM_ENABLE_GEMM_RS": "0",
        "TOKENSPEED_MLA_FP16_PARTIALS": "0"
      },
      "json_args": {
        "--speculative-config": {
          "revision": "6a2ef20a05efe88aabbf9ecaa12d020c3b0edf94",
          "attention_backend": "TOKENSPEED_MLA",
          "num_speculative_tokens": 4,
          "rejection_sample_method": "block"
        },
        "--kv-transfer-config": {
          "kv_connector_extra_config": {
            "cpu_bytes_to_use_per_rank": 281125000000
          }
        },
        "--compilation-config": {
          "cudagraph_mode": "FULL_DECODE_ONLY"
        }
      },
      "points": {
        "c1": {
          "label": "1",
          "concurrency": 1,
          "args": [
            "--max-num-seqs",
            "2"
          ],
          "json_args": {
            "--compilation-config": {
              "cudagraph_capture_sizes": [
                5,
                10
              ]
            }
          }
        },
        "c6": {
          "label": "6",
          "concurrency": 6,
          "args": [
            "--max-num-seqs",
            "6"
          ],
          "json_args": {
            "--compilation-config": {
              "cudagraph_capture_sizes": [
                5,
                10,
                15,
                20,
                25,
                30
              ]
            }
          }
        }
      },
      "runbook": "The single-node B300 TP8 + DCP8 AgentX configuration at concurrency 1 and 6. It follows the `agentic.yaml` points of InferenceX's B300 vLLM AgentX lane (SemiAnalysisAI/InferenceX#4871) and has been measured only on the image digest the preset pins. It serves text with tool-call and reasoning parsers.\n\n**Serving config.**\n- **TP8 with DCP8.** DCP8 shards the MLA latent and indexer-key caches across the TP group, with the A2A DCP backend, `--cp-kv-cache-interleave-size 1` and the direct DCP all-to-all and gather paths (`VLLM_USE_DIRECT_DCP_*`). The MLA KV cache is FP8 and the indexer key cache MXFP4 (`indexer_kv_dtype`).\n- **Low-latency decode** (the Low-latency decode feature, on in this preset) runs every pure decode step on the fused CuTe DSL path, on the V2 model runner, in FULL_DECODE_ONLY CUDA graphs. Each point captures 5, 10, 15, … rows (1 + 4 draft tokens per request) up to its largest planned decode batch; a decode step above the largest captured size runs eager.\n- **Spec decoding** (on in this preset) uses the K3.1 DSpark draft [`Inferact/Kimi-K3.1-Pro-DSpark-preview`](https://huggingface.co/Inferact/Kimi-K3.1-Pro-DSpark-preview) at revision `6a2ef20a`, with 4 tokens, TOKENSPEED_MLA attention for the draft layers, probabilistic drafting and block rejection. `tokenspeed_mla_min_split_kv: 32` in `--attention-config` sets the minimum number of KV splits of that decode attention (larger values can take more workspace memory), and `TOKENSPEED_MLA_FP16_PARTIALS=0` turns off its FP16 partials.\n- **`--use-replayssm`.** With speculative decoding it turns on the KDA layers' RecoverSSM path: verification replays the accepted prefix instead of keeping one recurrent state per draft token. The KDA recurrent state is FP32 (`--mamba-ssm-cache-dtype float32`), as the K3.1 precision policy requires.\n- **Collectives.** FlashInfer all-reduce stays on (`VLLM_ALLREDUCE_USE_FLASHINFER=1`); NCCL symmetric memory, the fused GEMM + reduce-scatter and the sharded sequence-parallel shared expert stay off (`VLLM_USE_NCCL_SYMM_MEM=0`, `VLLM_ENABLE_GEMM_RS=0`, `VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT=0`).\n- **Points.** `--max-num-seqs` is 2 at concurrency 1 and 6 at concurrency 6. The capture sizes cover 5 rows per admitted request. Size both to your own concurrency.\n- The **Simple** KV-offload option gets 281,125,000,000 bytes of host DRAM per rank. Size `cpu_bytes_to_use_per_rank` to your host.\n- `--no-enable-flashinfer-autotune`, `--max-num-batched-tokens 16384`, `--gpu-memory-utilization 0.9`, `--stream-interval 10` and `--disable-uvicorn-access-log` as in the lane. `VLLM_ENGINE_READY_TIMEOUT_S=7200` leaves time for the weight load and the specialized decode's JIT; `VLLM_HTTP_TIMEOUT_KEEP_ALIVE=900` keeps pooled client connections open between agent turns.\n\n**Benchmark-only settings left out.** The lane also sets `PYTHONHASHSEED=42`, so that identical prefixes hash to the same block keys from run to run, and `PYTHONUNBUFFERED=1` for its logs; neither changes serving. Its `--served-model-name` equals the model ID.\n\n### Throughput measurement only (synthetic acceptance)\n\nTo measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 4 draft tokens, 3.19. Only `--speculative-config` changes:\n\n```bash\n--speculative-config '{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":4,\"method\":\"dspark\",\"draft_sample_method\":\"probabilistic\",\"revision\":\"6a2ef20a05efe88aabbf9ecaa12d020c3b0edf94\",\"attention_backend\":\"TOKENSPEED_MLA\",\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":3.19}'\n```\n\nSynthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.\n",
      "json": "/moonshotai/Kimi-K3.1-Pro/presets/b300_agentx.json"
    },
    "b300_agentx_mixed_graphs": {
      "label": "AgentX TP8 + DCP8, mixed-step graphs",
      "hardware": "b300",
      "strategy": "single_node_tp",
      "workload": "Agentic (InferenceX AgentX)",
      "description": "Concurrency 12 and 24 on one B300 node: TP8 with DCP8, CUDA graphs for decode and for prefill and mixed steps up to 4096 tokens, prefill admitted every 8 engine steps, the low-latency decode backend, the K3.1 DSpark draft with 3 tokens, and host KV offload.",
      "source": {
        "label": "InferenceX #4871",
        "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4871",
        "key": "kimik3.1-fp4-b300-vllm-agentic-dspark",
        "revision": "75b1a3c7b745ddf77f79d00fc3a4c939bbcf8774"
      },
      "image": "vllm/vllm-openai:kimi-k31-mc@sha256:f81998764e8594d71b40788c5285ae112acf9230129d5d0d238b5df83da5acf6",
      "select": {
        "frontend": "python",
        "features": {
          "tool_calling": true,
          "reasoning": true,
          "spec_decoding": true,
          "text_only": true,
          "low_latency_decode": true,
          "nvfp4_kv_cache": false
        },
        "dcp": 8,
        "kv_offload": "simple"
      },
      "args": [
        "--decode-context-parallel-size",
        "8",
        "--dcp-comm-backend",
        "a2a",
        "--enable-prefix-caching",
        "--kv-cache-dtype",
        "fp8",
        "--mamba-ssm-cache-dtype",
        "float32",
        "--use-replayssm",
        "--prefill-schedule-interval",
        "8",
        "--max-num-seqs",
        "96",
        "--max-num-batched-tokens",
        "16384",
        "--gpu-memory-utilization",
        "0.95",
        "--stream-interval",
        "10",
        "--disable-uvicorn-access-log",
        "--attention-config",
        "{\"indexer_kv_dtype\":\"mxfp4\"}"
      ],
      "env": {
        "VLLM_ENGINE_READY_TIMEOUT_S": "7200",
        "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "900",
        "VLLM_USE_DIRECT_DCP_A2A": "1",
        "VLLM_USE_DIRECT_DCP_Q_GATHER": "1",
        "VLLM_USE_DIRECT_DCP_KV_GATHER": "1"
      },
      "json_args": {
        "--speculative-config": {
          "revision": "6a2ef20a05efe88aabbf9ecaa12d020c3b0edf94",
          "num_speculative_tokens": 3,
          "rejection_sample_method": "block"
        },
        "--kv-transfer-config": {
          "kv_connector_extra_config": {
            "cpu_bytes_to_use_per_rank": 281125000000
          }
        },
        "--compilation-config": {
          "max_cudagraph_capture_size": 4096,
          "cudagraph_capture_sizes": [
            1,
            2,
            4,
            8,
            12,
            16,
            20,
            24,
            28,
            32,
            36,
            40,
            44,
            48,
            52,
            56,
            60,
            64,
            68,
            72,
            76,
            80,
            84,
            88,
            92,
            96,
            100,
            104,
            108,
            112,
            116,
            120,
            124,
            128,
            132,
            136,
            140,
            144,
            148,
            152,
            156,
            160,
            164,
            168,
            172,
            176,
            180,
            184,
            188,
            192,
            196,
            200,
            204,
            208,
            212,
            216,
            220,
            224,
            228,
            232,
            236,
            240,
            244,
            248,
            252,
            256,
            260,
            264,
            268,
            272,
            276,
            280,
            284,
            288,
            292,
            296,
            300,
            304,
            308,
            312,
            316,
            320,
            324,
            328,
            332,
            336,
            340,
            344,
            348,
            352,
            356,
            360,
            364,
            368,
            372,
            376,
            380,
            384,
            512,
            640,
            768,
            1024,
            1280,
            1536,
            1792,
            2048,
            2560,
            3072,
            3584,
            4096
          ]
        }
      },
      "points": {
        "c12": {
          "label": "12",
          "concurrency": 12
        },
        "c24": {
          "label": "24",
          "concurrency": 24
        }
      },
      "runbook": "The concurrency-12 and -24 points of InferenceX's B300 vLLM AgentX lane (SemiAnalysisAI/InferenceX#4871, `agentic-mixed-graphs.yaml`), measured only on the image digest the preset pins. Both points run the same server config. It keeps the TP8 + DCP8 layout, the low-latency decode backend, FP8 KV with the MXFP4 indexer cache, `--use-replayssm`, the FP32 KDA recurrent state and the Simple KV offload of [AgentX TP8 + DCP8](#preset-b300_agentx), and changes the following:\n- **CUDA graphs also cover prefill and mixed steps up to 4096 tokens.** The default CUDA graph mode with `max_cudagraph_capture_size` 4096 captures 1, 2, then 4 to 384 in steps of 4 for decode, then 512 to 4096. InferenceX passes these as `--cudagraph-capture-sizes` and `--max-cudagraph-capture-size`, which set the same compilation-config fields.\n- **`--prefill-schedule-interval 8`.** Prefill is admitted once every 8 engine steps, so the specialized decode serves the pure decode steps in between. A long prompt takes up to 8 times as many engine steps to finish prefilling.\n- **Spec decoding** drafts 3 tokens on the draft's default attention backend, with probabilistic drafting and block rejection.\n- `--max-num-seqs 96` and `--gpu-memory-utilization 0.95`. This config leaves FlashInfer autotuning on and `--cp-kv-cache-interleave-size` at its default, and its env sets only the direct DCP paths, `VLLM_ENGINE_READY_TIMEOUT_S=7200` and `VLLM_HTTP_TIMEOUT_KEEP_ALIVE=900`; everything else stays at the image defaults.\n\nAs in the other B300 presets, the lane's `PYTHONHASHSEED=42` and `PYTHONUNBUFFERED=1` are benchmark-only and left out.\n\n### Throughput measurement only (synthetic acceptance)\n\nTo measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 3 draft tokens, 2.88. Only `--speculative-config` changes:\n\n```bash\n--speculative-config '{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":3,\"method\":\"dspark\",\"draft_sample_method\":\"probabilistic\",\"revision\":\"6a2ef20a05efe88aabbf9ecaa12d020c3b0edf94\",\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":2.88}'\n```\n\nSynthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.\n",
      "json": "/moonshotai/Kimi-K3.1-Pro/presets/b300_agentx_mixed_graphs.json"
    },
    "b300_agentx_ep": {
      "label": "AgentX TP8 + DCP8 + EP8",
      "hardware": "b300",
      "strategy": "single_node_tep",
      "workload": "Agentic (InferenceX AgentX)",
      "description": "Concurrency 48 and 64 on one B300 node: TP8 + DCP8 attention with routed experts EP8 on MegaMoE, the low-latency decode backend at every decode batch on FULL decode CUDA graphs, prefill admitted every 8 engine steps, the K3.1 DSpark draft with 4 tokens on TOKENSPEED_MLA, and host KV offload.",
      "source": {
        "label": "InferenceX #4871",
        "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4871",
        "key": "kimik3.1-fp4-b300-vllm-agentic-dspark",
        "revision": "75b1a3c7b745ddf77f79d00fc3a4c939bbcf8774"
      },
      "image": "vllm/vllm-openai:kimi-k31-mc@sha256:f81998764e8594d71b40788c5285ae112acf9230129d5d0d238b5df83da5acf6",
      "select": {
        "frontend": "python",
        "features": {
          "tool_calling": true,
          "reasoning": true,
          "spec_decoding": true,
          "text_only": true,
          "nvfp4_kv_cache": false
        },
        "dcp": 8,
        "kv_offload": "simple"
      },
      "args": [
        "--decode-context-parallel-size",
        "8",
        "--dcp-comm-backend",
        "a2a",
        "--cp-kv-cache-interleave-size",
        "1",
        "--enable-prefix-caching",
        "--kv-cache-dtype",
        "fp8",
        "--mamba-ssm-cache-dtype",
        "float32",
        "--use-replayssm",
        "--no-enable-flashinfer-autotune",
        "--additional-config",
        "{\"decode_backend\":\"latency_specialized\"}",
        "--prefill-schedule-interval",
        "8",
        "--max-num-batched-tokens",
        "16384",
        "--stream-interval",
        "10",
        "--disable-uvicorn-access-log",
        "--attention-config",
        "{\"indexer_kv_dtype\":\"mxfp4\",\"tokenspeed_mla_min_split_kv\":16}"
      ],
      "remove_args": [
        "--all2all-backend"
      ],
      "env": {
        "VLLM_ENGINE_READY_TIMEOUT_S": "7200",
        "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "900",
        "VLLM_USE_V2_MODEL_RUNNER": "1",
        "VLLM_USE_DIRECT_DCP_A2A": "1",
        "VLLM_USE_DIRECT_DCP_Q_GATHER": "1",
        "VLLM_USE_DIRECT_DCP_KV_GATHER": "1",
        "VLLM_ALLREDUCE_USE_FLASHINFER": "1",
        "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0",
        "VLLM_USE_NCCL_SYMM_MEM": "0",
        "VLLM_ENABLE_GEMM_RS": "0",
        "TOKENSPEED_MLA_FP16_PARTIALS": "0"
      },
      "json_args": {
        "--speculative-config": {
          "revision": "6a2ef20a05efe88aabbf9ecaa12d020c3b0edf94",
          "attention_backend": "TOKENSPEED_MLA",
          "num_speculative_tokens": 4,
          "rejection_sample_method": "block"
        },
        "--kv-transfer-config": {
          "kv_connector_extra_config": {
            "cpu_bytes_to_use_per_rank": 374875000000
          }
        },
        "--compilation-config": {
          "cudagraph_mode": "FULL_DECODE_ONLY"
        }
      },
      "points": {
        "c48": {
          "label": "48",
          "concurrency": 48,
          "args": [
            "--max-num-seqs",
            "48",
            "--gpu-memory-utilization",
            "0.97"
          ],
          "json_args": {
            "--compilation-config": {
              "cudagraph_capture_sizes": [
                5,
                10,
                15,
                20,
                25,
                30,
                35,
                40,
                45,
                50,
                55,
                60,
                65,
                70,
                75,
                80,
                85,
                90,
                95,
                100,
                105,
                110,
                115,
                120,
                125,
                130,
                135,
                140,
                145,
                150,
                155,
                160,
                165,
                170,
                175,
                180,
                185,
                190,
                195,
                200,
                205,
                210,
                215,
                220,
                225,
                230,
                235,
                240
              ]
            }
          }
        },
        "c64": {
          "label": "64",
          "concurrency": 64,
          "args": [
            "--max-num-seqs",
            "64",
            "--gpu-memory-utilization",
            "0.97"
          ],
          "json_args": {
            "--compilation-config": {
              "cudagraph_capture_sizes": [
                5,
                10,
                15,
                20,
                25,
                30,
                35,
                40,
                45,
                50,
                55,
                60,
                65,
                70,
                75,
                80,
                85,
                90,
                95,
                100,
                105,
                110,
                115,
                120,
                125,
                130,
                135,
                140,
                145,
                150,
                155,
                160,
                165,
                170,
                175,
                180,
                185,
                190,
                195,
                200,
                205,
                210,
                215,
                220,
                225,
                230,
                235,
                240,
                245,
                250,
                255,
                260,
                265,
                270,
                275,
                280,
                285,
                290,
                295,
                300,
                305,
                310,
                315,
                320
              ]
            }
          }
        }
      },
      "runbook": "The concurrency 48 and 64 points of InferenceX's B300 vLLM AgentX lane (SemiAnalysisAI/InferenceX#4871, `agentic-ep.yaml`), measured only on the image digest the preset pins. It is [AgentX TP8 + DCP8](#preset-b300_agentx) with routed experts on expert parallelism, and changes the following:\n- **Routed experts EP8 on MegaMoE.** Attention stays TP8 + DCP8; the Tensor + Expert Parallel strategy adds `--enable-expert-parallel` and `--moe-backend deep_gemm_mega_moe`. The lane keeps the default all-to-all backend, so the preset drops the strategy's `--all2all-backend`.\n- **Low-latency decode at every decode batch.** The specialized decode also runs this hybrid layout (TP8 attention, routed experts EP over every rank) on MegaMoE, with `VLLM_ENABLE_GEMM_RS=0`. The Low-latency decode toggle is offered only without expert parallelism, so the preset sets `--additional-config '{\"decode_backend\":\"latency_specialized\"}'` itself.\n- **`--prefill-schedule-interval 8`.** Prefill is admitted once every 8 engine steps, so the specialized decode serves the pure decode steps in between.\n- **Points.** `--max-num-seqs` equals the concurrency, and FULL_DECODE_ONLY graphs capture 5 rows (1 + 4 draft tokens) per request up to it. `--gpu-memory-utilization` is 0.97 at both points.\n- `tokenspeed_mla_min_split_kv` (the minimum number of KV splits of the draft's TOKENSPEED_MLA decode attention) is 16. With `VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT=0`, sequence-parallel MoE replicates the shared expert on every rank, which costs weight memory on each GPU.\n- The **Simple** KV-offload option gets 374,875,000,000 bytes of host DRAM per rank. Size `cpu_bytes_to_use_per_rank` to your host.\n\nAs in the other B300 presets, the lane's `PYTHONHASHSEED=42` and `PYTHONUNBUFFERED=1` are benchmark-only and left out.\n\n### Throughput measurement only (synthetic acceptance)\n\nTo measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 4 draft tokens, 3.19. Only `--speculative-config` changes:\n\n```bash\n--speculative-config '{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":4,\"method\":\"dspark\",\"draft_sample_method\":\"probabilistic\",\"revision\":\"6a2ef20a05efe88aabbf9ecaa12d020c3b0edf94\",\"attention_backend\":\"TOKENSPEED_MLA\",\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":3.19}'\n```\n\nSynthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.\n",
      "json": "/moonshotai/Kimi-K3.1-Pro/presets/b300_agentx_ep.json"
    },
    "v7x_validated": {
      "label": "Validated TEP32 (v7x)",
      "hardware": "ironwood",
      "strategy": "multi_node_tep",
      "workload": "Accuracy validation (GSM8K)",
      "description": "The #2775 validation config on a 2x2x4 v7x slice: complete GSM8K at 97.50%, prefix-cache hits returning identical tokens and logprobs, BEAM-1M matching the GPU reference.",
      "select": {
        "nodes": 4
      },
      "args": [
        "--max-model-len",
        "10240",
        "--max-num-batched-tokens",
        "8192",
        "--max-num-seqs",
        "16",
        "--gpu-memory-utilization",
        "0.82",
        "--compilation-config",
        "{\"compile_sizes\":[1,8,16,8192]}"
      ],
      "source": {
        "href": "https://github.com/vllm-project/vllm-torchtpu/pull/2775",
        "key": "kimi-k31-pro-v7x-tp32-ep32",
        "revision": "af0fce9305f4e591e54892153ea86d49deaf1ccb"
      },
      "runbook": "The configuration vllm-torchtpu#2775 validated on the real checkpoint on a 2x2x4 TPU v7x slice: complete GSM8K at **97.50%** (1,286/1,319), prefix-cache probes returning identical tokens and logprobs, and BEAM-1M long context matching the GPU reference.\n\n#2775 ran one `vllm serve` driver over a Ray cluster (`TPU_MULTIHOST_BACKEND=ray`); the builder renders torchtpu's default multiprocessing backend (per-host `--nnodes` / `--node-rank` / `--headless`). The flags match; the multi-host transport differs. GSM8K ran through the completions API, so the default-on chat and tool-call parsers were not exercised on TPU.\n\nGSM8K capped outputs at 1,024 tokens with 8 concurrent requests; `--max-model-len 10240` and `--max-num-seqs 16` come from that run, not from a serving-quality target.\n",
      "json": "/moonshotai/Kimi-K3.1-Pro/presets/v7x_validated.json"
    },
    "v7x_dcp8": {
      "label": "Validated TEP32 + DCP8 (v7x)",
      "hardware": "ironwood",
      "strategy": "multi_node_tep",
      "workload": "Accuracy validation (GSM8K)",
      "description": "TEP32 + DCP8 on the same 2x2x4 v7x slice: complete GSM8K at 97.42%. Pick it when KV capacity, not decode speed, is the limit. DSpark does not support DCP on TPU, so this preset keeps Spec decoding off.",
      "select": {
        "nodes": 4,
        "features": {
          "spec_decoding": false
        }
      },
      "args": [
        "--decode-context-parallel-size",
        "8",
        "--cp-kv-cache-interleave-size",
        "4",
        "--max-model-len",
        "10240",
        "--max-num-batched-tokens",
        "8192",
        "--max-num-seqs",
        "16",
        "--gpu-memory-utilization",
        "0.82",
        "--compilation-config",
        "{\"compile_sizes\":[8,16,8192]}"
      ],
      "source": {
        "href": "https://github.com/vllm-project/vllm-torchtpu/pull/2775",
        "key": "kimi-k31-pro-v7x-tp32-ep32-dcp8",
        "revision": "af0fce9305f4e591e54892153ea86d49deaf1ccb"
      },
      "runbook": "The TP32 + DCP8 arm of the vllm-torchtpu#2775 validation on the same slice: complete GSM8K at **97.42%** (1,285/1,319), with prefix-cache and long-context checks matching the TP32 arm. Compile sizes drop the 1-batch capture (DCP decode batches start at 8).\n",
      "json": "/moonshotai/Kimi-K3.1-Pro/presets/v7x_dcp8.json"
    },
    "gb300_agentx_pd": {
      "label": "AgentX 1P3D TP8 + DCP8",
      "hardware": "gb300",
      "strategy": "pd_cluster",
      "workload": "Agentic (InferenceX AgentX)",
      "description": "Concurrency 20 to 56 on 32 GPUs: one TP8 + DCP8 prefill worker with EP8 on MegaMoE and three TP8 + DCP8 decode workers on the low-latency decode backend, two GB300 trays each, served by Dynamo with the K3.1 DSpark draft (4 tokens) and a shared Mooncake KV store in host DRAM.",
      "source": {
        "label": "InferenceX #4907",
        "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4907",
        "key": "kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg",
        "revision": "f2bb2756fffac8fc8e5d241fe0f936e32c3c3e90"
      },
      "image": {
        "vllm": "vllm/vllm-openai:kimi-k31-mc@sha256:f29efae15f9cb47f6215add0cba70b4b7d3596e6f3990d8b46c7cad0f8fbbc8b",
        "dynamo": "3dd0744eb72c707d76d4c520ae4dea9e286a4ab6"
      },
      "select": {
        "pd_router": "dynamo",
        "frontend": "rust",
        "features": {
          "tool_calling": true,
          "reasoning": true,
          "spec_decoding": true,
          "text_only": true,
          "low_latency_decode": false,
          "nvfp4_kv_cache": false
        },
        "dcp": 8,
        "kv_offload": "kv_store_distributed_mooncake"
      },
      "args": [
        "--decode-context-parallel-size",
        "8",
        "--dcp-comm-backend",
        "a2a",
        "--cp-kv-cache-interleave-size",
        "1",
        "--kv-cache-dtype",
        "fp8",
        "--attention-config",
        "{\"indexer_kv_dtype\":\"mxfp4\"}",
        "--mamba-cache-dtype",
        "float32",
        "--mamba-ssm-cache-dtype",
        "float32",
        "--enable-prefix-caching",
        "--prefix-match-unit",
        "128",
        "--gpu-memory-utilization",
        "0.95",
        "--enable-cumem-allocator",
        "--no-enable-flashinfer-autotune",
        "--stream-interval",
        "10",
        "--dyn-enable-structural-tag"
      ],
      "env": {
        "VLLM_USE_V2_MODEL_RUNNER": "1",
        "VLLM_USE_DIRECT_DCP_A2A": "1",
        "VLLM_USE_DIRECT_DCP_Q_GATHER": "1",
        "VLLM_USE_DIRECT_DCP_KV_GATHER": "1",
        "VLLM_ALLREDUCE_USE_FLASHINFER": "1",
        "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0",
        "VLLM_USE_NCCL_SYMM_MEM": "0",
        "TOKENSPEED_MLA_FP16_PARTIALS": "0",
        "NCCL_CUMEM_ENABLE": "1",
        "NCCL_MNNVL_ENABLE": "1",
        "NCCL_NVLS_ENABLE": "1",
        "NCCL_P2P_LEVEL": "NVL",
        "NCCL_NET_PLUGIN": "none",
        "UCX_MEMTYPE_REG_WHOLE": "n",
        "UCX_TCP_AF_PRIO": "inet",
        "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4",
        "VLLM_MOONCAKE_STORE_SELECTIVE_LEASE": "1",
        "MC_SLICE_SIZE": "1048576",
        "MC_MAX_MR_SIZE": "4294967296",
        "MC_ENABLE_DEST_DEVICE_AFFINITY": "1",
        "WITH_NVIDIA_PEERMEM": "0",
        "DYN_TCP_REQUEST_TIMEOUT": "120",
        "ETCD_LEASE_TTL": "600",
        "VLLM_RPC_TIMEOUT": "600000",
        "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800",
        "TILELANG_CLEANUP_TEMP_FILES": "1"
      },
      "json_args": {
        "--speculative-config": {
          "num_speculative_tokens": 4,
          "rejection_sample_method": "block"
        },
        "--kv-transfer-config": {
          "kv_load_failure_policy": "recompute",
          "NixlConnector": {
            "kv_buffer_device": "cuda",
            "kv_connector_extra_config": {
              "enforce_handshake_compat": false,
              "enable_cross_layers_blocks": false
            }
          },
          "MooncakeStoreConnector": {
            "kv_load_failure_policy": "recompute",
            "kv_connector_extra_config": {
              "enable_cross_layers_blocks": false,
              "enable_offload": false
            }
          }
        }
      },
      "roles": {
        "prefill": {
          "parallelism": "tep",
          "nodes": 2,
          "args": [
            "--moe-backend",
            "deep_gemm_mega_moe",
            "--max-num-seqs",
            "32",
            "--max-cudagraph-capture-size",
            "8192",
            "--compilation-config",
            "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}"
          ],
          "env": {
            "VLLM_ENABLE_GEMM_RS": "1",
            "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024",
            "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024"
          }
        },
        "decode": {
          "parallelism": "tp",
          "nodes": 2,
          "args": [
            "--additional-config",
            "{\"decode_backend\":\"latency_specialized\"}",
            "--use-replayssm",
            "--max-num-seqs",
            "64",
            "--max-cudagraph-capture-size",
            "320",
            "--compilation-config",
            "{\"cudagraph_capture_sizes\":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195,200,205,210,215,220,225,230,235,240,245,250,255,260,265,270,275,280,285,290,295,300,305,310,315,320]}"
          ],
          "env": {
            "VLLM_ENABLE_GEMM_RS": "0"
          }
        }
      },
      "points": {
        "c20": {
          "label": "20 (1P3D)",
          "concurrency": 20,
          "description": "One prefill and three decode workers (32 GPUs), 64 decode sequences.",
          "args": [
            "--numa-bind",
            "--numa-bind-nodes",
            "0",
            "0",
            "1",
            "1"
          ]
        },
        "c32": {
          "label": "32 (1P3D)",
          "concurrency": 32,
          "description": "One prefill and three decode workers (32 GPUs), 64 decode sequences.",
          "args": [
            "--numa-bind",
            "--numa-bind-nodes",
            "0",
            "0",
            "1",
            "1"
          ]
        },
        "c56": {
          "label": "56 (1P3D)",
          "concurrency": 56,
          "description": "One prefill and three decode workers (32 GPUs), 256 decode sequences.",
          "env": {
            "VLLM_WORKER_MULTIPROC_METHOD": "spawn"
          },
          "roles": {
            "decode": {
              "args": [
                "--max-num-seqs",
                "256",
                "--max-num-batched-tokens",
                "1280",
                "--max-cudagraph-capture-size",
                "1280",
                "--compilation-config",
                "{\"cudagraph_capture_sizes\":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195,200,205,210,215,220,225,230,235,240,245,250,255,260,265,270,275,280,285,290,295,300,305,310,315,320,340,360,380,400,420,440,460,480,500,520,540,560,580,600,620,640,660,680,700,720,740,760,780,800,820,840,860,880,900,920,940,960,980,1000,1020,1040,1060,1080,1100,1120,1140,1160,1180,1200,1220,1240,1260,1280]}"
              ]
            }
          }
        }
      },
      "runbook": "The concurrency 20, 32 and 56 points of the GB300 AgentX disaggregated lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg`, [SemiAnalysisAI/InferenceX#4907](https://github.com/SemiAnalysisAI/InferenceX/pull/4907)). Dynamo + vLLM run one prefill worker and three decode workers, each TP8 + DCP8 across two 4-GPU GB300 trays (32 GPUs). The prefill worker also runs the routed experts EP8 on MegaMoE; the decode workers use no expert parallelism and run the low-latency decode backend. KV moves from prefill to decode over NIXL, and all workers share an RDMA Mooncake store in host DRAM. The lane ran the image digest the preset pins, Dynamo built from `3dd0744eb72c707d76d4c520ae4dea9e286a4ab6` ([ai-dynamo/dynamo#18124](https://github.com/ai-dynamo/dynamo/pull/18124), the first Dynamo revision that registers the K3.1 model type) and Mooncake `0.3.14`; the preset has been measured only on that combination.\n\n| Clients | Prefill workers | Decode workers | GPUs | Decode `--max-num-seqs` | Decode capture sizes | CPU binding |\n|---:|---:|---:|---:|---:|---|---|\n| 20, 32 | 1 | 3 | 32 | 64 | 5 to 320 in steps of 5 | `--numa-bind` |\n| 56 | 1 | 3 | 32 | 256 | 5 to 320 in steps of 5, then 340 to 1280 in steps of 20 | `VLLM_WORKER_MULTIPROC_METHOD=spawn` |\n\nEach point renders one prefill worker and one decode worker; start the table's number of workers per role with the same commands, two trays each. Run every worker command on both of its trays with the same `--master-addr` (the first tray's address), `--nnodes 2` and `--node-rank 0` or `1`.\n\n**Serving config.**\n- **Prefill:** TP8 + DCP8 with `--enable-expert-parallel` and `--moe-backend deep_gemm_mega_moe`, `--max-num-seqs 32`, and `FULL_AND_PIECEWISE` CUDA graphs for 4 to 8192 tokens. `VLLM_ENABLE_GEMM_RS=1` turns on the fused GEMM + reduce-scatter. `DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT` and `DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT` set Dynamo's admission limits for the prefill engine to 1024 requests each.\n- **Decode:** TP8 + DCP8 without expert parallelism, with `VLLM_ENABLE_GEMM_RS=0`. `--additional-config '{\"decode_backend\":\"latency_specialized\"}'` runs pure decode steps on the fused low-latency path; the Low-latency decode toggle is not offered under PD, so the preset sets the flag in the decode args. `--use-replayssm` makes KDA verification replay the accepted prefix instead of keeping one recurrent state per draft token. The decode `--compilation-config` sets only the capture sizes (multiples of 5 rows: 1 + 4 draft tokens per request) and replaces the PD strategy's `FULL_DECODE_ONLY`, so the CUDA graph mode is vLLM's default, as in the lane.\n- **Both roles:** FP8 MLA KV cache with an MXFP4 indexer key cache (`indexer_kv_dtype`); FP32 KDA conv and recurrent state (`--mamba-cache-dtype float32 --mamba-ssm-cache-dtype float32`), as the K3.1 precision policy requires; the A2A DCP backend and the direct DCP all-to-all and gather paths (`VLLM_USE_DIRECT_DCP_*`); prefix caching with `--prefix-match-unit 128`; `--gpu-memory-utilization 0.95`, `--enable-cumem-allocator`, `--no-enable-flashinfer-autotune` and `--stream-interval 10`. Both roles pin `--cp-kv-cache-interleave-size 1`: left unset, NIXL P/D raises the DCP KV interleave after the specialized decode has read it.\n- **Spec decoding** (on in this preset) uses the K3.1 DSpark draft [`Inferact/Kimi-K3.1-Pro-DSpark-preview`](https://huggingface.co/Inferact/Kimi-K3.1-Pro-DSpark-preview) with 4 tokens, probabilistic drafting and block rejection.\n- **Collectives and kernels:** FlashInfer all-reduce stays on (`VLLM_ALLREDUCE_USE_FLASHINFER=1`); NCCL symmetric memory and the sharded sequence-parallel shared expert stay off (`VLLM_USE_NCCL_SYMM_MEM=0`, `VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT=0`), and so do TOKENSPEED_MLA's FP16 partials (`TOKENSPEED_MLA_FP16_PARTIALS=0`). NCCL runs over the NVLink domain (`NCCL_MNNVL_ENABLE=1`, `NCCL_NVLS_ENABLE=1`, `NCCL_CUMEM_ENABLE=1`, `NCCL_P2P_LEVEL=NVL`) without a network plugin (`NCCL_NET_PLUGIN=none`).\n- **KV transfer:** NIXL uses `UCX_MEMTYPE_REG_WHOLE=n` and `UCX_TCP_AF_PRIO=inet`. The Mooncake transfer engine uses `MC_SLICE_SIZE=1048576`, `MC_MAX_MR_SIZE=4294967296` and `MC_ENABLE_DEST_DEVICE_AFFINITY=1`, and `WITH_NVIDIA_PEERMEM=0` for hosts without the nvidia-peermem module. vLLM loads from the store with 4 receive threads (`VLLM_MOONCAKE_LOAD_RECV_THREADS=4`), and `VLLM_MOONCAKE_STORE_SELECTIVE_LEASE=1` makes a store lookup lease only the keys that a hit loads.\n- **Timeouts and housekeeping:** `VLLM_ENGINE_READY_TIMEOUT_S=3600` (from the recipe's Blackwell settings), `VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800`, `VLLM_RPC_TIMEOUT=600000`, `DYN_TCP_REQUEST_TIMEOUT=120` and `ETCD_LEASE_TTL=600`. `TILELANG_CLEANUP_TEMP_FILES=1` removes TileLang's temporary build files.\n- **CPU binding:** at 20 and 32 clients, `--numa-bind --numa-bind-nodes 0 0 1 1` binds each worker process to its GPU's NUMA node (GPUs 0 and 1 on node 0, GPUs 2 and 3 on node 1 of a tray). At 56 clients, workers start with `VLLM_WORKER_MULTIPROC_METHOD=spawn` and each tray's GPUs are pinned with `CUDA_VISIBLE_DEVICES`, as the rendered commands do.\n\n**Tool calling and reasoning** are on in this preset. Under Dynamo the builder renders `--dyn-tool-call-parser kimi_k3`, `--dyn-reasoning-parser kimi_k3` and vLLM's `--reasoning-parser kimi_k3`, and the preset adds `--dyn-enable-structural-tag`: the flags of the lane's accuracy runs.\n\nThe preset selects the distributed (embedded) Mooncake store on the **KV Offload** row, which wraps NIXL and the `MooncakeStoreConnector` in one `MultiConnector` on both roles; the preset sets the lane's connector options on it (recompute on a failed store load, the NIXL KV buffer on the GPU, cross-layer blocks and store offload off). Documented here only: the worker counts, the lane's Mooncake master options and store sizing (the builder's Mooncake option shows its own defaults), and the frontend's router mode and session affinity (the builder renders round-robin).\n\n### Services\n\nStart etcd, NATS and a Mooncake master once:\n\n```bash\netcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<infra-host>:2379\nprintf 'max_payload: 33554432\\n' > nats.conf && nats-server -c nats.conf\nmooncake_master --default_kv_lease_ttl=60000 --eviction_high_watermark_ratio=0.95 --eviction_ratio=0.10\n```\n\nEvery worker reads the Mooncake store configuration from the JSON file that `MOONCAKE_CONFIG_PATH` points to. The lane registered 160GB of host DRAM per GPU:\n\n```json\n{\"master_server_address\": \"<infra-host>:<master-port>\", \"metadata_server\": \"P2PHANDSHAKE\",\n \"global_segment_size\": \"160GB\", \"local_buffer_size\": \"4GB\", \"protocol\": \"rdma\",\n \"device_name\": \"<the tray's RDMA devices>\", \"mode\": \"embedded\", \"enable_offload\": false}\n```\n\n### Frontend\n\n```bash\nexport DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TOKENIZER_CACHE_BYTES=8589934592 DYN_TCP_REQUEST_TIMEOUT=120\nexport DYN_DISABLE_OPENAI_STRICT_SCHEMA_VALIDATION=1\npython3 -m dynamo.frontend --router-mode least-loaded --router-session-affinity-ttl-secs 1\n```\n\n`DYN_DISABLE_OPENAI_STRICT_SCHEMA_VALIDATION=1` skips the frontend's preflight validation of tool parameter schemas marked `strict: true`. Session affinity keys on the `X-Dynamo-Session-Id` request header, so send one ID per conversation.\n\n### Benchmark-only settings left out\n\nThe lane also passes `--served-model-name moonshotai/Kimi-K3.1-Pro`, which equals the model ID here, and sets `VLLM_SERVER_DEV_MODE=1`, which turns on vLLM's development-only endpoints; serving needs neither. Its throughput runs leave out the tool-call and reasoning parser flags. Its job settings (model cache path, health-check polling, telemetry and the benchmark client's own environment) are not server settings.\n\n### Throughput measurement only (synthetic acceptance)\n\nTo measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 4 draft tokens, 3.19. Only `--speculative-config` changes, on both roles:\n\n```bash\n--speculative-config '{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":4,\"method\":\"dspark\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":3.19}'\n```\n\nSynthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.\n",
      "json": "/moonshotai/Kimi-K3.1-Pro/presets/gb300_agentx_pd.json"
    },
    "gb300_agentx_pd_hybrid": {
      "label": "AgentX xP1D, DP4 x TP8 decode",
      "hardware": "gb300",
      "strategy": "pd_cluster",
      "workload": "Agentic (InferenceX AgentX)",
      "description": "Concurrency 192 and 384: two or three TP8 + DCP8 prefill workers with EP8 on MegaMoE, and one decode worker across eight GB300 trays that runs four TP8 + DCP8 data-parallel ranks with EP32 on MegaMoE and the low-latency decode backend, served by Dynamo with the K3.1 DSpark draft and a shared Mooncake KV store in host DRAM.",
      "source": {
        "label": "InferenceX #4907",
        "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4907",
        "key": "kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg",
        "revision": "f2bb2756fffac8fc8e5d241fe0f936e32c3c3e90"
      },
      "image": {
        "vllm": "vllm/vllm-openai:kimi-k31-mc@sha256:f29efae15f9cb47f6215add0cba70b4b7d3596e6f3990d8b46c7cad0f8fbbc8b",
        "dynamo": "3dd0744eb72c707d76d4c520ae4dea9e286a4ab6"
      },
      "select": {
        "pd_router": "dynamo",
        "frontend": "rust",
        "features": {
          "tool_calling": true,
          "reasoning": true,
          "spec_decoding": true,
          "text_only": true,
          "low_latency_decode": false,
          "nvfp4_kv_cache": false
        },
        "dcp": 8,
        "kv_offload": "kv_store_distributed_mooncake"
      },
      "args": [
        "--decode-context-parallel-size",
        "8",
        "--dcp-comm-backend",
        "a2a",
        "--cp-kv-cache-interleave-size",
        "1",
        "--kv-cache-dtype",
        "fp8",
        "--attention-config",
        "{\"indexer_kv_dtype\":\"mxfp4\"}",
        "--mamba-cache-dtype",
        "float32",
        "--mamba-ssm-cache-dtype",
        "float32",
        "--enable-prefix-caching",
        "--prefix-match-unit",
        "128",
        "--gpu-memory-utilization",
        "0.95",
        "--enable-cumem-allocator",
        "--no-enable-flashinfer-autotune",
        "--stream-interval",
        "10",
        "--dyn-enable-structural-tag"
      ],
      "env": {
        "VLLM_USE_V2_MODEL_RUNNER": "1",
        "VLLM_USE_DIRECT_DCP_A2A": "1",
        "VLLM_USE_DIRECT_DCP_Q_GATHER": "1",
        "VLLM_USE_DIRECT_DCP_KV_GATHER": "1",
        "VLLM_ALLREDUCE_USE_FLASHINFER": "1",
        "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0",
        "VLLM_USE_NCCL_SYMM_MEM": "0",
        "TOKENSPEED_MLA_FP16_PARTIALS": "0",
        "NCCL_CUMEM_ENABLE": "1",
        "NCCL_MNNVL_ENABLE": "1",
        "NCCL_NVLS_ENABLE": "1",
        "NCCL_P2P_LEVEL": "NVL",
        "NCCL_NET_PLUGIN": "none",
        "UCX_MEMTYPE_REG_WHOLE": "n",
        "UCX_TCP_AF_PRIO": "inet",
        "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4",
        "VLLM_MOONCAKE_STORE_SELECTIVE_LEASE": "1",
        "MC_SLICE_SIZE": "1048576",
        "MC_MAX_MR_SIZE": "4294967296",
        "MC_ENABLE_DEST_DEVICE_AFFINITY": "1",
        "WITH_NVIDIA_PEERMEM": "0",
        "DYN_TCP_REQUEST_TIMEOUT": "120",
        "ETCD_LEASE_TTL": "600",
        "VLLM_RPC_TIMEOUT": "600000",
        "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800",
        "TILELANG_CLEANUP_TEMP_FILES": "1"
      },
      "json_args": {
        "--speculative-config": {
          "num_speculative_tokens": 4,
          "rejection_sample_method": "block"
        },
        "--kv-transfer-config": {
          "kv_load_failure_policy": "recompute",
          "NixlConnector": {
            "kv_buffer_device": "cuda",
            "kv_connector_extra_config": {
              "enforce_handshake_compat": false,
              "enable_cross_layers_blocks": false
            }
          },
          "MooncakeStoreConnector": {
            "kv_load_failure_policy": "recompute",
            "kv_connector_extra_config": {
              "enable_cross_layers_blocks": false,
              "enable_offload": false
            }
          }
        }
      },
      "roles": {
        "prefill": {
          "parallelism": "tep",
          "nodes": 2,
          "args": [
            "--moe-backend",
            "deep_gemm_mega_moe",
            "--max-num-seqs",
            "32",
            "--max-num-batched-tokens",
            "16384",
            "--max-cudagraph-capture-size",
            "8192",
            "--compilation-config",
            "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}"
          ],
          "env": {
            "VLLM_ENABLE_GEMM_RS": "1",
            "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024",
            "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024",
            "VLLM_WORKER_MULTIPROC_METHOD": "spawn"
          }
        },
        "decode": {
          "parallelism": "tep",
          "nodes": 8,
          "args": [
            "--tensor-parallel-size",
            "8",
            "--data-parallel-size",
            "4",
            "--data-parallel-address",
            "$DECODE_NODE_1",
            "--data-parallel-rpc-port",
            "13345",
            "--moe-backend",
            "deep_gemm_mega_moe",
            "--gpu-memory-utilization",
            "0.92",
            "--additional-config",
            "{\"decode_backend\":\"latency_specialized\"}",
            "--use-replayssm",
            "--max-num-seqs",
            "256"
          ],
          "env": {
            "VLLM_ENABLE_GEMM_RS": "0",
            "VLLM_WORKER_MULTIPROC_METHOD": "spawn"
          }
        }
      },
      "points": {
        "c192": {
          "label": "192 (2P1D)",
          "concurrency": 192,
          "description": "Two prefill workers and one DP4 x TP8 decode worker (48 GPUs), 6 draft tokens.",
          "json_args": {
            "--speculative-config": {
              "num_speculative_tokens": 6
            }
          },
          "roles": {
            "prefill": {
              "args": [
                "--long-prefill-token-threshold",
                "4096",
                "--long-prefill-token-threshold-adaptive"
              ]
            },
            "decode": {
              "args": [
                "--max-num-batched-tokens",
                "1792",
                "--max-cudagraph-capture-size",
                "1792",
                "--compilation-config",
                "{\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,105,112,119,126,133,140,147,154,161,168,175,182,189,196,203,210,217,224,231,238,245,252,259,266,273,280,287,294,301,308,315,322,329,336,343,350,357,364,371,378,385,392,399,406,413,420,427,434,441,448,476,504,532,560,588,616,644,672,700,728,756,784,812,840,868,896,924,952,980,1008,1036,1064,1092,1120,1148,1176,1204,1232,1260,1288,1316,1344,1372,1400,1428,1456,1484,1512,1540,1568,1596,1624,1652,1680,1708,1736,1764,1792]}"
              ]
            }
          }
        },
        "c384": {
          "label": "384 (3P1D)",
          "concurrency": 384,
          "description": "Three prefill workers and one DP4 x TP8 decode worker (56 GPUs), 4 draft tokens.",
          "roles": {
            "decode": {
              "args": [
                "--max-num-batched-tokens",
                "1280",
                "--max-cudagraph-capture-size",
                "1280",
                "--compilation-config",
                "{\"cudagraph_capture_sizes\":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195,200,205,210,215,220,225,230,235,240,245,250,255,260,265,270,275,280,285,290,295,300,305,310,315,320,340,360,380,400,420,440,460,480,500,520,540,560,580,600,620,640,660,680,700,720,740,760,780,800,820,840,860,880,900,920,940,960,980,1000,1020,1040,1060,1080,1100,1120,1140,1160,1180,1200,1220,1240,1260,1280]}"
              ]
            }
          }
        }
      },
      "runbook": "The concurrency 192 and 384 points of the GB300 AgentX disaggregated lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg`, [SemiAnalysisAI/InferenceX#4907](https://github.com/SemiAnalysisAI/InferenceX/pull/4907)), on the image, Dynamo revision and Mooncake version of the [AgentX 1P3D TP8 + DCP8 preset](#preset-gb300_agentx_pd). The prefill workers are that preset's: TP8 + DCP8 with EP8 on MegaMoE across two trays each. The single decode worker spans eight trays (32 GPUs) and runs four data-parallel ranks, each TP8 + DCP8 across two trays, with the routed experts EP32 on MegaMoE across all 32 GPUs and the low-latency decode backend. KV moves over NIXL, and all workers share the RDMA Mooncake store in host DRAM.\n\n| Clients | Prefill workers | Decode workers | GPUs | Draft tokens | Decode `--max-num-batched-tokens` and largest capture size | Prefill chunking |\n|---:|---:|---:|---:|---:|---:|---|\n| 192 | 2 | 1 | 48 | 6 | 1792 | `--long-prefill-token-threshold 4096`, adaptive |\n| 384 | 3 | 1 | 56 | 4 | 1280 | default |\n\nEach point renders one prefill worker and the decode worker; start the table's number of prefill workers with the same command, two trays each, with the same `--master-addr` (the first tray's address), `--nnodes 2` and `--node-rank 0` or `1`. Run the decode command on each of its eight trays with `--nnodes 8`, `--node-rank 0` to `7`, and the first tray's address as both `--master-addr` and `--data-parallel-address`; ranks 1 to 7 add `--headless`, as rendered.\n\n**Changes from the [1P3D preset](#preset-gb300_agentx_pd).**\n- **Decode layout.** The builder renders the decode pool as one TEP engine across eight trays (`--tensor-parallel-size 32 --enable-expert-parallel`); the decode args `--tensor-parallel-size 8 --data-parallel-size 4` replace its TP size, so the engine runs four TP8 data-parallel ranks with DCP8 inside each, and expert parallelism spans all 32 GPUs. `--data-parallel-rpc-port 13345` is the lane's choice; any free port works.\n- **Decode batch.** MegaMoE (`--moe-backend deep_gemm_mega_moe`) on decode as well, `--max-num-seqs 256` per data-parallel rank, `--gpu-memory-utilization 0.92`, and `--max-num-batched-tokens` and CUDA graph sizes up to 256 × (1 + draft tokens): multiples of 7 up to 448 and then of 28 up to 1792 with 6 draft tokens, multiples of 5 up to 320 and then of 20 up to 1280 with 4.\n- **Prefill.** `--max-num-batched-tokens 16384`. At 192 clients the prefill workers also chunk long prompts with `--long-prefill-token-threshold 4096 --long-prefill-token-threshold-adaptive`, and both roles draft 6 tokens; at 384 clients both roles draft 4.\n- **Worker start.** Every worker starts with `VLLM_WORKER_MULTIPROC_METHOD=spawn`, and each tray's GPUs are pinned with `CUDA_VISIBLE_DEVICES`, as the rendered commands do; no `--numa-bind`.\n\nEverything else (env, connector options, tool-call and reasoning parsers, services and frontend) is as in the [1P3D preset](#preset-gb300_agentx_pd), including its benchmark-only settings left out.\n\n### Throughput measurement only (synthetic acceptance)\n\nTo measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for the draft length: 3.57 for 6 draft tokens (192 clients) and 3.19 for 4 (384 clients). Only `--speculative-config` changes, on both roles:\n\n```bash\n# 192 clients\n--speculative-config '{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":6,\"method\":\"dspark\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":3.57}'\n# 384 clients\n--speculative-config '{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":4,\"method\":\"dspark\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":3.19}'\n```\n\nSynthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.\n",
      "json": "/moonshotai/Kimi-K3.1-Pro/presets/gb300_agentx_pd_hybrid.json"
    },
    "gb300_agentx_pd_dep16": {
      "label": "AgentX 1P1D DEP16",
      "hardware": "gb300",
      "strategy": "pd_cluster",
      "workload": "Agentic (InferenceX AgentX)",
      "description": "Concurrency 352 on 32 GPUs: one DEP16 prefill worker and one DEP16 decode worker (TP1 x DP16 with EP16 on MegaMoE), four GB300 trays each, with the low-latency decode backend, served by Dynamo with the K3.1 DSpark draft (4 tokens) and a shared Mooncake KV store in host DRAM.",
      "source": {
        "label": "InferenceX #4907",
        "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4907",
        "key": "kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dep16-disagg",
        "revision": "f2bb2756fffac8fc8e5d241fe0f936e32c3c3e90"
      },
      "image": {
        "vllm": "vllm/vllm-openai:kimi-k31-mc@sha256:5c243ce00d2d289a295065e3d794d5beb783e9c9a85eba3d2eef7024f822a531",
        "dynamo": "3dd0744eb72c707d76d4c520ae4dea9e286a4ab6"
      },
      "select": {
        "pd_router": "dynamo",
        "frontend": "rust",
        "features": {
          "tool_calling": true,
          "reasoning": true,
          "spec_decoding": true,
          "text_only": true,
          "low_latency_decode": false,
          "nvfp4_kv_cache": false
        },
        "kv_offload": "kv_store_distributed_mooncake"
      },
      "args": [
        "--moe-backend",
        "deep_gemm_mega_moe",
        "--block-size",
        "16384",
        "--kv-cache-dtype",
        "fp8",
        "--attention-config",
        "{\"indexer_kv_dtype\":\"mxfp4\"}",
        "--mamba-cache-dtype",
        "float32",
        "--mamba-ssm-cache-dtype",
        "float32",
        "--enable-prefix-caching",
        "--prefix-match-unit",
        "128",
        "--gpu-memory-utilization",
        "0.95",
        "--max-num-seqs",
        "32",
        "--enable-cumem-allocator",
        "--no-enable-flashinfer-autotune",
        "--stream-interval",
        "10",
        "--dyn-enable-structural-tag"
      ],
      "env": {
        "VLLM_USE_V2_MODEL_RUNNER": "1",
        "VLLM_USE_DIRECT_DCP_A2A": "1",
        "VLLM_USE_DIRECT_DCP_Q_GATHER": "1",
        "VLLM_USE_DIRECT_DCP_KV_GATHER": "1",
        "VLLM_ALLREDUCE_USE_FLASHINFER": "1",
        "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0",
        "VLLM_USE_NCCL_SYMM_MEM": "0",
        "TOKENSPEED_MLA_FP16_PARTIALS": "0",
        "NCCL_CUMEM_ENABLE": "1",
        "NCCL_MNNVL_ENABLE": "1",
        "NCCL_NVLS_ENABLE": "1",
        "NCCL_P2P_LEVEL": "NVL",
        "NCCL_NET_PLUGIN": "none",
        "UCX_MEMTYPE_REG_WHOLE": "n",
        "UCX_TCP_AF_PRIO": "inet",
        "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4",
        "VLLM_MOONCAKE_STORE_SELECTIVE_LEASE": "1",
        "MC_SLICE_SIZE": "1048576",
        "MC_MAX_MR_SIZE": "4294967296",
        "MC_ENABLE_DEST_DEVICE_AFFINITY": "1",
        "WITH_NVIDIA_PEERMEM": "0",
        "DYN_TCP_REQUEST_TIMEOUT": "120",
        "ETCD_LEASE_TTL": "600",
        "VLLM_RPC_TIMEOUT": "600000",
        "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800",
        "TILELANG_CLEANUP_TEMP_FILES": "1",
        "VLLM_WORKER_MULTIPROC_METHOD": "spawn"
      },
      "json_args": {
        "--speculative-config": {
          "num_speculative_tokens": 4,
          "rejection_sample_method": "block"
        },
        "--kv-transfer-config": {
          "kv_load_failure_policy": "recompute",
          "NixlConnector": {
            "kv_buffer_device": "cuda",
            "kv_connector_extra_config": {
              "enforce_handshake_compat": false,
              "enable_cross_layers_blocks": false
            }
          },
          "MooncakeStoreConnector": {
            "kv_load_failure_policy": "recompute",
            "kv_connector_extra_config": {
              "enable_cross_layers_blocks": false,
              "enable_offload": false
            }
          }
        }
      },
      "roles": {
        "prefill": {
          "parallelism": "dep",
          "nodes": 4,
          "args": [
            "--kv-cache-memory-bytes",
            "61926160384",
            "--max-num-batched-tokens",
            "2048",
            "--long-prefill-token-threshold",
            "512",
            "--long-prefill-token-threshold-adaptive",
            "--max-cudagraph-capture-size",
            "2048",
            "--compilation-config",
            "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048]}"
          ],
          "env": {
            "VLLM_ENABLE_GEMM_RS": "1",
            "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024",
            "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024"
          }
        },
        "decode": {
          "parallelism": "dep",
          "nodes": 4,
          "args": [
            "--kv-cache-memory-bytes",
            "64475268096",
            "--additional-config",
            "{\"decode_backend\":\"latency_specialized\"}",
            "--use-replayssm",
            "--max-num-batched-tokens",
            "160",
            "--max-cudagraph-capture-size",
            "160",
            "--compilation-config",
            "{\"cudagraph_capture_sizes\":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160]}"
          ],
          "env": {
            "VLLM_ENABLE_GEMM_RS": "0"
          }
        }
      },
      "points": {
        "c352": {
          "label": "352 (1P1D)",
          "concurrency": 352,
          "description": "One DEP16 prefill worker and one DEP16 decode worker (32 GPUs)."
        }
      },
      "runbook": "The concurrency-352 point of the GB300 AgentX DEP16 lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dep16-disagg`, [SemiAnalysisAI/InferenceX#4907](https://github.com/SemiAnalysisAI/InferenceX/pull/4907)). Dynamo + vLLM run one prefill worker and one decode worker, each DEP16 (`--tensor-parallel-size 1 --data-parallel-size 16` with expert parallelism) across four 4-GPU GB300 trays, 32 GPUs in all, with MegaMoE on both roles and no DCP. KV moves over NIXL, and all workers share an RDMA Mooncake store in host DRAM. The lane ran the image digest the preset pins, with the Dynamo revision and Mooncake version of the [AgentX 1P3D TP8 + DCP8 preset](#preset-gb300_agentx_pd); the preset has been measured only on that combination.\n\nRun each worker command on each of its four trays with `--data-parallel-start-rank` 0, 4, 8 and 12 and the same `--data-parallel-address` (the first tray's address), as rendered.\n\n**Serving config.**\n- **Both roles:** `--max-num-seqs 32` per data-parallel rank and `--block-size 16384`: the image does not pick a power-of-two attention block size at TP1 by itself, so the lane pins the same size on both roles. `--kv-cache-memory-bytes` sizes the KV cache per rank explicitly (61926160384 bytes on prefill, 64475268096 on decode): the size vLLM reported it could use on this image, minus 3 GiB. Size it to your own image and memory budget; the explicit size replaces the KV sizing from `--gpu-memory-utilization 0.95`.\n- **Prefill:** `--max-num-batched-tokens 2048` with `--long-prefill-token-threshold 512 --long-prefill-token-threshold-adaptive`, so one long prompt does not fill a rank's step, and `FULL_AND_PIECEWISE` CUDA graphs for 4 to 2048 tokens. `VLLM_ENABLE_GEMM_RS=1` and Dynamo's admission limits (1024) as in the 1P3D preset.\n- **Decode:** the low-latency decode backend (`--additional-config '{\"decode_backend\":\"latency_specialized\"}'`) with `--use-replayssm`, `--max-num-batched-tokens 160`, and CUDA graphs for 5 to 160 rows in steps of 5 (32 requests × (1 + 4 draft tokens)) in vLLM's default graph mode, which the decode `--compilation-config` keeps by replacing the PD strategy's `FULL_DECODE_ONLY`. `VLLM_ENABLE_GEMM_RS=0`.\n- Everything else (precision, kernels, collectives, KV transfer, timeouts, tool-call and reasoning parsers, connector options) is as in the [1P3D preset](#preset-gb300_agentx_pd). The lane's env also keeps the direct DCP paths (`VLLM_USE_DIRECT_DCP_*=1`), which have no effect without DCP. All workers start with `VLLM_WORKER_MULTIPROC_METHOD=spawn`, and each tray's GPUs are pinned with `CUDA_VISIBLE_DEVICES`, as the rendered commands do. The data-parallel RPC ports are the builder's placeholders; the lane used 13355 for prefill and 13345 for decode.\n\nStart the services and the frontend as for the [1P3D preset](#preset-gb300_agentx_pd); the same benchmark-only settings are left out.\n\n### Throughput measurement only (synthetic acceptance)\n\nTo measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 4 draft tokens, 3.19. Only `--speculative-config` changes, on both roles:\n\n```bash\n--speculative-config '{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":4,\"method\":\"dspark\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":3.19}'\n```\n\nSynthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.\n",
      "json": "/moonshotai/Kimi-K3.1-Pro/presets/gb300_agentx_pd_dep16.json"
    },
    "gb300_agentx_c1": {
      "label": "AgentX TP16 + DCP16, concurrency 1",
      "hardware": "gb300",
      "strategy": "multi_node_tp",
      "workload": "Agentic (InferenceX AgentX)",
      "description": "Concurrency 1 on four GB300 trays: one aggregated TP16 + DCP16 engine on the low-latency decode backend, with the K3.1 DSpark draft (7 tokens) and the Mooncake KV store in host DRAM.",
      "source": {
        "label": "InferenceX #4907",
        "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4907",
        "key": "kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-agg",
        "revision": "f2bb2756fffac8fc8e5d241fe0f936e32c3c3e90"
      },
      "image": {
        "vllm": "vllm/vllm-openai:kimi-k31-mc@sha256:5c243ce00d2d289a295065e3d794d5beb783e9c9a85eba3d2eef7024f822a531",
        "dynamo": "3dd0744eb72c707d76d4c520ae4dea9e286a4ab6"
      },
      "select": {
        "nodes": 4,
        "frontend": "rust",
        "features": {
          "tool_calling": true,
          "reasoning": true,
          "spec_decoding": true,
          "text_only": true,
          "low_latency_decode": true,
          "nvfp4_kv_cache": false
        },
        "dcp": 16,
        "kv_offload": "kv_store_distributed_mooncake"
      },
      "args": [
        "--decode-context-parallel-size",
        "16",
        "--dcp-comm-backend",
        "a2a",
        "--cp-kv-cache-interleave-size",
        "1",
        "--kv-cache-dtype",
        "fp8",
        "--attention-config",
        "{\"indexer_kv_dtype\":\"mxfp4\"}",
        "--mamba-cache-dtype",
        "float32",
        "--mamba-ssm-cache-dtype",
        "float32",
        "--use-replayssm",
        "--enable-prefix-caching",
        "--prefix-match-unit",
        "128",
        "--gpu-memory-utilization",
        "0.90",
        "--enable-cumem-allocator",
        "--no-enable-flashinfer-autotune",
        "--stream-interval",
        "10",
        "--max-num-seqs",
        "1",
        "--max-num-batched-tokens",
        "16384",
        "--long-prefill-token-threshold",
        "4096",
        "--max-cudagraph-capture-size",
        "2048",
        "--compilation-config",
        "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[7,8,14,16,28,32,56,64,128,256,512,1024,2048]}"
      ],
      "env": {
        "VLLM_USE_V2_MODEL_RUNNER": "1",
        "VLLM_USE_DIRECT_DCP_A2A": "1",
        "VLLM_USE_DIRECT_DCP_Q_GATHER": "1",
        "VLLM_USE_DIRECT_DCP_KV_GATHER": "1",
        "VLLM_ALLREDUCE_USE_FLASHINFER": "1",
        "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0",
        "VLLM_USE_NCCL_SYMM_MEM": "0",
        "VLLM_ENABLE_GEMM_RS": "0",
        "TOKENSPEED_MLA_FP16_PARTIALS": "0",
        "VLLM_SSM_CONV_STATE_LAYOUT": "DS",
        "NCCL_CUMEM_ENABLE": "1",
        "NCCL_MNNVL_ENABLE": "1",
        "NCCL_NVLS_ENABLE": "1",
        "NCCL_P2P_LEVEL": "NVL",
        "NCCL_NET_PLUGIN": "none",
        "UCX_MEMTYPE_REG_WHOLE": "n",
        "UCX_TCP_AF_PRIO": "inet",
        "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4",
        "VLLM_MOONCAKE_STORE_SELECTIVE_LEASE": "1",
        "MC_SLICE_SIZE": "1048576",
        "MC_MAX_MR_SIZE": "4294967296",
        "MC_ENABLE_DEST_DEVICE_AFFINITY": "1",
        "WITH_NVIDIA_PEERMEM": "0",
        "VLLM_RPC_TIMEOUT": "600000",
        "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800",
        "TILELANG_CLEANUP_TEMP_FILES": "1"
      },
      "json_args": {
        "--speculative-config": {
          "num_speculative_tokens": 7,
          "rejection_sample_method": "block"
        },
        "--kv-transfer-config": {
          "kv_load_failure_policy": "recompute",
          "kv_connector_extra_config": {
            "load_async": true,
            "lookup_async": true,
            "enable_cross_layers_blocks": false,
            "enable_offload": false
          }
        }
      },
      "points": {
        "c1": {
          "label": "1",
          "concurrency": 1,
          "description": "One aggregated TP16 + DCP16 engine (16 GPUs)."
        }
      },
      "runbook": "The concurrency-1 point of the GB300 AgentX aggregated lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`kimik3.1-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-agg`, [SemiAnalysisAI/InferenceX#4907](https://github.com/SemiAnalysisAI/InferenceX/pull/4907)): one engine serves prefill and decode with TP16 + DCP16 across four 4-GPU GB300 trays, without expert parallelism, on the low-latency decode backend, with the Mooncake store in host DRAM as its KV tier (no NIXL). The lane ran the image digest the preset pins; the preset has been measured only on that image.\n\n**Serving config.**\n- **TP16 + DCP16.** DCP16 shards the MLA latent and indexer-key caches across the whole TP group, with the A2A DCP backend, `--cp-kv-cache-interleave-size 1` and the direct DCP paths (`VLLM_USE_DIRECT_DCP_*`). FP8 MLA KV cache with an MXFP4 indexer key cache; FP32 KDA conv and recurrent state.\n- **Low-latency decode** (the Low-latency decode feature, on in this preset) runs pure decode steps on the fused path, with `--use-replayssm`. A decode step is 8 rows (1 + 7 draft tokens); `FULL_AND_PIECEWISE` CUDA graphs cover 7 to 2048 tokens.\n- **Spec decoding** (on in this preset) uses the K3.1 DSpark draft with 7 tokens, probabilistic drafting and block rejection.\n- `--max-num-seqs 1`, `--gpu-memory-utilization 0.90`, and `--max-num-batched-tokens 16384` with `--long-prefill-token-threshold 4096`: a lone prompt prefills in chunks of up to 16384 tokens, and next to other requests each prompt's chunk stops at 4096 tokens.\n- The env follows the [1P3D preset](#preset-gb300_agentx_pd) (collectives, kernels, Mooncake transfer settings, timeouts) with `VLLM_ENABLE_GEMM_RS=0` and `VLLM_SSM_CONV_STATE_LAYOUT=DS`. It also keeps the lane's `UCX_*` settings, which have no effect without NIXL.\n- The preset selects the distributed (embedded) Mooncake store on the **KV Offload** row and sets the lane's options on its `MooncakeStoreConnector`: recompute on a failed load, asynchronous load and lookup, and cross-layer blocks and store offload off. Start the Mooncake master and size the store as for the [1P3D preset](#preset-gb300_agentx_pd).\n\n**Served through Dynamo in the lane.** InferenceX ran this engine as one aggregated Dynamo worker (`python3 -m dynamo.vllm` on each tray with the same engine flags) behind the Dynamo frontend of the [1P3D preset](#preset-gb300_agentx_pd), with `DYN_REQUEST_PLANE=tcp`, `DYN_TCP_REQUEST_TIMEOUT=120` and `ETCD_LEASE_TTL=600`. The builder renders the same engine flags for `vllm serve` on the Multi-Node TP strategy, with vLLM's Rust frontend. Under Dynamo, tool calling and reasoning take `--dyn-tool-call-parser kimi_k3 --dyn-reasoning-parser kimi_k3 --dyn-enable-structural-tag` and `--reasoning-parser kimi_k3` in place of the rendered parser flags, as in the lane's accuracy runs.\n\n**Benchmark-only settings left out.** As in the [1P3D preset](#preset-gb300_agentx_pd): `--served-model-name` (the model ID), `VLLM_SERVER_DEV_MODE=1`, and the job and benchmark-client settings. The throughput runs leave out the parser flags.\n\n### Throughput measurement only (synthetic acceptance)\n\nTo measure throughput the AgentX way, replace real verification with synthetic acceptance at InferenceX's K3.1 DSpark golden acceptance length for 7 draft tokens, 3.69. Only `--speculative-config` changes:\n\n```bash\n--speculative-config '{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":7,\"method\":\"dspark\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":3.69}'\n```\n\nSynthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.\n",
      "json": "/moonshotai/Kimi-K3.1-Pro/presets/gb300_agentx_c1.json"
    }
  },
  "decode_context_parallel": {
    "sizes": {
      "b300": [
        8
      ],
      "gb300": [
        8,
        16
      ]
    },
    "match_tp": true
  },
  "notes": [
    {
      "title": "Text-only on TPU",
      "when": {
        "hardware": [
          "ironwood"
        ]
      },
      "body": "TPU serving runs through vllm-torchtpu, which does not load the vision tower for K3.1 — the command always carries `--language-model-only` and image/video are disabled."
    }
  ],
  "guide": "## Overview\n\nKimi-K3.1 keeps the Kimi-K3 layout (Kimi Delta Attention, 896 routed experts with 16 active\nper token, a latent MoE projection and a vision encoder) and adds sparse\nattention (DSA, top-2048 indexer) to 24 of its MLA layers. The checkpoint ships MXFP4\nrouted-expert weights; everything else is BF16.\n\nThe baseline is the simplest working setup on NVIDIA: one tensor-parallel group of 8 GPUs,\neither two GB300 trays or one B300 node. Throughput was not tuned.\n\nVR200 (preview) takes the same TP8 layout on two trays, in the `vllm/vllm-openai:kimi-k31-cu134`\nimage (CUDA 13.4). It uses the Blackwell flags and hasn't been tuned or verified on Rubin.\n\nOn one B300 node, three AgentX presets follow InferenceX's B300 vLLM AgentX lane\n(SemiAnalysisAI/InferenceX#4871), all with TP8 + DCP8 attention, the K3.1 DSpark draft, the\nlow-latency decode backend and host KV offload:\n[AgentX TP8 + DCP8](#preset-b300_agentx) at concurrency 1 and 6,\n[concurrency 12 and 24](#preset-b300_agentx_mixed_graphs) with CUDA graphs for prefill and mixed steps, and,\nwith routed experts EP8 on MegaMoE, [AgentX TP8 + DCP8 + EP8](#preset-b300_agentx_ep) at concurrency 48 and 64.\n\nOn MI355X, one node takes the same TP8 layout with the AMD profile's AITER environment, in the\n`vllm/vllm-openai-rocm:kimi-k31-5c14fe8` image. The [AgentX TP8](#preset-mi355x_agentx) preset\nfollows InferenceX's MI355X vLLM AgentX lane (SemiAnalysisAI/InferenceX#5089) with one point per\nconcurrency, all with the K3.1 DSpark draft: plain TP8 with GPU KV and the low-latency decode\nbackend at concurrency 1, DCP8 with GPU KV and the low-latency decode backend at 4, and DCP8\nwith host KV offload from 16 to 48. It needs the image the preset pins.\n\nOn Zhenwu PPUs, the default MXFP4 checkpoint runs without speculative decoding on\ntwo 16-PPU 810E nodes with TP32 / EP32, or four 8-PPU M890 nodes with TP8 / DP4 / EP32.\nOn 810E, `ppu_deep_gemm_w4a16` loads the checkpoint with a W4A16 compute path.\nUse the PPU image specified for each hardware configuration.\n\nOn TPU, the model is served by [vLLM torch-TPU](https://github.com/vllm-project/vllm-torchtpu),\ntext-only: the vision tower is not loaded. The validated layout is TP32 with expert\nparallelism on a 2x2x4 TPU v7x slice (16 chips, 32 devices, on 4 hosts). See [TPU serving](#tpu-serving).\n\n## Tensor + expert parallelism\n\nOn one B300 node, the **Tensor + Expert Parallel** strategy keeps attention at TP8 and runs\nthe routed experts with expert parallelism on the MegaMoE kernel. It has the lowest\nlong-prompt time to first token of the single-node layouts: a single 340K-token prompt\nprefills in about 10 s, against about 15 s at plain TP8 (8x B300, one request at a time,\npre-release checkpoint).\n\n## Long context: decode context parallelism\n\nThe **Decode Context Parallelism** row shards the MLA latent and indexer-key caches across\nthe 8 GPUs of the TP group instead of replicating them, so DCP8 holds about 7x the KV cache\ntokens of TP8 (about 18.5M vs 2.6M tokens with FP8 KV on 8x B300). Prefill is as fast as\nat TP8. Decode steps are 7–25% slower than TP8 at the same batch, so use DCP when KV\ncapacity is the limit: very long contexts or many long-lived agentic sessions.\n\n## Low-latency decode\n\nThe **Low-latency decode** feature switches decode steps to a fused CuTe DSL path tuned for\nsmall batches; prefill and mixed steps still run the default path. On 8x B300 at TP8 it cuts\nthe decode step by 1.6–1.7x at batch 1 and 1.2–1.3x at batch 32–64. Under DCP8 it brings decode\nto within 6% of TP8. It requires the V2 model runner and FULL decode CUDA graphs (the\ndefaults), and TP8 or TP8 + DCP8. The feature toggle is offered only without expert\nparallelism; the B300 AgentX preset with routed experts EP8 on MegaMoE\n([concurrency 48 and 64](#preset-b300_agentx_ep)) sets\nthe same `decode_backend` in its own args.\n\nOn MI355X the same switch selects fused FlyDSL decode kernels. They cover only pure decode\nsteps of up to 16 rows, so among other checks they also need speculative decoding,\n`--max-num-seqs` 1, 2, 4 or 8 with `--max-num-seqs` × (1 + draft tokens) ≤ 16, BF16 KDA\nprojections, an FP32 KDA recurrent state (`--mamba-ssm-cache-dtype float32`) and no KV\noffload. Without DCP they serve only 3 or 7 draft tokens at `--max-num-seqs` 1 or 2, so the\nMI355X default of 5 draft tokens needs DCP8. The concurrency-1 (no DCP) and 4 (DCP8) points\nof the [AgentX TP8 preset](#preset-mi355x_agentx) show working setups, on the image it pins;\ncurrent vLLM main does not yet run K3.1 on this path without DCP. Its points from\nconcurrency 16 do not qualify, so leave the feature off there.\n\n## NVFP4 KV cache\n\nThe **NVFP4 KV cache** feature stores the MLA latent in NVFP4 (352 bytes per token per layer\ninstead of 576) and holds about 1.5x the KV cache tokens of FP8. The indexer keys stay FP8,\nso the top-2048 token selection is unchanged. GSM8K matched FP8 on 8x GB300 (96.1% vs 96.2%);\nlong-context accuracy has not been measured yet.\n\n## TPU serving\n\nTPU serving runs through [vLLM torch-TPU](https://github.com/vllm-project/vllm-torchtpu) and\nis text-only: the vision tower is not loaded. K3.1 support is under review as\n[vllm-torchtpu#2775](https://github.com/vllm-project/vllm-torchtpu/pull/2775); until it merges,\nthe Install tab on TPU builds from the `kimi-k31pro` staging branch (Python 3.12, uv). No\npublished image or wheel carries it yet.\n\nThe generated command uses vllm-torchtpu's default multiprocessing multi-host backend: run the\nhead command on the slice's first host and the worker commands on the remaining hosts (the\nbuilder renders both). The **Multi-Node TEP** strategy at 4 nodes matches the validated flags of\n[vllm-torchtpu#2775](https://github.com/vllm-project/vllm-torchtpu/pull/2775) with two\ndifferences: #2775 ran one driver over a Ray cluster (`TPU_MULTIHOST_BACKEND=ray`), and its\nGSM8K validation used the completions API, so the default-on chat and tool-call parsers were not\nexercised on TPU. Validated on a 2x2x4 TPU v7x slice with an FP8 latent KV cache, the unified\nblock pool and prefix caching: the complete GSM8K split scored 97.50% (TP32) and 97.42%\n(TP32 + DCP8), prefix-cache hits returned identical tokens and logprobs, and the BEAM-1M\nlong-context benchmark matched the GPU reference. The run-specific sizing of that validation\n(10K context, 16 sequences, compile sizes) is not the TPU default — it lives in the two\n**Validated** presets: [TEP32](#preset-v7x_validated) and\n[TEP32 + DCP8](#preset-v7x_dcp8) for KV-capacity-bound workloads.\n\n**Spec decoding is gated off on TPU for now**: DSpark with `--async-scheduling` (always on in\nthis configuration) needs [vllm-torchtpu#2770](https://github.com/vllm-project/vllm-torchtpu/pull/2770),\nstill open. #2775 validated DSpark with 7 draft tokens once that fix is in.\n\n## Prefill/decode disaggregation\n\nThe **Prefill/Decode Disaggregation** strategy runs one TP8 prefill pool (with expert\nparallelism) and one DEP16 decode pool (TP1 x DP16 with expert parallelism), and decode\npulls each request's KV from prefill over NIXL. Every PD worker sets\n`VLLM_SSM_CONV_STATE_LAYOUT=DS`: NIXL transfers the Kimi Delta Attention conv state in three\nreads, which needs that layout, and the NIXL connector refuses to start without it.\n\n## KV cache offloading\n\nThe **Simple** option on the KV Offload row adds `SimpleCPUOffloadConnector`, a host-DRAM\nprefix tier for each GPU. Under Prefill/Decode Disaggregation it is added to the prefill pool\nonly, next to NIXL in a MultiConnector; decode keeps plain NIXL. The tier pins\n`cpu_bytes_to_use_per_rank` of host memory for every GPU, so size it from the host DRAM that\nis free after the engines are up. The tested GB300 prefill workers used 128 GiB per GPU\n(512 GiB per tray).\n\n## Client usage\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"moonshotai/Kimi-K3.1-Pro\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Explain sparse attention in two sentences.\"}],\n    \"temperature\": 0.7,\n    \"top_p\": 0.95\n  }'\n```\n\n## References\n\n- [Model card](https://huggingface.co/moonshotai/Kimi-K3.1-Pro)\n- [Kimi-K3 recipe](https://recipes.vllm.ai/moonshotai/Kimi-K3)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "1bf414e97acf7563297726531f702cdbea6b2921",
    "path": "recipes/models/moonshotai/Kimi-K3.1-Pro.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/moonshotai/Kimi-K3.1-Pro.yaml",
    "overlays": []
  }
}