{
  "hf_id": "IFM/PAN2",
  "meta": {
    "title": "PAN 2",
    "slug": "pan2",
    "provider": "IFM",
    "description": "29B foundation world model for text-to-video and image-to-video: a dual-stream diffusion transformer that simulates how a scene evolves under a time-grounded plan",
    "date_added": "2026-10-12",
    "date_updated": "2026-11-05",
    "difficulty": "advanced",
    "tasks": [
      "omni"
    ],
    "performance_headline": "29B video DiT — 13 s of 704x1248 video in ~119 s on 4×B300 (CFG parallel × Ulysses 2)",
    "hardware": {
      "b300": "verified",
      "gb300": "supported",
      "b200": "supported",
      "gb200": "supported",
      "h200": "verified",
      "h100": "supported",
      "rtx_pro_6000": "verified",
      "rtx_pro_6000_2x": "verified",
      "l40s": "supported"
    }
  },
  "model": {
    "model_id": "IFM/PAN2",
    "min_vllm_version": "0.33.0",
    "architecture": "dense",
    "parameter_count": "29B",
    "active_parameters": "29B",
    "context_length": 0,
    "base_args": [],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv pip install git+https://github.com/vllm-project/vllm-omni.git"
      },
      "extras": [
        {
          "note": "PAN 2 guardrails: they may block a request with HTTP 400.",
          "command": "uv pip install git+https://github.com/ifm-ai/pan2-guardrail"
        }
      ]
    }
  },
  "omni": {
    "tasks": [
      {
        "id": "t2v",
        "description": "Text → video (704x1248, 241 frames = 10 s at 24 fps)",
        "curl": "curl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"prompt=<prompt.json\" \\\n  -F \"negative_prompt=<negative_prompt.json\" \\\n  -F \"size=1248x704\" \\\n  -F \"num_frames=241\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=50\" \\\n  -F \"guidance_scale=2.5\" \\\n  -F \"seed=42\" \\\n  --output pan2_t2v.mp4\n"
      },
      {
        "id": "i2v",
        "description": "First frame + text → video (704x1248, 241 frames = 10 s at 24 fps)",
        "curl": "curl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"input_reference=@first_frame.jpg\" \\\n  -F \"prompt=<i2v_prompt.json\" \\\n  -F \"negative_prompt=<i2v_negative_prompt.json\" \\\n  -F \"size=1248x704\" \\\n  -F \"num_frames=241\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=50\" \\\n  -F \"guidance_scale=2.5\" \\\n  -F \"seed=42\" \\\n  --output pan2_i2v.mp4\n"
      }
    ]
  },
  "features": {},
  "opt_in_features": [],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 108,
      "description": "BF16 — one ≥141 GB GPU holds the whole pipeline (107 GiB peak at 481 frames); smaller GPUs use tensor parallelism and layerwise offload",
      "hardware_overrides": {
        "h200": {
          "extra_args": [
            "--num-gpus",
            "4",
            "--cfg-parallel-size",
            "2",
            "--usp",
            "2",
            "--vae-patch-parallel-size",
            "4",
            "--vae-parallel-mode",
            "tile"
          ]
        },
        "b200": {
          "extra_args": [
            "--num-gpus",
            "4",
            "--cfg-parallel-size",
            "2",
            "--usp",
            "2",
            "--vae-patch-parallel-size",
            "4",
            "--vae-parallel-mode",
            "tile"
          ]
        },
        "b300": {
          "extra_args": [
            "--num-gpus",
            "4",
            "--cfg-parallel-size",
            "2",
            "--usp",
            "2",
            "--vae-patch-parallel-size",
            "4",
            "--vae-parallel-mode",
            "tile"
          ]
        },
        "gb200": {
          "extra_args": [
            "--num-gpus",
            "4",
            "--cfg-parallel-size",
            "2",
            "--usp",
            "2",
            "--vae-patch-parallel-size",
            "4",
            "--vae-parallel-mode",
            "tile"
          ]
        },
        "gb300": {
          "extra_args": [
            "--num-gpus",
            "4",
            "--cfg-parallel-size",
            "2",
            "--usp",
            "2",
            "--vae-patch-parallel-size",
            "4",
            "--vae-parallel-mode",
            "tile"
          ]
        },
        "h100": {
          "extra_args": [
            "--num-gpus",
            "8",
            "--tensor-parallel-size",
            "4",
            "--cfg-parallel-size",
            "2",
            "--vae-patch-parallel-size",
            "8",
            "--vae-parallel-mode",
            "tile",
            "--stage-overrides",
            "{\"0\": {\"model_config\": {\"offload_guardrail_models\": true}}}"
          ]
        },
        "rtx_pro_6000": {
          "extra_args": [
            "--vae-use-tiling",
            "--enable-distributed-layerwise-offload",
            "--dlo-no-use-allgather",
            "--dlo-resident-layers",
            "48",
            "--stage-overrides",
            "{\"0\": {\"model_config\": {\"offload_guardrail_models\": true}}}"
          ],
          "extra_env": {
            "VLLM_OMNI_VIDEO_SYNC_TIMEOUT": "14400",
            "VLLM_OMNI_DLO_DP_WAVE_TIMEOUT": "14400"
          }
        },
        "rtx_pro_6000_2x": {
          "extra_args": [
            "--num-gpus",
            "2",
            "--tensor-parallel-size",
            "2",
            "--vae-patch-parallel-size",
            "2",
            "--vae-parallel-mode",
            "tile"
          ],
          "extra_env": {
            "VLLM_OMNI_VIDEO_SYNC_TIMEOUT": "14400",
            "VLLM_OMNI_DLO_DP_WAVE_TIMEOUT": "14400"
          }
        },
        "l40s": {
          "extra_args": [
            "--vae-use-tiling",
            "--enable-distributed-layerwise-offload",
            "--dlo-no-use-allgather",
            "--dlo-resident-layers",
            "10",
            "--enforce-eager",
            "--stage-overrides",
            "{\"0\": {\"model_config\": {\"offload_guardrail_models\": true}}}"
          ]
        }
      }
    }
  },
  "hardware_overrides": {},
  "guide": "## Overview\n\n**PAN 2** is a foundation world model that simulates how a scene evolves\nunder a time-grounded plan of actions and camera movements. Its 29B\ngenerative simulator is a dual-stream multimodal diffusion transformer\n(MMDiT): the prompt is encoded into a continuous state stream, and the state\nand video streams, each with its own projections and feed-forward networks,\nare joined by self-attention at every layer. The plan guides video denoising,\nwhile the developing video refines the state. One server handles both\ntasks: a request with a first-frame image runs image-to-video (I2V), a\nrequest without one runs text-to-video (T2V).\n\n| Item | Contract |\n| --- | --- |\n| Tasks | T2V; I2V from one first-frame image |\n| Prompt | One structured-caption prompt per request, plus an optional negative prompt |\n| Default output | 704x1248, 313 frames (13 s) at 24 fps |\n| Longest output | 481 frames (20 s) at 24 fps |\n| Size | `height` and `width` multiples of 32 |\n| Frame count | `num_frames` of the form `4k+1` (`duration_sec * 24 + 1` for whole seconds) |\n| Sampling | 50 steps and CFG 2.5 by default; PAN 2's own flow-matching schedule |\n\n## Prompt format\n\nPAN 2 needs a structured prompt. It is trained on the captions written by its\n[prompt rewriter](https://github.com/ifm-ai/PAN2/blob/main/inference/pan-prompt-rewriter/SKILL.md),\nand expects the prompt as that caption serialized to **compact one-line\nJSON** (`json.dumps(caption, ensure_ascii=False, separators=(\",\", \":\"))`).\nA plain one-line prompt produces unusable video. The caption describes the\nstyle, lighting, camera, subjects, the first frame, and a timeline of events:\n\n```json\n{\n  \"video_style\": \"Serene aerial travel footage during blue hour, captured in one uninterrupted continuous shot.\",\n  \"lighting\": \"Soft blue skylight after sunset, a pale violet horizon and small pools of amber light ...\",\n  \"perspective_view\": \"Wide drone view looking obliquely down along Santorini's caldera rim ...\",\n  \"subjects_lifecycle\": [],\n  \"init_frame_visual_description\": \"Two cobalt-blue church domes rise among tightly stepped white Cycladic buildings ...\",\n  \"events\": [\n    {\n      \"event_index\": 0,\n      \"time_range_sec\": \"0 to 4\",\n      \"event_description\": \"The drone glides gently right along the cliff ...\",\n      \"camera_motion\": [{\"time_range_sec\": \"0 to 4\", \"camera_motion\": [\"track_right\"]}],\n      \"contained_actions\": []\n    }\n  ],\n  \"quality\": {\"video_quality\": \"Resolved architectural edges ...\", \"text_logos_overlays\": null, \"artifacts\": null}\n}\n```\n\nThe negative prompt uses the same format. Without one, or with an empty one,\nPAN 2 uses the negative prompt it was released with, its time ranges set to\nthe clip's duration. The request examples read the prompt and negative\nprompt from `prompt.json` and `negative_prompt.json`\n(`i2v_prompt.json` and `i2v_negative_prompt.json` for image-to-video).\n\n## Guardrails\n\nPAN 2 runs guardrails on the prompt and the generated video, and they may\nblock a request with HTTP 400. They need the `pan2-guardrail` package.\n\n## Hardware profiles\n\nPick the hardware above and the command updates. Every profile is sized for\nthe longest 481-frame, 704x1248 output. Peaks are per GPU.\n\n| GPU memory | GPUs | Key arguments | Peak memory |\n| --- | ---: | --- | ---: |\n| ≥141 GB (H200, B200, B300, GB200, GB300) | 1 | (none) | 107.4 GiB |\n| | 4 | `--cfg-parallel-size 2 --usp 2` | 110.7 GiB |\n| | 8 | `--cfg-parallel-size 2 --usp 4` | 110.4 GiB |\n| 96 GB (RTX PRO 6000) | 1 | `--enable-distributed-layerwise-offload --dlo-resident-layers 48`, `offload_guardrail_models` | 84.4 GiB |\n| | 2 | `--tensor-parallel-size 2` | 84.5 GiB |\n\n`offload_guardrail_models` is set with\n`--stage-overrides '{\"0\": {\"model_config\": {\"offload_guardrail_models\": true}}}'`.\nWith `--enable-distributed-layerwise-offload`, the DiT blocks stay in pinned\nhost memory and stream to the GPU each step, with `--dlo-resident-layers`\nleading blocks kept on the GPU. The offloaded weights live in pinned system\nRAM, so these profiles need a large amount of it, and each CFG-parallel\nbranch adds its own copy.\n\n## Request options\n\n```bash\n-F \"size=1248x704\" -F \"num_frames=481\" \\\n-F \"num_inference_steps=50\" -F \"guidance_scale=2.5\"\n```\n\nUse `num_frames` to set the length; `seconds` is not used. Requests with a\nsize that is not a multiple of 32, a frame count that is not `4k+1`, more\nthan one image, video input, or custom `sigmas`/`timesteps` are rejected\nwith HTTP 400 before they reach the GPUs.\n\n### Request-scoped quality\n\nPAN 2 can switch Cache-DiT per request without a startup option:\n`-F 'quality=lossless'` runs the reference path, `-F 'quality=high'` PAN 2's\nconservative Cache-DiT profile. `high` is about 1.3× faster than `lossless`;\nthe speedup depends on the prompt and the parallel configuration.\n\n## References\n\n- [Model card](https://huggingface.co/IFM/PAN2)\n- [vLLM-Omni PR #8602](https://github.com/vllm-project/vllm-omni/pull/8602)\n- [PAN 2 guardrail](https://github.com/ifm-ai/pan2-guardrail)\n",
  "source": {
    "source": "vllm-omni",
    "repo": "vllm-project/vllm-omni",
    "ref": "main",
    "sha": "ed8d8f00728036cd3ea317e95bb567182ddb9830",
    "path": "recipes/models/IFM/PAN2.yaml",
    "edit_url": "https://github.com/vllm-project/vllm-omni/edit/main/recipes/models/IFM/PAN2.yaml",
    "overlays": []
  }
}