vLLM/Recipes
Search
IFM

IFM/PAN2

29B foundation world model for text-to-video and image-to-video: a dual-stream diffusion transformer that simulates how a scene evolves under a time-grounded plan

29B video DiT — 13 s of 704x1248 video in ~119 s on 4×B300 (CFG parallel × Ulysses 2)

Default configuration: 8× H200 · BF16
vllm serve IFM/PAN2 \
  --num-gpus 4 \
  --cfg-parallel-size 2 \
  --usp 2 \
  --vae-patch-parallel-size 4 \
  --vae-parallel-mode tile \
  --omni

The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /IFM/PAN2.md.

Guide

Overview

PAN 2 is a foundation world model that simulates how a scene evolves under a time-grounded plan of actions and camera movements. Its 29B generative simulator is a dual-stream multimodal diffusion transformer (MMDiT): the prompt is encoded into a continuous state stream, and the state and video streams, each with its own projections and feed-forward networks, are joined by self-attention at every layer. The plan guides video denoising, while the developing video refines the state. One server handles both tasks: a request with a first-frame image runs image-to-video (I2V), a request without one runs text-to-video (T2V).

ItemContract
TasksT2V; I2V from one first-frame image
PromptOne structured-caption prompt per request, plus an optional negative prompt
Default output704x1248, 313 frames (13 s) at 24 fps
Longest output481 frames (20 s) at 24 fps
Sizeheight and width multiples of 32
Frame countnum_frames of the form 4k+1 (duration_sec * 24 + 1 for whole seconds)
Sampling50 steps and CFG 2.5 by default; PAN 2's own flow-matching schedule

Prompt format

PAN 2 needs a structured prompt. It is trained on the captions written by its prompt rewriter, and expects the prompt as that caption serialized to compact one-line JSON (json.dumps(caption, ensure_ascii=False, separators=(",", ":"))). A plain one-line prompt produces unusable video. The caption describes the style, lighting, camera, subjects, the first frame, and a timeline of events:

{
  "video_style": "Serene aerial travel footage during blue hour, captured in one uninterrupted continuous shot.",
  "lighting": "Soft blue skylight after sunset, a pale violet horizon and small pools of amber light ...",
  "perspective_view": "Wide drone view looking obliquely down along Santorini's caldera rim ...",
  "subjects_lifecycle": [],
  "init_frame_visual_description": "Two cobalt-blue church domes rise among tightly stepped white Cycladic buildings ...",
  "events": [
    {
      "event_index": 0,
      "time_range_sec": "0 to 4",
      "event_description": "The drone glides gently right along the cliff ...",
      "camera_motion": [{"time_range_sec": "0 to 4", "camera_motion": ["track_right"]}],
      "contained_actions": []
    }
  ],
  "quality": {"video_quality": "Resolved architectural edges ...", "text_logos_overlays": null, "artifacts": null}
}

The negative prompt uses the same format. Without one, or with an empty one, PAN 2 uses the negative prompt it was released with, its time ranges set to the clip's duration. The request examples read the prompt and negative prompt from prompt.json and negative_prompt.json (i2v_prompt.json and i2v_negative_prompt.json for image-to-video).

Guardrails

PAN 2 runs guardrails on the prompt and the generated video, and they may block a request with HTTP 400. They need the pan2-guardrail package.

Hardware profiles

Pick the hardware above and the command updates. Every profile is sized for the longest 481-frame, 704x1248 output. Peaks are per GPU.

GPU memoryGPUsKey argumentsPeak memory
≥141 GB (H200, B200, B300, GB200, GB300)1(none)107.4 GiB
4--cfg-parallel-size 2 --usp 2110.7 GiB
8--cfg-parallel-size 2 --usp 4110.4 GiB
96 GB (RTX PRO 6000)1--enable-distributed-layerwise-offload --dlo-resident-layers 48, offload_guardrail_models84.4 GiB
2--tensor-parallel-size 284.5 GiB

offload_guardrail_models is set with --stage-overrides '{"0": {"model_config": {"offload_guardrail_models": true}}}'. With --enable-distributed-layerwise-offload, the DiT blocks stay in pinned host memory and stream to the GPU each step, with --dlo-resident-layers leading blocks kept on the GPU. The offloaded weights live in pinned system RAM, so these profiles need a large amount of it, and each CFG-parallel branch adds its own copy.

Request options

-F "size=1248x704" -F "num_frames=481" \
-F "num_inference_steps=50" -F "guidance_scale=2.5"

Use num_frames to set the length; seconds is not used. Requests with a size that is not a multiple of 32, a frame count that is not 4k+1, more than one image, video input, or custom sigmas/timesteps are rejected with HTTP 400 before they reach the GPUs.

Request-scoped quality

PAN 2 can switch Cache-DiT per request without a startup option: -F 'quality=lossless' runs the reference path, -F 'quality=high' PAN 2's conservative Cache-DiT profile. high is about 1.3× faster than lossless; the speedup depends on the prompt and the parallel configuration.

References