# IFM/PAN2 — vLLM serving recipe

> 29B foundation world model for text-to-video and image-to-video: a dual-stream diffusion transformer that simulates how a scene evolves under a time-grounded plan

Configuration: hardware `h200` (8× H200) · variant `default` (BF16) · task `t2v`

- Interactive page: https://vllm-recipes-hm812xejs-inferact-inc.vercel.app/IFM/PAN2
- This configuration as JSON: https://vllm-recipes-hm812xejs-inferact-inc.vercel.app/IFM/PAN2.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `rtx_pro_6000`, `rtx_pro_6000_2x`, `l40s`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `task`: `t2v`, `i2v`

## Install

Requires vLLM >= 0.33.0.

```bash
uv pip install git+https://github.com/vllm-project/vllm-omni.git
```

Extra dependencies:

- PAN 2 guardrails: they may block a request with HTTP 400.

```bash
uv pip install git+https://github.com/ifm-ai/pan2-guardrail
```

## Serve

vLLM-Omni recipe — the generation endpoints and request examples are in the guide below.

```bash
vllm serve IFM/PAN2 \
  --num-gpus 4 \
  --cfg-parallel-size 2 \
  --usp 2 \
  --vae-patch-parallel-size 4 \
  --vae-parallel-mode tile \
  --omni
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

**PAN 2** is a foundation world model that simulates how a scene evolves
under a time-grounded plan of actions and camera movements. Its 29B
generative simulator is a dual-stream multimodal diffusion transformer
(MMDiT): the prompt is encoded into a continuous state stream, and the state
and video streams, each with its own projections and feed-forward networks,
are joined by self-attention at every layer. The plan guides video denoising,
while the developing video refines the state. One server handles both
tasks: a request with a first-frame image runs image-to-video (I2V), a
request without one runs text-to-video (T2V).

| Item | Contract |
| --- | --- |
| Tasks | T2V; I2V from one first-frame image |
| Prompt | One structured-caption prompt per request, plus an optional negative prompt |
| Default output | 704x1248, 313 frames (13 s) at 24 fps |
| Longest output | 481 frames (20 s) at 24 fps |
| Size | `height` and `width` multiples of 32 |
| Frame count | `num_frames` of the form `4k+1` (`duration_sec * 24 + 1` for whole seconds) |
| Sampling | 50 steps and CFG 2.5 by default; PAN 2's own flow-matching schedule |

### Prompt format

PAN 2 needs a structured prompt. It is trained on the captions written by its
[prompt rewriter](https://github.com/ifm-ai/PAN2/blob/main/inference/pan-prompt-rewriter/SKILL.md),
and expects the prompt as that caption serialized to **compact one-line
JSON** (`json.dumps(caption, ensure_ascii=False, separators=(",", ":"))`).
A plain one-line prompt produces unusable video. The caption describes the
style, lighting, camera, subjects, the first frame, and a timeline of events:

```json
{
  "video_style": "Serene aerial travel footage during blue hour, captured in one uninterrupted continuous shot.",
  "lighting": "Soft blue skylight after sunset, a pale violet horizon and small pools of amber light ...",
  "perspective_view": "Wide drone view looking obliquely down along Santorini's caldera rim ...",
  "subjects_lifecycle": [],
  "init_frame_visual_description": "Two cobalt-blue church domes rise among tightly stepped white Cycladic buildings ...",
  "events": [
    {
      "event_index": 0,
      "time_range_sec": "0 to 4",
      "event_description": "The drone glides gently right along the cliff ...",
      "camera_motion": [{"time_range_sec": "0 to 4", "camera_motion": ["track_right"]}],
      "contained_actions": []
    }
  ],
  "quality": {"video_quality": "Resolved architectural edges ...", "text_logos_overlays": null, "artifacts": null}
}
```

The negative prompt uses the same format. Without one, or with an empty one,
PAN 2 uses the negative prompt it was released with, its time ranges set to
the clip's duration. The request examples read the prompt and negative
prompt from `prompt.json` and `negative_prompt.json`
(`i2v_prompt.json` and `i2v_negative_prompt.json` for image-to-video).

### Guardrails

PAN 2 runs guardrails on the prompt and the generated video, and they may
block a request with HTTP 400. They need the `pan2-guardrail` package.

### Hardware profiles

Pick the hardware above and the command updates. Every profile is sized for
the longest 481-frame, 704x1248 output. Peaks are per GPU.

| GPU memory | GPUs | Key arguments | Peak memory |
| --- | ---: | --- | ---: |
| ≥141 GB (H200, B200, B300, GB200, GB300) | 1 | (none) | 107.4 GiB |
| | 4 | `--cfg-parallel-size 2 --usp 2` | 110.7 GiB |
| | 8 | `--cfg-parallel-size 2 --usp 4` | 110.4 GiB |
| 96 GB (RTX PRO 6000) | 1 | `--enable-distributed-layerwise-offload --dlo-resident-layers 48`, `offload_guardrail_models` | 84.4 GiB |
| | 2 | `--tensor-parallel-size 2` | 84.5 GiB |

`offload_guardrail_models` is set with
`--stage-overrides '{"0": {"model_config": {"offload_guardrail_models": true}}}'`.
With `--enable-distributed-layerwise-offload`, the DiT blocks stay in pinned
host memory and stream to the GPU each step, with `--dlo-resident-layers`
leading blocks kept on the GPU. The offloaded weights live in pinned system
RAM, so these profiles need a large amount of it, and each CFG-parallel
branch adds its own copy.

### Request options

```bash
-F "size=1248x704" -F "num_frames=481" \
-F "num_inference_steps=50" -F "guidance_scale=2.5"
```

Use `num_frames` to set the length; `seconds` is not used. Requests with a
size that is not a multiple of 32, a frame count that is not `4k+1`, more
than one image, video input, or custom `sigmas`/`timesteps` are rejected
with HTTP 400 before they reach the GPUs.

#### Request-scoped quality

PAN 2 can switch Cache-DiT per request without a startup option:
`-F 'quality=lossless'` runs the reference path, `-F 'quality=high'` PAN 2's
conservative Cache-DiT profile. `high` is about 1.3× faster than `lossless`;
the speedup depends on the prompt and the parallel configuration.

### References

- [Model card](https://huggingface.co/IFM/PAN2)
- [vLLM-Omni PR #8602](https://github.com/vllm-project/vllm-omni/pull/8602)
- [PAN 2 guardrail](https://github.com/ifm-ai/pan2-guardrail)
