One latent interface. Every dynamics you see below.
Every video here is generated by ShadowDancer from a first frame and a stream of demonstrated actions — first-person gunfights, third-person swordplay and spellcasting, monster hunts, human motion under orbiting cameras, and robot manipulation. Each showcase runs on a model specialized to its world: no labels, an action is just a clip, read once by a frozen encoder and replayed as a stream of reusable assets.
Every video is a shadow.
In Plato's cave, the prisoners mistake shadows for the things themselves. A video makes the same confusion easy: it shows its dynamics — what moves, and how — only through one particular appearance, one character, one scene, one camera. Trained on single videos, latent action models cannot tell which traits belong to the action and which merely co-occur with it — they entangle the two indiscriminately.
Our answer is to observe the same dynamics twice. A shadow pair is two frame-synchronized renders of one dynamics with everything else independently resampled — two shadows of one motion. Whatever survives the pairing is the action; nothing else can help predict one shadow from the other. And that cuts both ways, by design: traits an action owes its identity to — the weapon in a combat move, the flames of a fire spell — stay inside the action simply by not being resampled. The representation is invariant to what a pairing resamples, and faithful to what it preserves: what it couples to is a choice of the protocol, not an accident of the data.
Below is the Shadow Library itself — browse the pairing protocols. Each tab is one entry family; watch what stays identical between the two videos (the dynamics) and what changes (everything that was resampled). The pairing is the definition of the action: what a tab preserves is exactly what the latent will control.
From shadow pairs to a world model.
Construct shadow pairs
The Shadow Library replays one dynamics while everything else — character, scene, lighting, camera — is independently resampled. Animation suites, open-world games, and robot simulators all emit the same artifact: two frame-synchronized clips sharing one dynamics.
Cross-shadow prediction
A latent action model reads actions from one shadow and must re-enact them in the other. What the pairing resamples cannot cross the pair, so the latent keeps only what the pair shares — the dynamics, and any trait bound to it — provably the minimal sufficient representation.
Block-causal world model
A video diffusion backbone, conditioned on the action latents and on source motion assets, is converted to a block-causal generator that streams long rollouts from reusable, variable-length action assets.
The same action, in a world it has never seen.
Unlike the per-world demos above, everything in this section comes from a single ShadowDancer model — one checkpoint, one latent interface, four dynamics families. We extract the action from one video and generate from the first frame of its shadow — the ground truth is the other shadow itself. Against Olaf-World, the state-of-the-art self-reconstruction latent-action model retrained on our data with the same backbone, the entangled latent drags along whatever appearance happened to accompany the action and warps its subjects; the shadow-trained latent re-enacts the dynamics faithfully.
Chained commands, judged head-to-head.
Discrete commands become library lookups of stored action assets and stream through the block-causal generator. With no ground truth for a chained command stream, rollouts are judged in blinded two-alternative forced choice against interactive world models driven through their native interfaces — LingBot-World 2.0 (camera poses + text) and Yume-1.5 (text + keystrokes) — and against Olaf-World's latent interface. Each strip below shares the first frame and the command stream; any method whose rollout ends early holds its last frame.
Actions the model has never seen.
“And then he would see the sun itself, and not mere reflections of it — and he would know it to be the source of all that he and his fellows had been seeing.”
The freed prisoner does not understand by staring harder at the wall — he understands the moment he turns around and sees the sun. Once you know the dynamics rather than the shadow, a new shadow is no obstacle. We mod an open-world game with assets that exist nowhere in the training library — a new rifle, a new character, a greatsword — record each primitive action once, and replay the extracted assets elsewhere in the world. The encoder reads dynamics, not memorized action categories — so the action library is open-ended: a newly demonstrated behavior simply joins it.
Every cell below is one action asset: a single primitive — move (W/A/S/D), look (I/J/K/L), fire or attack (F) — demonstrated once with the modded asset and encoded once by the frozen encoder. These nine clips per mod are the model's entire knowledge of the new weapon and character; there is no other label, no fine-tuning.
The extracted assets are chained and replayed elsewhere in the world. The environments are ones the model has seen — the assets are not: the modded gait, the raise, the recoil, and the sword swing all survive the replay.
BibTeX
@article{cao2026shadowdancer,
title = {ShadowDancer: Teaching Video World Models Any Action by Learning
Unified Dynamics Representations from a Video and Its Shadow},
author = {Cao, Jin and Meng, Zian and Zhang, Kaipeng},
journal = {arXiv preprint},
year = {2026}
}