Video-of-Thought
Bring step-by-step reasoning to video: work from low-level pixel perception up to high-level interpretation, grounding each step in a spatial-temporal map of what is happening, so the model reasons about a clip instead of just captioning it.
*These are learning tools, not prompts. They teach you to write your own, think of them as training wheels that guide you while you learn, then fall away once you can ride on your own.
Reasoning Through a Video, Not Just Watching It
Video is hard because it is both spatial and temporal: objects move, interact, and cause effects over time. Ask a model “why did the second car stop?” and a single-pass answer often skips the perception work, missing which objects are involved and how their actions connect across frames.
Video-of-Thought applies chain-of-thought reasoning to video. It breaks a complex question into simpler sub-problems and solves them step by step, moving from low-level pixel perception, what objects are present and how they move, up to high-level cognition, what it means and why it happened. Instead of jumping to an interpretation, the model builds up to one.
To keep those steps grounded, the framework pairs with a video model that represents the clip as a spatial-temporal scene graph: objects, their locations, and how they relate and change over time, tracked at a fine, pixel-level grain. That structured representation gives the reasoning something concrete to stand on, so each step refers to actual tracked objects and events rather than a vague memory of the footage. It was the first approach to bring chain-of-thought reasoning to video in this way.
The core discipline is order: perceive before you interpret. By resolving what is in the video and how it moves before asking what it means, Video-of-Thought avoids the common failure of confidently explaining an event the model never correctly perceived. The spatial-temporal scene graph is what makes this reliable, it anchors each reasoning step to specific objects and their tracked interactions, so high-level conclusions trace back to grounded low-level observations.
The Video-of-Thought Process
Four stages from pixels to grounded interpretation
Perceive Objects and Motion
Start at the pixel level: identify the objects in the clip and track how they move across frames. This low-level perception is captured as a spatial-temporal scene graph, a structured record of what is present, where, and how it changes over time.
For a traffic clip: two cars and a pedestrian are tracked, with their positions and speeds recorded frame by frame.
Decompose the Question
Break the complex question into simpler sub-problems that can be answered in sequence. This turns a single hard interpretation into a chain of manageable steps, each one building on the grounded perception from the previous stage.
“Why did the second car stop?” becomes: what did the pedestrian do, when, and how did the second car’s motion change in response?
Reason From Perception to Cognition
Work through the sub-problems in order, moving from concrete observations toward higher-level meaning. Each step consults the scene graph, so inferences about intent, cause, or outcome stay tied to the tracked objects and their interactions rather than drifting into guesswork.
“The pedestrian stepped into the crosswalk at second 3; the second car’s speed drops to zero right after, so it stopped to yield.”
Compose the Final Answer
Combine the sub-answers into a coherent interpretation of the video. Because every step is grounded in perceived, tracked events, the final answer is both more accurate on complex video questions and traceable back through the reasoning chain.
“The second car stopped because a pedestrian entered the crosswalk directly in its path, and it yielded to avoid a collision.”
See the Difference
Single-pass video answering versus grounded step-by-step reasoning
Single-Pass Answer
In this clip, why does the player suddenly change direction?
The player changes direction because they are dribbling and decided to go the other way.
Video-of-Thought
Perceive: track the player, the ball, and a defender approaching from the left. Reason: the defender closes the lane just before the direction change; the player cuts away from the defender toward open space.
The player changes direction to avoid the defender who cut off the left lane, moving into the open space that opened on the right.
Video-of-Thought in Action
Perception-to-cognition reasoning across different video questions
“A glass falls off a table in this clip. What caused it to fall?”
Perceive: track the glass, a cat on the table, and the cat’s paw across frames.
Decompose and reason: just before the glass moves, the cat’s paw contacts it; the glass then slides to the edge and falls. The timing links the contact to the fall.
Answer: the cat knocked the glass off the table with its paw, grounded in the tracked contact that immediately preceded the fall.
“In this cooking clip, did the chef season the dish before or after tasting it, and why does the order matter?”
Perceive: track the spoon (tasting) and the salt (seasoning) actions with their timestamps.
Decompose and reason: the tasting action occurs first, then the seasoning; tasting before seasoning is how a cook decides how much salt to add.
Answer: the chef tasted first, then seasoned, using the taste to judge the amount of salt needed, an intent inferred from the grounded order of actions.
“Two people pass a package in a busy hallway. Who ends up holding it, and did anyone else touch it?”
Perceive: track the package and every hand that comes near it across the clip.
Decompose and reason: the package moves from person A to person B; a third passerby brushes past but never makes contact with it.
Answer: person B ends up holding the package, and no one else touched it, resolved by following the package and the tracked hand contacts frame by frame.
When to Use Video-of-Thought
Best for complex video questions that need grounded reasoning
Perfect For
Video questions about why something happened, where the answer depends on correctly perceiving which objects interacted and in what order.
Tasks about the order, timing, or progression of events, where tracking actions across frames is essential to a correct answer.
Busy scenes with several moving objects, where a grounded scene graph keeps track of who did what to whom over time.
When you need the reasoning trail, which observations led to which conclusion, not just a one-line interpretation of a clip.
Skip It When
For a one-line summary of an obvious scene, the full perception-to-cognition pipeline is more machinery than the task needs.
If the answer is visible in one still frame with no motion or timing involved, image reasoning is simpler and sufficient.
The benefit depends on fine-grained tracking and a scene-graph representation. Without that grounding, the reasoning has nothing solid to stand on.
Use Cases
Where Video-of-Thought delivers the most value
Video Question Answering
Answer complex questions about clips, causal, temporal, and interaction questions, with reasoning grounded in tracked objects and events.
Surveillance and Incident Review
Reconstruct what happened in a clip by tracking objects and interactions, producing an inspectable account of the sequence of events.
Sports and Action Analysis
Explain plays and movements by tracking players and the ball over time, reasoning from their timed interactions rather than a single glance.
Autonomous and Robotics Perception
Reason about dynamic scenes, why an agent stopped or yielded, by grounding conclusions in tracked motion and interactions.
Instructional Video Understanding
Follow how-to clips step by step, tracking the order of actions and their purpose to answer questions about procedure and intent.
Content Moderation and Analysis
Assess events in video with a grounded, step-by-step account, so decisions rest on tracked evidence rather than a single impression.
Where Video-of-Thought Fits
Extending chain-of-thought reasoning into video
Chain-of-thought reshaped text reasoning, and Video-of-Thought was the first framework to carry that idea into video in a grounded way. Its discipline, perceive before you interpret, and its pairing with a fine-grained spatial-temporal scene graph let a model reason about causes, timing, and interactions instead of captioning a clip in one pass. That grounding is the key: because each high-level conclusion traces back to tracked, low-level observations, the reasoning holds up on exactly the complex video questions where single-pass answers tend to fail.
Related Techniques
Explore techniques connected to Video-of-Thought
Reason Through the Whole Clip
Apply Video-of-Thought to your own complex video questions, or build structured multimodal prompts with our interactive tools.