Interleaved-Modal Chain-of-Thought
Instead of reasoning about an image in words alone, weave the actual image regions into each step of the chain, so every rationale pairs what the model says with what it is looking at.
*These are learning tools, not prompts. They teach you to write your own, think of them as training wheels that guide you while you learn, then fall away once you can ride on your own.
Reasoning That Points Back at the Image
Most multimodal reasoning is only multimodal at the start. The model sees an image, then produces a chain of thought made entirely of text. Once reasoning begins, the picture is left behind, and every later step relies on the model’s own written description of what it saw. If that first description missed a detail, nothing downstream can recover it.
Interleaved-Modal Chain-of-Thought keeps the image in the loop. Each reasoning step is a pair: a textual rationale alongside the specific visual region it depends on. Rather than describing a chart once and reasoning from memory, the chain re-grounds itself in the actual pixels at the moment they matter, so the evidence and the argument stay side by side.
The mechanism that makes this practical is Attention-driven Selection. The model already forms an internal attention map over the image while it reasons. ICoT reads that map to find which regions the current step is focused on, then inserts those regions into the reasoning sequence. Because it works off attention the model already computes, it needs no extra training and no new parameters, it plugs into an existing vision-language model as-is.
A text-only rationale is a lossy snapshot of the image taken at step one. Any visual detail it omits is gone for the rest of the chain. Interleaving instead lets each step reach back to the pixels it needs, so a later step can attend to a small label or edge the opening description never mentioned. Pairing text with the exact region it refers to also makes the reasoning auditable: you can see not just what the model concluded, but which part of the image it was looking at when it concluded it.
The Interleaved-Modal Process
Four stages that alternate between looking and reasoning
Present Image and Question
Provide the full image and the question together. The model forms its initial understanding of the scene and begins the first reasoning step, exactly as it would for standard multimodal reasoning. Nothing special is required of the prompt itself, the interleaving happens inside the reasoning process.
“This product photo shows a shelf of packaged goods. Which item has the earliest expiry date printed on its label?”
Read the Attention Map
As the model reasons, it attends more strongly to some parts of the image than others. Attention-driven Selection inspects this attention map to identify the regions the current step is actually relying on. This selection is derived from signals the model already produces, so it adds no training and no new parameters.
While considering expiry dates, the model’s attention concentrates on the small printed date fields near the base of each package rather than the brand logos.
Insert the Selected Region
The chosen image region is inserted back into the reasoning sequence next to the textual rationale for that step. The result is an interleaved step: a written observation paired with the exact visual evidence behind it. The model now reasons about the region directly instead of about an earlier text summary of it.
“Focusing on the date field of the second package [region inserted], the printed value reads 2026-03. This is earlier than the dates visible on the neighboring items.”
Continue and Conclude
The chain repeats: reason, attend, insert, reason again, comparing regions across steps until the question is resolved. Because each step carries the region it relied on, the final answer is both grounded and interpretable, you can trace which piece of the image supported each part of the conclusion.
“Comparing the date fields across all four packages, the second item at 2026-03 has the earliest expiry. Answer: the second package from the left.”
See the Difference
How re-grounding in the image prevents lost-detail errors
Text-Only Multimodal CoT
“The image shows several instruments on a dashboard. The speedometer appears to read around 60. The fuel gauge looks near half. Based on this, the driver has plenty of fuel to continue.”
The tank is about half full, so no refueling is needed.
Interleaved-Modal CoT
Text: To judge fuel, I need the fuel gauge specifically. Attend and insert: the region around the fuel gauge is selected [region inserted]. Text: Zoomed to that region, the needle sits just above the E mark with the low-fuel light lit, not near half.
The fuel gauge reads near empty and the low-fuel warning is on. The driver should refuel soon.
Interleaved-Modal CoT in Action
See how alternating between looking and reasoning works across different domains
“In this street photo, what does the smaller sign mounted below the main traffic sign say?”
Step 1 (text): The question is about a small secondary sign, so I need the region directly beneath the large sign, not the whole scene.
Step 2 (attend and insert): Attention concentrates on the panel below the main sign; that region is inserted into the chain.
Step 3 (text on region): Reading the inserted region closely, the smaller panel shows the text “Except Permit Holders.”
Answer: The secondary sign reads “Except Permit Holders,” qualifying the parking restriction above it.
“This figure has four small line charts. In which panel does the blue line cross above the orange line?”
Step 1 (text): I need to inspect each panel’s crossing point separately rather than judge the figure as a whole.
Step 2 (attend and insert): The chain attends to each panel in turn, inserting the top-left, top-right, bottom-left, and bottom-right regions one at a time.
Step 3 (text on regions): In the inserted top-right region, the blue line rises above the orange line near the midpoint. In the other three inserted regions, blue stays below orange throughout.
Answer: The blue line crosses above the orange line only in the top-right panel.
“On this scanned invoice, is the total consistent with the sum of the line items?”
Step 1 (text): I need the line-item column and the total field, two separate regions.
Step 2 (attend and insert): The chain inserts the itemized rows region, then the total region.
Step 3 (text on regions): The inserted rows read 120.00, 45.50, and 18.75, summing to 184.25. The inserted total field reads 184.25.
Answer: Yes, the printed total of 184.25 matches the sum of the three visible line items.
When to Use Interleaved-Modal CoT
Best for tasks where small visual details decide the answer
Perfect For
Tasks where the answer hinges on a small region, a tiny label, a single gauge, one panel of many, that a whole-image description would gloss over.
Crowded scenes, multi-panel figures, and busy dashboards where the model needs to isolate and inspect one region at a time to avoid mixing up details.
Situations where you need to verify not just the answer but which part of the image justified it, since each step carries the region it relied on.
Deployments where retraining is not an option; the attention-driven selection plugs into a model as-is, adding no parameters.
Skip It When
For overall scene captioning or mood description, there is no single region to isolate, and interleaving adds overhead without benefit.
When one look answers the question, a full interleaved chain is unnecessary machinery for a task the model already handles directly.
With no image to attend to, there is nothing to interleave; a standard text chain-of-thought is the right tool.
Use Cases
Where Interleaved-Modal CoT delivers the most value
Detailed Image QA
Answer questions that depend on tiny labels, badges, or readouts by isolating and inspecting the exact region each question points to rather than the whole frame.
Chart and Figure Analysis
Read multi-panel figures and dense plots panel by panel, inserting each region as its turn comes so crossings, peaks, and outliers are judged on the real pixels.
Document Understanding
Cross-check totals, dates, and fields on scanned forms and invoices by pulling each relevant field into the chain and reasoning over it directly.
Medical and Scientific Imaging
Support careful review by grounding each observation in the specific region it came from, making the reasoning trail inspectable rather than a single opaque verdict.
Visual Inspection and QA
Spot defects, mismatches, or missing parts in product and manufacturing images by focusing the chain on one candidate region at a time.
Interpretable Multimodal Systems
Build assistants whose visual reasoning can be explained region by region, useful anywhere a decision needs a visible evidence trail.
Where Interleaved-Modal CoT Fits
From text rationales to reasoning that carries its own visual evidence
Multimodal Chain-of-Thought made reasoning about images explicit, but its rationales were still written in text, a description of the image, reasoned over from memory. Interleaved-Modal CoT closes that gap by carrying the visual evidence forward into every step. Its Attention-driven Selection needs no retraining because it reuses attention the model already computes, which makes it a practical upgrade for existing vision-language systems rather than a new model to build. The principle it embodies, keep the evidence next to the argument, is why interleaving improves both accuracy and interpretability at once.
Related Techniques
Explore techniques connected to Interleaved-Modal CoT
Keep the Image in the Loop
Apply Interleaved-Modal CoT to your own detail-critical visual tasks, or build structured multimodal prompts with our interactive tools.