Visual Sketchpad
Give a multimodal model a sketchpad: let it draw its own lines, boxes, and marks, and call on vision tools, as intermediate steps, so it reasons by sketching the way people do, not just by describing what it sees.
*These are learning tools, not prompts. They teach you to write your own, think of them as training wheels that guide you while you learn, then fall away once you can ride on your own.
Reasoning by Drawing
When people solve a geometry problem or plan a route, they reach for a pencil. They add a line, mark an angle, circle a landmark, and the act of drawing helps them think. Multimodal models, by contrast, usually reason about an image only in words, describing what they see and hoping the description captures what matters.
Visual Sketchpad gives the model a pencil. As it reasons, it can draw auxiliary visual artifacts, lines, boxes, marks, directly as intermediate steps, and it can call on specialist vision models such as object detection and segmentation to sharpen its perception. Each sketch becomes a step in a visual chain of thought, a drawn intermediate result the model can look at and build on.
This is closer to how humans actually sketch than earlier approaches that only generated a finished picture. By letting the model annotate and mark up as it goes, and by grounding perception with dedicated vision tools, Visual Sketchpad turns reasoning about an image into an active, iterative process, draw a little, look, reason, draw again, rather than a single pass of description.
A drawn mark externalizes a thought: circling the relevant region commits the model to it, and a construction line makes a spatial relationship explicit rather than implicit. Pairing this with specialist vision tools means the model is not guessing at where an object is, it can detect and segment it, then reason on top of a grounded sketch. The result is a visual scratchpad that supports exactly the tasks, math and vision problems, where reasoning purely in words tends to slip.
The Visual Sketchpad Process
Four stages of sketching as a visual chain of thought
Receive the Visual Task
Start with a problem where a drawing would help, a geometry question, a spatial reasoning task, or a detailed perception question. The model treats the sketchpad as a workspace it can draw on, not just an image to describe.
“Given this triangle with two known angles, find the third and identify the type of triangle.”
Sketch and Invoke Vision Tools
The model draws auxiliary artifacts, construction lines, boxes around regions, marks on key points, and where perception is uncertain, it calls specialist vision models like detection or segmentation to ground what it sees. Each drawn or detected element is an intermediate reasoning step.
It marks the two known angles, draws an auxiliary line to complete the reasoning, and labels the vertices.
Look at the Sketch and Reason
The model examines its own updated sketch and reasons from it, reading off relationships that are now explicit on the page. Because the intermediate result is visual, the model can catch and correct spatial mistakes that a text-only description would hide.
“From the marked angles, the remaining angle is 180 minus their sum; the sketch shows all three are unequal.”
Iterate to the Answer
The model repeats the loop, sketch, look, reason, adding artifacts until the problem is solved. The final answer rests on a visible sequence of sketches, so the reasoning is both more reliable on visual and mathematical tasks and easier to follow.
“The third angle is 40 degrees, and since all three angles differ, it is a scalene triangle.”
See the Difference
Describing an image versus drawing on it to reason
Describe and Answer
In this diagram of overlapping shapes, how many regions are formed where exactly two shapes overlap?
There appear to be about three regions where two shapes overlap.
Visual Sketchpad
Draw: segment each shape, then shade every region and label how many shapes cover it. Look: count only the regions labeled with exactly two shapes, marking each as it is counted.
Four regions are covered by exactly two shapes, counted directly off the shaded, labeled sketch rather than estimated.
Visual Sketchpad in Action
Sketching intermediate steps across math and vision tasks
“A ladder leans against a wall. Given its length and the distance of its foot from the wall, how high up the wall does it reach?”
Sketch: draw the wall, the ground, and the ladder as a right triangle; label the ladder as the hypotenuse and the foot distance as the base.
Look and reason: with the right angle marked, the height is the missing leg, found by the Pythagorean relationship between the labeled sides.
Answer: the ladder reaches the computed height; drawing the triangle first prevents mislabeling which side is the hypotenuse.
“In this crowded shelf photo, is the small red bottle to the left or right of the tall blue box?”
Invoke tools: run object detection to place bounding boxes around the red bottle and the blue box, grounding their exact positions.
Sketch and look: mark the two boxes and compare their horizontal centers on the annotated image.
Answer: the red bottle sits to the left of the blue box, read directly from the detected, marked positions rather than a vague impression.
“On this floor plan, what is the shortest path from the entrance to the meeting room that avoids the storage area?”
Sketch: mark the entrance, the meeting room, and box off the storage area as a no-go zone; draw candidate paths as lines.
Look and reason: compare the drawn routes, discard any crossing the storage box, and trace the shortest remaining line.
Answer: the highlighted route is shortest while avoiding storage, chosen by drawing and comparing paths rather than describing directions in prose.
When to Use Visual Sketchpad
Best when drawing an intermediate step clarifies the problem
Perfect For
Problems where an auxiliary line, a labeled figure, or a marked construction makes the relationships explicit and prevents slips.
Detailed questions about crowded scenes, where invoking detection or segmentation to ground positions beats a loose verbal description.
Floor plans, maps, and diagrams where marking regions and drawing candidate routes or boundaries makes the answer readable off the sketch.
Settings where the model can both draw and call specialist vision models, so perception is grounded rather than guessed.
Skip It When
When there is nothing visual or spatial to draw, a sketchpad adds machinery with no benefit; a text chain of thought is the right tool.
For a quick “what is this?” where one look suffices, drawing intermediate artifacts is unnecessary overhead.
If the model cannot actually draw on a sketchpad or call vision tools, the approach reduces to ordinary image description.
Use Cases
Where Visual Sketchpad delivers the most value
Math and Geometry Tutoring
Work geometry and spatial math problems by drawing labeled figures and auxiliary constructions, showing students the sketch behind each step.
Detailed Image Analysis
Answer fine-grained perception questions on cluttered images by grounding objects with detection and marking them before reasoning.
Maps and Floor Plans
Plan routes and analyze layouts by marking regions and drawing paths, reading the answer off the annotated plan.
Technical Diagram Interpretation
Trace connections and mark components on schematics and blueprints, building the answer from annotated intermediate sketches.
Scientific Figure Analysis
Mark and measure features in scientific images and plots, using drawn annotations to support each observation.
Visual QA Systems
Build assistants that draw and ground their perception before answering, making visual answers more accurate and inspectable.
Where Visual Sketchpad Fits
From describing images to drawing on them as reasoning
Earlier visual reasoning kept the image fixed and reasoned about it in text. Visual Sketchpad changes what the model can do: it draws its own intermediate artifacts and calls specialist vision tools to ground perception, turning the image into an active workspace. This is closer to how people sketch to think, and it is why the approach helps most on the tasks, mathematical and detailed-perception problems, where reasoning purely in words tends to lose the thread. It sits alongside the other visual chain-of-thought methods here, distinguished by letting the model put marks on the page.
Related Techniques
Explore techniques connected to Visual Sketchpad
Let the Model Draw to Think
Apply Visual Sketchpad to your own math and visual reasoning tasks, or build structured multimodal prompts with our interactive tools.