Cookbook — Giving Claude a Zoom Tool for Fine Image Detail

Source: ai-research/anthropic-cookbook-crop-tool-zoom-2026-07-24.md — full text of multimodal/crop_tool.ipynb from anthropics/claude-cookbooks, fetched 2026-07-24. Compared against the vault’s April copy at raw/claude-cookbooks-main/multimodal/crop_tool.ipynb (see The April to July rewrite). Upstream commit provenance added 2026-07-24 from raw/anthropic-watch-anthropic-cookbook-commit-6c63a145b7136a610092dbd2c3ef412b055e003b.md + raw/anthropic-watch-anthropic-cookbook-commit-85016cacf5b7923aa62aa2b514399325e9d159e9.md.

Claude sees an image once, at a fixed effective resolution. When a chart label or a pair of nearly-touching lines is too small at that resolution, no amount of prompting recovers it — the pixels are gone before the model reads them. This cookbook gives Claude the lean-in move: a zoom tool that takes absolute pixel coordinates, crops that region from the full-resolution original, and returns it magnified to fill the image-token budget. On Surge AI’s Chartography benchmark it takes claude-fable-5 from 29% to 73%. The pattern generalizes past charts to any image where detail is small relative to the whole: dense documents, UI screenshots, schematics, scanned forms, photos of labels.

Key Takeaways

• The whole design follows from how Claude bills images. An image costs one visual token per 28×28-pixel patch. Standard-resolution tier: no side over 1568 px, at most 1568 visual tokens. High-resolution tier (claude-fable-5, claude-sonnet-5): 2576 px / 4784 visual tokens. Anything over the limits is automatically scaled down server-side before Claude sees it — and small details scale down with it.
• The coordinate trap, and its fix. The pixel coordinates Claude emits refer to the image Claude saw — i.e. after that automatic downscale. If you hold the original and Claude describes the downscaled copy, every crop lands in the wrong place. The fix is to pre-resize client-side with the docs’ resized_size() reference implementation so the image you hold is the image Claude sees and coordinates map 1:1.
• Absolute pixels, never normalized. Anthropic’s vision-coordinates guide recommends asking for pixel coordinates explicitly and recommends against normalized 0–1 coordinates. State the dimensions and the top-left origin convention in the prompt.
• Crop from the original, then scale up. Cropping the downscaled copy throws away the pixels you’re trying to recover. Crop the full-res original, then upscale the crop to the largest size the budget allows so every element inside it covers as many 28×28 patches as possible. The notebook’s trick for the upscale target: call resized_size(width * 10_000, height * 10_000) — deliberately overshoot the limits and let the function’s binary search come back down to the largest fitting size.
• Return crops as JPEG, not PNG. Every tool result stays in the conversation; a few full-budget PNG crops of a detailed chart can exceed the API’s 32 MB request size limit. The notebook uses quality=92, subsampling=0 — full chroma resolution, because 4:2:0 subsampling halves color detail and smears the 1–2px colored lines a chart-reading zoom exists to resolve. (For conversations that accumulate many large images, the Files API keeps payloads small regardless of history length.)
• Composite transparency onto white, don’t .convert("RGB"). A bare RGB conversion flattens transparent backgrounds — common in web charts — to black.
• The measured result: +44 points for claude-fable-5 (29% → 73%) and +31 points for claude-sonnet-5 (13% → 44%) on 100 chart-reading questions. It is bought with tokens: mean cost per question goes $0.08\to$0.95 (Fable 5) and $0.02\to$0.64 (Sonnet 5).
• Anthropic’s own escape hatch, stated in the notebook’s summary: if you already run Claude in an agent framework with code execution, you may not need to build any of this — an agent that can run code will typically write its own crop-and-magnify script on demand. This pattern is for the plain Messages API.

The benchmark

Measured on Chartography, a public Surge AI benchmark of 100 chart-reading questions over dense, real-world charts. Every question was asked twice per model — with and without the tool — and free-form answers graded by a model judge per the dataset card.

Model	No tool	With zoom tool	Delta	Mean cost/question (no tool → tool)
claude-fable-5	29%	73%	+44 pts	$0.08\to$0.95
claude-sonnet-5	13%	44%	+31 pts	$0.02\to$0.64

Measurement conditions, stated in the notebook: accuracy over all 100 questions per model per condition on 2026-07-18; cost as the mean of a 30-question usage sample on 2026-07-19 at public API prices (Sonnet 5 at its introductory price), judge excluded. The zoom arms used prompt caching; the no-tool arms were single uncached calls, since caching doesn’t apply to one-shot requests. The loop’s wrap-up call misses the cache, so an optimized client runs slightly cheaper than these figures.

Chartography is a held-out evaluation set — the dataset card ships a canary string precisely so it can be excluded from training. Prompts, golden answers, and metadata are © Surge AI, CC BY 4.0; chart images sourced from the web remain their rights holders'. The cookbook redistributes nothing from the dataset.

When the full view lies — the geometry failure mode

Reading small text is only half of it. The notebook’s second demo asks which of two lines is higher at month 48 of a six-series chart. The two lines sit 26 units apart — roughly 8 pixels, a fraction of one 28×28 patch — right beside a genuine crossing. At that separation the ordering physically cannot be resolved at patch granularity.

• Without the tool: Claude answers confidently and wrong in 36 of 40 recorded attempts (90%), narrating the nearby crossing as if it had already happened.
• With the tool: it zooms the crossing region repeatedly, each crop tighter than the last, and is right in 40 of 40.

The notebook’s own caveat is worth carrying: this near-tie contrast is tuned for claude-fable-5, the strongest chart reader in the benchmark. claude-sonnet-5 called the same crossing correctly only once in 10 attempts even with the tool available. The zoom recovers the pixels; resolving an almost-touching crossing still takes the model’s best visual reasoning. Magnification is a prerequisite, not a substitute.

Implementation

• Tool/Service: Anthropic Messages API + Pillow. pip install anthropic pillow matplotlib; Python 3.11+.
• Setup: copy the notebook’s tool cell — it is deliberately self-contained (Pillow plus the SDK’s type annotations only) so it lifts straight into a project.
• Cost: per the benchmark table above — roughly 10× the no-tool cost per question on detail-bound work, in exchange for the accuracy delta. Each zoom call adds a model round-trip and re-sends the conversation plus a magnified crop.
• Integration notes: the four moving parts are resized_size() (pre-resize + upscale-target math), prepare_image() (make the held image identical to the seen image), zoom_size() (the overshoot trick), and handle_zoom() (clamp → map to original → crop → upscale → return text + JPEG).

The tool schema is four required integers plus an optional image selector:

ZOOM_TOOL = {
    "name": "zoom",
    "description": (
        "Zoom into a rectangular region of an image. Returns the region cropped "
        "from the full-resolution original and magnified, so that small details "
        "become legible."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "x1": {"type": "integer", "description": "Left edge of the region, in pixels (the origin is the image's top-left corner)"},
            "y1": {"type": "integer", "description": "Top edge of the region, in pixels"},
            "x2": {"type": "integer", "description": "Right edge of the region, in pixels (must be greater than x1)"},
            "y2": {"type": "integer", "description": "Bottom edge of the region, in pixels"},
            "image_index": {"type": "integer", "description": "Which image to zoom into, counting images in the conversation from 0. Omit when there is only one image."},
        },
        "required": ["x1", "y1", "x2", "y2"],
    },
}

And the coordinate-mapping core of the handler — note that one scale factor covers both axes because the pre-resize preserved aspect ratio:

scale = original.width / view_w
box = (round(x1 * scale), round(y1 * scale), round(x2 * scale), round(y2 * scale))
cropped = original.crop(box)
zoomed = cropped.resize(zoom_size(cropped.width, cropped.height), PILImage.Resampling.LANCZOS)

The tool result returns text plus image — the text states the requested region, the source region size, and the magnified size, so Claude knows what it is looking at and can decide whether to zoom again.

Running the loop — and the two truncation traps

The notebook drives the exchange with the SDK’s client.beta.messages.tool_runner (@beta_tool builds the schema from the function signature and docstring; max_iterations=20 guards runaway loops). Two failure modes get explicit handling, and both are the reusable part:

• stop_reason == "pause_turn". The API can pause a very long turn server-side, and the runner does not resume these itself. The notebook wraps it in a bounded re-entry loop (5 attempts) that replays the transcript and continues. Skip this and long, thinking-heavy turns silently stop mid-task.
• The iteration budget running out mid-zoom, leaving no text answer at all. The fix is one plain wrap-up turn — “You have used your zoom budget. Answer the original question now, based on everything you have seen.” The notebook is careful about transcript shape here: when the budget ran out, the runner has already appended the final assistant message and its tool results, so appending it again would duplicate the tool calls and the API would reject the transcript.

A third, subtler one: join every text block of the final turn, not just the last. Models that think can emit several, and the answer is not always in the last one.

The appendix rewrites the whole loop with plain client.messages.create calls, which is where the remaining edge cases become visible: executing tool calls whenever they appear (keyed on the presence of calls, not on stop_reason, so a turn that emitted a complete call and then hit the token cap still gets its result), dispatching tool names explicitly so a hallucinated tool name gets an error result back instead of crashing the loop, and catching TypeError on malformed arguments from a call cut off mid-emission.

Multi-image conversations

The schema’s optional image_index (0 for the first image, 1 for the second) handles conversations carrying several images. Each image is labeled with its index and pixel dimensions in the prompt, so Claude can name the image it wants and use the right coordinate space for it:

ask_with_zoom_tool([chart_image, other_chart], "Which of the two charts has more categories?")

The April to July rewrite

The vault’s corpus copy (raw/claude-cookbooks-main/multimodal/crop_tool.ipynb, captured 2026-04-10) is a materially different, much simpler notebook — titled “Giving Claude a Crop Tool for Better Image Analysis”, 22 cells, framed as “here is a useful tool.” The current version is a rewrite: “Giving Claude a Zoom Tool for Reading Fine Image Detail”, 24 cells, framed as a measured result. What the rewrite added:

• The Chartography benchmark with per-model accuracy and cost, and the repeated-run near-tie demo (36/40 wrong → 40/40 right)
• The resized_size() pre-resize discipline and the 1:1 coordinate-mapping rationale — absent from the April version, and the single most likely thing to get silently wrong
• tool_runner + pause_turn handling and the wrap-up turn, versus the April version’s simpler hand-rolled loop
• The JPEG / subsampling=0 / 32 MB and transparency-compositing production notes
• The “you may not need this if your agent can run code” caveat

Anyone who read the April notebook should re-read this one; the coordinate-space fix in particular changes the correctness of the pattern, not just its polish.

Upstream provenance (added 2026-07-24). The rewrite above was originally inferred by diffing the live notebook against the vault’s April corpus copy. The anthropic-ecosystem-watch feed now corroborates it directly with the commit record:

• Commit 6c63a145b7136a610092dbd2c3ef412b055e003b — “Rewrite the crop-tool cookbook around a measured zoom tool”, author cj-ant, 2026-07-23T17:31:03Z, on branch cj-ant/crop-tool-zoom.
• Merge commit 85016cacf5b7923aa62aa2b514399325e9d159e9 — “Merge pull request #793 from anthropics/cj-ant/crop-tool-zoom”, 2026-07-23T17:32:40Z. **PR 793, merged ~90 seconds after the rewrite commit.

The commit message independently confirms every element the diff surfaced, in Anthropic’s own words: “Replace the crop tool with a zoom tool that takes absolute pixel coordinates, crops from the full-resolution original, and returns the region magnified to the model’s image budget. Adds the pre-resize step, multi-image support, benchmark results and figures, and a hand-written loop appendix.” That maps one-to-one onto the five bullets above, so the rewrite inventory is now extracted rather than inferred. It also dates the rewrite to 2026-07-23 — one day before the fetch, which is why the April corpus copy was still the vault’s only record.

Try It

1. Reproduce the failure first. Take a dense chart you actually care about, ask Claude a question hinging on a small label or a near-tie, and record the answer. Without a baseline you can’t tell whether the tool bought you anything.
2. Lift the tool cell verbatim from the notebook — it’s self-contained by design. Set MAX_EDGE/MAX_TOKENS to your model’s tier: 2576 / 4784 for Fable 5 and Sonnet 5, 1568 / 1568 for standard-tier models. Getting this wrong is silent.
3. Pre-resize before you upload, always. Run prepare_image() and confirm the printed “size Claude sees” matches the image you hold. This is the step that makes coordinates trustworthy.
4. Put the dimensions and origin in the prompt — “Coordinates are absolute pixels with the origin at the top-left corner” — and tell Claude to zoom on anything it can’t read confidently.
5. Budget before you ship. At roughly 10× per-question cost, this is for detail-bound questions, not every image call. Route by question type, not by default.
6. Handle pause_turn and the empty-answer case even in a prototype. Both fail silently, which is the worst way to discover them.
7. Check whether you need it at all. If your surface already has code execution, try asking the agent to crop and magnify with its own script first.

Related

• Anthropic Claude Cookbooks — the parent repo catalog; this notebook lives in multimodal/.
• Cookbook — Multi-Agent Research Pattern and Cookbook — Chief of Staff Agent — the other cookbook deep-dives split out of the catalog article.
• Claude Fable 5 + Mythos 5 — the high-resolution-tier model the benchmark headline belongs to.
• Claude Sonnet 5 — the second benchmarked model; its 1-in-10 near-tie result is the caution against assuming the tool closes the gap for smaller models.
• Cost & Intelligence Levers — the ~10× accuracy-for-tokens trade this article measures is a concrete instance of that framework.
• video-analyzer Skill — the adjacent “make Claude see more” problem, solved by routing to another model rather than by re-sending pixels.
• Claude Agent SDK — where the notebook’s “if your agent can run code, it will write this itself” caveat applies.

Open Questions

• No published latency numbers. The notebook names latency as half the trade-off (“higher latency and token usage”) but reports only accuracy and cost.
• Optimal zoom depth is unmeasured. The demos show two successive zooms working; max_iterations=20 is presented as a runaway guard, not a tuned value, and there is no data on where returns diminish.
• Generalization is asserted, not measured. Documents, UI screenshots, schematics, and scanned forms are named as applicable domains, but Chartography is charts only — no benchmark is offered for the others.
• The Files API alternative is mentioned, not demonstrated. It is cited as the fix for conversations accumulating many large images, with no worked example or cost comparison against the JPEG approach.