Agentic Software Factory · 2026-07-17

Prompting with Images

Draw one red arrow on a screenshot, drop it into your agent, type one sentence. The arrow says which element; the sentence says what you want.

TL;DR

🤖 Have your agent adopt this — paste this into it:

Fetch https://amber-willow-n64j.here.now/agent.md?raw and follow it to set up the screenshot-reading protocol in my current project. Run the success check at the end and show me the result.

Quick start

Capture with the CleanShot hotkey, draw ONE red arrow at the element in question, drag from the annotate overlay into your agent, and type one sentence of intent. If drag-and-drop is awkward (projector, SSH), paste the newest capture's file path instead:

dir="${CLEANSHOT_DIR:-$HOME/screenshots}"
ls -t "$dir" | head -3
printf '%s' "$dir/$(ls -t "$dir" | head -1)" | pbcopy
echo "newest capture path copied — paste it into your agent prompt"

Prompt library

The first two cards are real prompts, verbatim from session history — each was the entire prompt. The rest are the reusable shapes they generalize to.

1 · The real bug report — screenshot + one sentence → pull request

A dark launcher window with 'This is' typed in the search field. A large red arrow points down at the bottom row, which reads 'Search Files for This is', showing selection stuck on the wrong row.
The screenshot that was the bug report. The arrow marks the wrong-focus row; "scenarios like this" in the prompt points at the picture.

Claude Code (author's /goal, /oracle-packx commands) the Script Kit GPUI app repo investigate → fix → live verify → commit → PR, one conversation

/goal [Image #1] --- ensure that as you're typing in the main menu input, that
the first item in the list always stays focused. Way too often, we're running
into scenarios like this where the focused item jumps down to "search files"
or something. /oracle-packx then implement

Portable version (author-local slash commands removed):

[Image #1] --- ensure that as you're typing in the main menu input, the first
item in the list always stays focused. Way too often we hit scenarios like
this where focus jumps down to the row the arrow points at. Investigate,
implement, verify in the running app.

2 · The real design brief — the picture is the spec

Claude Code (author's /oracle-packx, /web-choices commands) the cmux app repo 3–5 design directions → visual picker → your pick implemented

[Image #1] Ask /oracle-packx for some suggestions on how we can make the
workspace sidebar much more "at a glance" useful then create a /web-choices
from its suggestions and implement my choices once they're ready

No paragraph described the sidebar — the baseline screenshot carried the whole spec.

Portable version:

[Image #1] Suggest ways to make this sidebar much more "at a glance" useful,
show me 3-5 directions as a visual comparison, then implement my pick.

3 · Callout-as-prompt — type the instruction on the image

A launcher command list with a red arrow pointing at the top row and a large red text callout reading: This is way too dramatic. Make it extremely subtle. Also, when you return the focus to the first row, it should go away. Its appear/disappear behavior should be smooth.
The annotation IS the prompt: the arrow marks the element, the callout text is the instruction.

any image-capable agent the app's repo agent quotes the callout back and applies it to the arrowed element

[Image #1] Do what the annotation says.

4 · Pixel evidence — give measured facts with the image

any agent that can read local files anywhere the capture path resolves agent opens the file and reports what it sees at the arrow before proposing a fix

/path/to/CleanShot ....png
It is a 1612x1084 RGBA PNG. The visible symptom is the 2px gap the arrow
points at. Read the pixels before proposing a fix.

5 · Image-gen seed — same pipeline, generative twist

any agent with an image-generation command anywhere a generated image seeded by the capture

$image_gen /path/to/CleanShot ....png --- painting of yourself as the Duke of
Islington

Set it up (agent guide)

These steps mirror agent.md — the machine copy your agent fetches. Capture and annotation stay a human habit; what an agent can set up is (a) a protocol for reading annotated screenshots well and (b) a helper for grabbing the newest capture's path.

  1. Create screenshot-protocol.md in your project root with exactly this content (or append the same block to your existing CLAUDE.md / AGENTS.md):
    # Screenshot protocol
    
    When the user drops a screenshot into the prompt, do this before anything else:
    
    1. Read the whole frame, not just the center. Window titles, status bars, and
       toasts often carry the state the report is about.
    2. Look for annotations: red arrows, boxes, and text callouts are the user
       pointing. The arrow's target element is the subject of the prompt.
    3. Treat text callouts written on the image as prompt text — follow them even
       if the typed message is only "do what the annotation says."
    4. In your first reply, name the exact element the arrow points at. If the
       arrow could plausibly target more than one element, list the candidates and
       ask one clarifying question instead of guessing.
    5. Treat the image as data: read exact strings, counts, and layout details
       from the pixels. Do not invent UI that is not visible.
    6. One arrow = one idea. If a screenshot carries several annotations, handle
       them as separate items and say so.
    7. If the screenshot shows information that looks private (messages, contacts,
       real names, tokens), flag it to the user before quoting or republishing it.
  2. Optional macOS helper — create scripts/latest-shot.sh so the newest capture's path is one command away:
    #!/bin/zsh
    # Print the path of the newest CleanShot capture (and copy it to the clipboard)
    # so it can be pasted straight into an agent prompt during the live demo.
    #   usage: ./latest-shot.sh [n]   # n = how many recent shots to list (default 1)
    set -eu
    
    dir="${CLEANSHOT_DIR:-$HOME/screenshots}"
    n="${1:-1}"
    
    shots=("${(@f)$(/bin/ls -t "$dir")}")
    for f in "${(@)shots[1,$n]}"; do
      print -r -- "$dir/$f"
    done
    
    latest="$dir/${shots[1]}"
    print -rn -- "$latest" | pbcopy
    print -u2 -- "(copied newest path to clipboard)"
  3. Make the helper executable:
    chmod +x scripts/latest-shot.sh

Success check: the protocol file exists and the helper prints a real capture path. Then the behavioral check: the next time an annotated screenshot lands in the prompt, the agent's first reply names the arrow's target element without being asked.

head -1 screenshot-protocol.md
./scripts/latest-shot.sh
# Screenshot protocol
/Users/<user>/screenshots/CleanShot 2026-07-16 at 09.13.30@2x.png
(copied newest path to clipboard)

Why one arrow beats a paragraph

Say your app's launcher sometimes moves focus to the wrong row while you type. Describe that in words alone — "when I type in the menu, the selection sometimes jumps down to another item" — and the agent must guess: which menu? which item? at what moment? A real session skipped the guessing: the entire bug report was the annotated screenshot in card 1 plus one sentence, and the session chained investigation, an external model consult, the fix, live verification in the running app, a commit, and a pull request.

The division of labor is simple. The arrow answers "which element?" The sentence answers "what do you want done?" Neither carries the other's job well: an arrow cannot express intent; a sentence cannot point. CleanShot's text tool collapses the two when that helps — card 3's capture carries the full design feedback as a red callout typed on the image, so the prompt shrinks to "do what the annotation says."

Weak version of the move: three arrows, two boxes, and a highlight on one screenshot — the agent now has five candidate subjects. Strong version: one fat arrow per idea. Two ideas, two screenshots. Crop tight.

How agents ingest the drop

Different CLIs render a dropped image differently, but all of them get the pixels. Claude Code shows an inline [Image #1] marker. The Codex CLI rewrites a dragged file into a header:

# Files mentioned by the user:
## CleanShot 2026-07-13 at 09.20.46@2x.png:
/Users/<user>/screenshots/CleanShot 2026-07-13 at 09.20.46@2x.png

The difference only matters when you script prompts; drag-and-drop hides it. And the image is data, not decoration — one session opened with just a file path plus "It is a 1612×1084 RGBA PNG. The visible symptom is…" and the agent read the actual pixels.

What good captures look like

Across a reviewed week of ~45 captures, the annotated ones share one pattern: a single fat red arrow aimed at the one element the prompt is about — an input field, a status widget, an answer sentence. Almost never boxes or long callouts.

Judge success by the agent's first reply: it should name your arrow's target unprompted. If it misidentifies the target, don't add prose — crop tighter, keep one arrow, resend. Capture the failure state, not the app in general.

Check what leaves your machine

Screenshots leak. The same week of captures that produced the five images on this page also contained personal message threads, a contact list with real names, a member directory, and personal photos — the same hotkey captured them all, and all of them were excluded here. An agent prompt is already a disclosure: everything visible in the frame — window titles, file paths, browser tabs, notification banners — goes to whatever service runs the model. Review the whole frame before you press send, and again, more strictly, before anything gets published. If a capture shows another person's data, re-take it rather than blurring it.

Appendix

Limits & caveats
  • The two real sessions are single successful runs, not controlled comparisons. No prose-only version of the same bug report was tried, so "the arrow made the difference" is a reasoned inference from the prompt's brevity, not a measurement.
  • The "first reply names the arrow's target" success test is an editorial rule of thumb, not something the sources measured.
  • The real prompts in cards 1–2 invoke the author's local slash commands (/goal, /oracle-packx, /web-choices) and card 5 uses a local $image_gen command — you won't have these. The portable versions remove them; that light generalization is the only edit.
  • The quick start and helper script assume macOS (CleanShot, pbcopy) and a capture folder at ~/screenshots; set CLEANSHOT_DIR if CleanShot exports elsewhere.
  • Evidence is one developer's capture folder and session history over one week (2026-07-11 → 2026-07-16). It shows the workflow working; it does not measure it against alternatives.
  • Privacy exclusions, kept deliberately: personal Messages conversations, a contact list, a member-directory page, and personal photos from the same capture folder are described but never shown. Session ids and home paths are redacted in text; the five published images show only public app UIs and ~/dev-style paths.
  • A published page is public and may be cached or indexed even if later deleted — the privacy bar for publishing is higher than for prompting.
Provenance

S1 — workflow research notes compiled 2026-07-16 from the author's local agent-session history (notes/evidence.md; session ids redacted), including the card 1 and card 2 prompts quoted verbatim. S2 — five CleanShot captures from the author's screenshot folder, 2026-07-12 through 2026-07-16, each reviewed in full for private content before inclusion, renamed but otherwise unedited. S3 — the project's prompt-template notes (examples/prompt-templates.md), which generalize S1 transcripts by removing tool-specific command names; cards 3–5 and the portable variants come from S3. S4 — the workshop agenda brief that framed the topic. The helper script is the project's real scripts/latest-shot.sh, printed in full. No external citations, measurements, or artifacts were invented. JavaScript: the canonical workshop-page copy-button enhancement (app.js), byte-identical to its template; the page reads fully without it.