Following this file sets up: a screenshot-reading protocol so you (the agent) consume annotated screenshots well — arrows, callouts, pixel evidence — plus an optional macOS helper that prints the newest capture's path. Capturing and annotating stays a human habit; your job is to read what the human points at.
Placeholders used below: <project-root> = the project you are working in. All other paths are literal.
Human companion pages on this site (you don't need them to execute this file): index.html is the full section guide, replay.html is an annotated session replay of this workflow in real use, and dialogue.html walks the same material as attendee Q&A.
Check each before starting:
uname -s
ls "${CLEANSHOT_DIR:-$HOME/screenshots}" | head -3
Expected: Darwin (the helper script in step 2 uses macOS pbcopy; the protocol in step 1 is OS-agnostic — on other platforms do step 1 only), and a listing of capture files. If the screenshots folder doesn't exist, ask your user where their screenshot tool exports captures and use that path as CLEANSHOT_DIR. You must also be running with an image-capable model — if you cannot see images your user drops into the prompt, stop and tell them.
Create <project-root>/screenshot-protocol.md with exactly this content. If the project already has an agent-instructions file (CLAUDE.md, AGENTS.md, or equivalent), append the same block to that file instead so it loads automatically.
# Screenshot protocol
When the user drops a screenshot into the prompt, do this before anything else:
1. Read the whole frame, not just the center. Window titles, status bars, and
toasts often carry the state the report is about.
2. Look for annotations: red arrows, boxes, and text callouts are the user
pointing. The arrow's target element is the subject of the prompt.
3. Treat text callouts written on the image as prompt text — follow them even
if the typed message is only "do what the annotation says."
4. In your first reply, name the exact element the arrow points at. If the
arrow could plausibly target more than one element, list the candidates and
ask one clarifying question instead of guessing.
5. Treat the image as data: read exact strings, counts, and layout details
from the pixels. Do not invent UI that is not visible.
6. One arrow = one idea. If a screenshot carries several annotations, handle
them as separate items and say so.
7. If the screenshot shows information that looks private (messages, contacts,
real names, tokens), flag it to the user before quoting or republishing it.
(Optional, macOS only) Create <project-root>/scripts/latest-shot.sh with exactly this content, so the newest capture's path is one command away when drag-and-drop is awkward:
#!/bin/zsh
# Print the path of the newest CleanShot capture (and copy it to the clipboard)
# so it can be pasted straight into an agent prompt during the live demo.
# usage: ./latest-shot.sh [n] # n = how many recent shots to list (default 1)
set -eu
dir="${CLEANSHOT_DIR:-$HOME/screenshots}"
n="${1:-1}"
shots=("${(@f)$(/bin/ls -t "$dir")}")
for f in "${(@)shots[1,$n]}"; do
print -r -- "$dir/$f"
done
latest="$dir/${shots[1]}"
print -rn -- "$latest" | pbcopy
print -u2 -- "(copied newest path to clipboard)"
Make the helper executable:
chmod +x scripts/latest-shot.sh
Run (from <project-root>):
head -1 screenshot-protocol.md
./scripts/latest-shot.sh
Expected observable result: the first command prints # Screenshot protocol (or, if you appended to an existing instructions file, grep -c "Screenshot protocol" CLAUDE.md AGENTS.md 2>/dev/null shows a non-zero count for that file); the second prints an absolute path to the newest capture, like /Users/<user>/screenshots/CleanShot 2026-07-16 at 09.13.30@2x.png, plus (copied newest path to clipboard) on stderr.
Then the behavioral check, which is the one that matters: the next time your user drops an annotated screenshot, your first reply must name the exact element the arrow points at — before proposing anything.
Report to your user: what you created (and where the protocol lives), the success-check output, and anything you had to adapt.
# Files mentioned by the user: header with a local file path instead of an inline image marker. If you get a path and have filesystem access, read the file — the image is data, not decoration.$HOME/screenshots but CleanShot can export anywhere. If latest-shot.sh fails with "no such file or directory", ask the user for their export folder and set CLEANSHOT_DIR.# Prompting with Images — agent setup guide
Following this file sets up: a screenshot-reading protocol so you (the agent) consume annotated screenshots well — arrows, callouts, pixel evidence — plus an optional macOS helper that prints the newest capture's path. Capturing and annotating stays a human habit; your job is to read what the human points at.
Placeholders used below: `<project-root>` = the project you are working in. All other paths are literal.
Human companion pages on this site (you don't need them to execute this file): `index.html` is the full section guide, `replay.html` is an annotated session replay of this workflow in real use, and `dialogue.html` walks the same material as attendee Q&A.
## Prerequisites
Check each before starting:
```bash
uname -s
ls "${CLEANSHOT_DIR:-$HOME/screenshots}" | head -3
```
Expected: `Darwin` (the helper script in step 2 uses macOS `pbcopy`; the protocol in step 1 is OS-agnostic — on other platforms do step 1 only), and a listing of capture files. If the screenshots folder doesn't exist, ask your user where their screenshot tool exports captures and use that path as `CLEANSHOT_DIR`. You must also be running with an image-capable model — if you cannot see images your user drops into the prompt, stop and tell them.
## Steps
1. Create `<project-root>/screenshot-protocol.md` with exactly this content. If the project already has an agent-instructions file (`CLAUDE.md`, `AGENTS.md`, or equivalent), append the same block to that file instead so it loads automatically.
```markdown
# Screenshot protocol
When the user drops a screenshot into the prompt, do this before anything else:
1. Read the whole frame, not just the center. Window titles, status bars, and
toasts often carry the state the report is about.
2. Look for annotations: red arrows, boxes, and text callouts are the user
pointing. The arrow's target element is the subject of the prompt.
3. Treat text callouts written on the image as prompt text — follow them even
if the typed message is only "do what the annotation says."
4. In your first reply, name the exact element the arrow points at. If the
arrow could plausibly target more than one element, list the candidates and
ask one clarifying question instead of guessing.
5. Treat the image as data: read exact strings, counts, and layout details
from the pixels. Do not invent UI that is not visible.
6. One arrow = one idea. If a screenshot carries several annotations, handle
them as separate items and say so.
7. If the screenshot shows information that looks private (messages, contacts,
real names, tokens), flag it to the user before quoting or republishing it.
```
2. (Optional, macOS only) Create `<project-root>/scripts/latest-shot.sh` with exactly this content, so the newest capture's path is one command away when drag-and-drop is awkward:
```zsh
#!/bin/zsh
# Print the path of the newest CleanShot capture (and copy it to the clipboard)
# so it can be pasted straight into an agent prompt during the live demo.
# usage: ./latest-shot.sh [n] # n = how many recent shots to list (default 1)
set -eu
dir="${CLEANSHOT_DIR:-$HOME/screenshots}"
n="${1:-1}"
shots=("${(@f)$(/bin/ls -t "$dir")}")
for f in "${(@)shots[1,$n]}"; do
print -r -- "$dir/$f"
done
latest="$dir/${shots[1]}"
print -rn -- "$latest" | pbcopy
print -u2 -- "(copied newest path to clipboard)"
```
3. Make the helper executable:
```bash
chmod +x scripts/latest-shot.sh
```
## Success check
Run (from `<project-root>`):
```bash
head -1 screenshot-protocol.md
./scripts/latest-shot.sh
```
Expected observable result: the first command prints `# Screenshot protocol` (or, if you appended to an existing instructions file, `grep -c "Screenshot protocol" CLAUDE.md AGENTS.md 2>/dev/null` shows a non-zero count for that file); the second prints an absolute path to the newest capture, like `/Users/<user>/screenshots/CleanShot 2026-07-16 at 09.13.30@2x.png`, plus `(copied newest path to clipboard)` on stderr.
Then the behavioral check, which is the one that matters: the next time your user drops an annotated screenshot, your first reply must name the exact element the arrow points at — before proposing anything.
Report to your user: what you created (and where the protocol lives), the success-check output, and anything you had to adapt.
## Notes for agents
- Multiple arrows or callouts on one image: don't average them into one request. Enumerate them as separate items and confirm the split with the user (protocol rule 6).
- Ambiguous arrow target (e.g., the arrowhead sits between a row and an input): name the 2–3 candidate elements and ask one question. Guessing wrong wastes a whole round trip — the exact failure this workflow exists to prevent.
- Some CLIs (e.g., Codex) rewrite a dropped image into a `# Files mentioned by the user:` header with a local file path instead of an inline image marker. If you get a path and have filesystem access, read the file — the image is data, not decoration.
- The screenshots folder defaults to `$HOME/screenshots` but CleanShot can export anywhere. If `latest-shot.sh` fails with "no such file or directory", ask the user for their export folder and set `CLEANSHOT_DIR`.
- If a dropped screenshot visibly contains private information (message threads, contact lists, tokens), flag it before quoting it back, writing it to files, or including it in anything that leaves the machine (protocol rule 7).