Pilot

A local bridge so any agent harness can drive your Chrome — clicks, keys, typing, tabs, and small screenshots — without stealing your window. You stay in control: clicking Connect in the popup is the handshake, every time.

Get Pilot

01You are the password

The relay is localhost-only. A Chrome profile appears on it only after you click Connect in the Pilot popup — and disappears when you click Disconnect or close Chrome. No auto-reconnect, no background bots.

Use different Chrome profiles for different jobs. Give each one a name, and the harness targets the right one:

# work profile, personal profile — same machine, same relay
node cli.js '{"action":"click","text":"Deploy"}' --profile work
node cli.js '{"action":"shot"}' --profile personal

02Works in the background

Commands run via chrome.scripting.executeScript, which works on inactive tabs. The harness clicks, types, and snapshots a background tab while you keep working. Driven tabs get a colored group (default red, named Harness — both configurable in the Pilot options) so you can always see what the agent is touching.

Background-tab screenshots use the Chrome DevTools Protocol — no tab activation needed. Screenshots are downscaled small so the harness can read them fast.

03Reads screenshots without a vision key

Pilot ships with an on-device OCR tool built on macOS Vision. Screenshot → disk → read, with zero API keys:

node cli.js '{"action":"shot"}' --out /tmp/page.jpg
./ocr /tmp/page.jpg          # plain text
./ocr /tmp/page.jpg --json   # boxes + confidence

04CLI cheat sheet

node cli.js --status	who is connected
{"action":"snap"}	page snapshot — title, url, text, clickable items
{"action":"click","sel":"button.x"}	click by CSS selector
{"action":"clickText","text":"Save"}	click by visible text
{"action":"clickXY","x":100,"y":200}	click at coordinates
{"action":"type","text":"hi"}	type into the focused composer
{"action":"key","key":"Enter"}	press a key
{"action":"fill","sel":"#name","value":"Ada"}	set an input
{"action":"form"} / {"action":"dialog"}	inspect the open form / dialog
{"action":"tabs"} / {"action":"navigate","url":"…"}	tabs and navigation
{"action":"shot"}	small screenshot → saved to ~/.pilot/shots/

Relay status: unknown

05Run it

# 1. relay
node server.js

# 2. load the extension
chrome://extensions  →  Developer mode  →  Load unpacked  →  extension/

# 3. click Connect in the Pilot popup — that's the handshake
# 4. drive it
node cli.js '{"action":"snap"}'

06Give it to an LLM

Copy the install + usage instructions and paste them into any agent harness — it will know how to drive your Chrome.