A browser agent that never sees the password and writes up what it did

What could go wrong, and how we would know

Two things go wrong with browser agents. They loop, clicking the same dead element until the budget runs out. And they leak, because the easiest way to log in is to paste the password into the prompt.

The loop is handled with a counter. Every action appends a line to the history; failures are prefixed FAILED and shown back to the model as "failed", so it sees its own misses. consecutive_failures resets on success. At 2 the prompt gets a warning suggesting Escape or a direct goto; at 4 the loop exits. Every exit path, including the aborts, routes into the documentation agent, so a stuck run still produces a write-up of how far it got.

The leak is handled by never giving the model the value. The action schema has credential_site and credential_type fields, both enums. The model says, in effect, "fill this index with the github password"; execute_action builds the key GITHUB_PASSWORD, reads it with os.getenv, types it, and prints [HIDDEN]. Usernames log as three characters and asterisks. A missing variable raises, which counts as a failed step rather than a silent fallback.

What I built

Four nodes. get_page_state runs a script in the page that collects every link, button, input, select, textarea and role="button" or role="link" element, drops the invisible ones, and only then assigns an index. The order matters: the number the model returns has to point at the element it saw. Each element carries a unique selector when it has an id, data-testid or input name, and its center coordinates either way.

plan_action sends a JPEG screenshot, the request, the history and the element list to gpt-5-mini bound to a Pydantic PlaywrightAction, so the reply is one of five actions (click, fill, press, goto, wait_for_user) plus a one-line reason. Past 60 interactive elements the list is trimmed: inputs first, then up to 40 named elements, then a short tail.

execute_action uses the selector when it exists and is visible, and clicks the recorded coordinates when it does not. wait_for_user blocks on input() so a human can clear 2FA or a CAPTCHA. After every successful action it stores a PNG plus the action, reason and URL for the documentation agent.

The documentation agent, and the four runs

After the loop, generate_documentation sends the request, the full action history and every post-action PNG to the same model bound to a TutorialOutput schema: title, introduction, steps that each reference a screenshot index, and a conclusion. The prompt says to keep only the essential states, "typically 3-7 screenshots", to skip failed actions and retries, and to merge micro-actions. export_tutorial writes tutorial.md, tutorial.json and only the selected PNGs into a timestamped folder.

Four runs are committed:

• YouTube, play a named channel's latest video: three steps.
• GitHub, add a README to an empty repository: six steps. Click Sign in, fill both credentials from the environment and submit, open the repository, click Add a README, edit, commit. This run exercises credential indirection end to end.
• Notion, create a page titled "hello" with a description: four steps. Notion is not in the credential enum (only github is), so that sign-in was manual or already authenticated; the tutorial does not record which.
• YouTube, search "langchain" and play the first result: three steps, with the search box click, the typing and the Enter press folded into one step by the curator.

Sixteen screenshots in all. None are embedded here: they were captured inside my own accounts and show a GitHub username and a Notion workspace name. That is also the limit of the indirection. It keeps the password out of the prompt and the logs, not the username or anything the page renders afterwards, and those frames go to the documentation model and into the repo.

Outcome, and what it does not prove

The four runs show the loop completing real tasks on live sites and the second agent producing readable tutorials with sensible step boundaries. They do not show reliability: there is no success rate, no run that hit the abort path is committed, and the curated tutorial drops failures by design, so the repo holds a clean-path record rather than an audit log. The full history is printed to stdout and never saved.

What I would do differently

Persist the unfiltered history and every captured frame next to the tutorial; pin requirements.txt; write a test for the element cap, because others[:20 - len(inputs)] goes negative once a page has more than 20 inputs, and a negative slice keeps everything except the last few, the opposite of a cap; and gate task_complete on a second look at the page instead of trusting the planner's own claim.