@castle-ai/connectors

@castle-ai/connectors

Connect Castle agents to outside systems. Each connector is its own entry point and brings its own optional dependency, so an application installs only what it uses.

Import	Connector	Install alongside
@castle-ai/connectors/browser	A Playwright-driven browser for the session-scoped browser tools in @castle-ai/node-runner	playwright-core

Browser

npm install @castle-ai/connectors playwright-core

Declare browser access

Add browser(...) to an Agent's capabilities. The capability provides the browser Tools and a browser instructions section with their usage rules. With your application's existing model binding:

import { defineAgent } from "@castle-ai/harness";
import { openSession, terminalApproval } from "@castle-ai/node-runner";
import { browser } from "@castle-ai/connectors/browser";

const Reviewer = defineAgent("example.browser-review", {
  model,
  instructions: "Inspect the page and verify its controls.",
  capabilities: [browser({
    mode: "managed",
    channel: "chrome", // Use installed Chrome in a separate, owned profile.
    origins: ["http://localhost:3302"],
  })],
});

const controller = new AbortController();
const session = await openSession({
  agent: Reviewer,
  approval: terminalApproval({
    input: process.stdin,
    output: process.stdout,
    onInterrupt: () => controller.abort(new Error("User stopped")),
  }),
});
try {
  console.log((await session.run("Inspect http://localhost:3302", { signal: controller.signal }).result).text);
} finally {
  await session.close();
  process.stdin.destroy(); // This CLI owns its approval input.
}

Collection and Session acquisition do not launch a browser or visit a page. The first approved browser_navigate opens it; the same browser is used across Runs until Session close. All ten tools use the ordinary permission policy; the default openSession policy requires approval for browser operations. enabled (a boolean, or a function of the turn context) hides the Tools and their rules for a turn; re-enabling reuses the same Session browser resource. Browser configuration is fixed for that Session.

Managed defaults are Chromium, headless, 10 s action timeout, temporary profile, resourceAccess: "same-origin", 16 KiB observation budget and 2 MiB image budget. playwright-core does not install browsers: supply an installed channel such as "chrome", or provision a browser matching Playwright 1.55.0. An unavailable browser fails explicitly. profilesDirectory and captureWorkspaceRoot take file URLs; the latter permits relative screenshot save paths. Omitting origins admits HTTP(S) URLs without embedded credentials; supplying an array limits navigation to those exact canonical Origins. Neither choice bypasses approval.

For an existing Chrome tab, the host connects and explicitly selects an entry from connection.listTabs() before creating the Agent:

capabilities: [browser({ mode: "selected", connection, tab })],

Here connection is an ExistingChromeConnection, and tab is the host-selected inventory entry. The capability neither discovers profiles nor chooses another tab. The grant is acquired only on the first approved action; browser_snapshot can therefore be the first tool. Closing this Session releases its grant, leaving the host's connection and tabs open. A revoked grant stays revoked; there is no automatic reacquisition or managed-browser fallback.

The capability and createBrowserSessionPlugin share Node Runner's createBrowserSessionTools owner. Custom integrations can use that factory directly and must await close() to finish all owned browser operations.

Browser behavior and boundaries

The Adapter owns one Browser per Runner Session. Navigation admits an exact Origin for the Session. browser_navigate with newTab: true keeps the current page open; browser_switch_tab selects an observed tab ID without navigation. Every observation lists the open tabs, so reference sites and local previews can retain independent form state. Tab IDs expire when the Browser closes at Session end. Accessible role/name locators remain the action boundary.

Accessible observations include opaque ref values for individual elements. For repeated controls, pass the latest observed ref alongside its role/name and optional ancestor. The Adapter intersects the exact referenced node with the semantic locator; a reference cannot override mismatched role/name/scope. Each observation replaces the reference set, and navigation clears it. Re-read the page after a stale-reference error. References identify observed DOM nodes, not durable application records, and do not survive browser restart.

The reference owner uses Playwright 1.55.0's private _snapshotForAI and aria-ref selector through one local typed Adapter seam. This is deliberately pinned: upgrades must pass the real-browser reference, focus and network tests. There is no fallback to positional matching, CSS supplied by the model, or a second browser runtime. Focus is observed before the page snapshot because Playwright stores one most-recent ARIA reference map per frame.

The factory receives the durable Runner Session identity. With a host-owned profilesDirectory, each Session uses its own persistent browser profile; website storage survives closing and reopening that Session. Without this option, Playwright uses a temporary profile. Desktop supplies a directory under its application data, never the user's personal browser profile. Profile storage does not grant Origin access: the existing authorization and network restrictions still apply every time the browser opens. Closing a Session releases its browser process but does not delete the host's persistent profile.

For browser_capture, it applies the Tool's explicit viewport, light | dark color scheme, and reduce | no-preference reduced-motion preference before capturing the visible viewport as PNG. The Node Runner returns the bounded page observation first and the PNG image block second. maxObservationBytes and maxImageBytes remain Node Runner trust-boundary limits; the Adapter owns only Playwright emulation and screenshot bytes.

Top-level navigation and redirects may move between the Session's admitted Origins. A link or redirect to an unapproved website fails with only the destination Origin; authentication paths and query parameters are not included in that permission error. The Agent can request access through browser_navigate in a separate tab, return to the original tab, and repeat the intended action. The host chooses resource access:

• resourceAccess: "same-origin" (default) restricts subresources and WebSockets to the Session's admitted Origins.
• resourceAccess: "http" allows HTTP(S) subresources and WS(S) connections used by ordinary websites, including CDNs, API services and embedded frames. Desktop uses this mode and includes these resources in its website approval copy.

Service Workers remain disabled so they cannot bypass routing. These policies do not provide a general network Sandbox or private-address isolation. Popup windows are not exposed by the current Tool API. Desktop opens a visible browser and can leave a login page open while the Agent waits for the user's response; this does not import login state from the user's personal browser.

Dragging uses Playwright 1.55.0 Locator.dragTo between the two accessible targets. It sends pointer input and observes the resulting page; it does not mutate application state or dispatch a fabricated drop result. Existing action timeouts, Origin policy, and Browser Session cancellation also apply.

Every page observation includes focusedElement and focusScope. Chromium reads the current document's native accessibility tree, including date segments and the calendar button when exposed. Its operation-scoped CDP attachment is released after the read, including timeout and late-attachment cleanup. Other engines report the DOM-focused container with focusScope: "dom-element"; that scope cannot establish whether focus moved between native subcontrols. An unchanged container is not evidence of an ineffective key or a focus trap. null means no focused node was observed, not proof that the document has focus. Node Runner presents the scope and focus before the page snapshot within the same observation byte budget. This does not expose selectors or JavaScript evaluation to Agents.

An unnamed control is represented by name: "" with exact: true; use its latest ref when several controls have the same empty name. The caller must use the observed name, without adding page labels to satisfy the SDK.

Tab inventory does not inspect background DOMs. Its title is the last title observed for that document, or null before observation/after main-frame navigation. The selected page still provides its current title and accessible content through the normal observation. A pending background navigation cannot block this inventory. Origin grants and selected-page failures are unchanged.

ExistingChromeConnection is the separate host-owned path for an existing personal Chrome instance. It uses Chrome's browser-level WebSocket directly: connection and inventory send only target-discovery commands. A granted read attaches the exact target ID, reads its native accessibility tree, then detaches that operation's session. It does not initialize other pages, install persistent page scripts, change emulation, or change browser download behavior. Cross-origin navigation revokes the grant even if the tab later returns. Disconnect closes the transport, not the user's browser. Inventory remains human-picker data, not model context.

The granted object implements BrowserSession: the host can return it from its Session factory to bind the same Node Runner browser tools. It supports snapshot, click, hover, fill, selection, key input, drag, same-Origin navigation in the selected tab, and capture with currentPage: true. Capture preserves the user's current viewport and appearance. Operations on one target are serialized; closing the grant revokes it and waits for its operations to settle.

This path does not expose cross-process iframe content, new-tab navigation, tab switching or emulated capture settings. It reads bounded viewport/target facts as needed but offers no arbitrary page evaluation tool to the Agent. The host supplies an explicit debugging endpoint, or an explicitly selected profile that already has Chrome's DevToolsActivePort file; this API neither enables debugging on a running browser nor silently scans personal profiles.

Hover-revealed controls

browser_hover / BrowserAction.kind: "hover" move the native pointer over the observed target without pressing a button. Managed pages use the pinned Playwright locator hover; selected Chrome reuses the existing pointer preparation and sends Input.dispatchMouseEvent(mouseMoved) only to the granted target. Both return a fresh observation, invalidate old refs, and retain the existing Origin, cancellation and Session ownership. Unlike click, hover does not require an enabled control; visibility, stable geometry and hit testing still apply. A hidden action must be revealed by hovering its visible row/title first.