web-autopilot（网络自动驾驶仪）

Web Autopilot

Record once in any web app, let AI handle it from now on.

Overview

Record → Analyze → Confirm Fields → Generate → Test → Register as Tool

🎬 Record         User performs the workflow once in a real browser (after login)
🔍 Analyze        AI analyzes network traffic, classifies fixed/dynamic/session fields
✅ Confirm Fields  [Required for submit tasks] User confirms field classifications
📝 Generate       Generates reusable TS script + field mapping
🧪 Test           Iterative test loop, up to 5 rounds of auto-fix
🔧 Register       Register as an OpenClaw tool for direct invocation

Task Types

📊 Query / Export

Data extraction and report generation. Scripts run and output results automatically — no manual intervention needed. Examples: pull sales reports, extract project data, export revenue details

📝 Submit

Submit forms such as expense reports, travel requests, payment requests, etc. Each run requires dynamic parameters. Examples: submit travel request, submit expense report, submit payment request

The key challenge for submit tasks: correctly distinguishing which fields are fixed vs. which change every time, and confirming with the user before generating the script.

Skill Directory

~/.openclaw/rpa/
├── recordings/<task-name>/recording.json
├── tasks/<task-name>/
│   ├── task-meta.json
│   ├── run.ts
│   └── field-mapping.json
└── sessions/<domain>.session.json

Skill scripts: /opt/homebrew/lib/node_modules/openclaw/skills/web-autopilot/scripts/

Commands

1. record — Record a workflow

Ask user: task name, login URL or app URL.

cd /opt/homebrew/lib/node_modules/openclaw/skills/web-autopilot

# Option A: Start from login page (SSO, OAuth, username/password, etc.)
npx ts-node scripts/record.ts --name "my-task" --sso-url "https://login.example.com"

# Option B: Start directly from app (if already logged in or no login needed)
npx ts-node scripts/record.ts --name "my-task" --app-url "https://app.example.com"

Run in PTY mode (pty: true, background: true). User operates browser, types "done" when finished.

Note: --sso-url is a legacy parameter name; it works for any login URL (SSO, OAuth, plain login page, etc.).

2. analyze — Analyze the recording (AI does this)

Read recording.json, separate login traffic from business traffic, identify core APIs.

Key steps:

1. Read ~/.openclaw/rpa/recordings/<task>/summary.txt for overview
2. Parse recording.json to extract all API calls to app domain
3. For each POST/PUT/PATCH with meaningful body:

- Classify fields: FIXED / DYNAMIC / SESSION / RELATIONAL - Detect protocol: rest-json / graphql / form-urlencoded / multipart

1. Map the complete API sequence (prerequisites → main operation → follow-ups)
2. Analyze ALL response fields and create field-mapping.json with human-readable labels
3. Create task-meta.json
4. [Submit tasks] After analysis, present the field classification confirmation table to the user (see below)

Field Classification

Field Analysis Rules (MANDATORY)

Every field must have a human-readable label. Including system-generated field names.

Inference priority:

1. Data value type: timestamp (10^12-13) / monetary amount (contextual) / enum (fixed values) / URL / JSON object
2. Field name pattern: *time/*date/*_at → datetime | *amount/*price/*cost → monetary | *id/*_key → ID | *status/*state → status
3. Business context: infer from related fields, API endpoint names
4. If uncertain → annotate as (unknown meaning: sample value)

Field Confirmation Step (MANDATORY for Submit tasks)

After analysis, you must present the following confirmation table to the user and wait for confirmation before generating the script:

📋 Field Classification Confirmation — <task name>

✅ FIXED (hardcoded):
  - approvalFlowId: "xxx"  → Approval Flow ID
  - companyId: "yyy"       → Company Entity
  - currency: "CNY"        → Currency

🔄 DYNAMIC (passed as parameters each run):
  - amount          → Amount (example: --amount 1500)
  - startDate       → Start Date (example: --startDate 2026-03-10)
  - endDate         → End Date (example: --endDate 2026-03-12)
  - destination     → Destination (example: --destination "New York")
  - reason          → Reason (example: --reason "Client visit")
  - attachments     → Attachment path (example: --attachments ~/Desktop/receipt.jpg)

🔗 RELATIONAL (auto-queried):
  - projectId       → Project ID (auto-looked up by project name, --projectName "Project X")

❓ Needs confirmation (AI uncertain):
  - field_abc123    → Unknown meaning (recorded value: "0"), suggest: FIXED("0") or DYNAMIC?

Please confirm the above classification or indicate any fields that need adjustment.

Only proceed to the Generate step after user confirmation.

CSV Export Rules (MANDATORY)

• Keep ALL fields, including hidden fields, dynamic fields, system fields — never crop
• Field order: preserve original order from data, never sort (sorting causes column misalignment)
• JSON/object fields → convert to JSON string for storage
• Use csv.writer + proper quoting to handle JSON fields containing commas

3. generate — Generate the task script

Pre-generation checklist (Query/Export tasks):

• ✅ All fields are in field-mapping.json
• ✅ All fields have human-readable labels
• ✅ CSV export uses field-mapping.json for column headers
• ✅ Field order preserves original order

Pre-generation checklist (Submit tasks):

• ✅ User has confirmed field classification (FIXED / DYNAMIC / RELATIONAL)
• ✅ All DYNAMIC fields converted to CLI parameters (with type, example value, required/optional)
• ✅ RELATIONAL fields have auto-query logic or corresponding parameters
• ✅ Script has --dry-run mode (prints request body without submitting, for testing)
• ✅ Script outputs submission result (success/failure + document number/link)

Submit task invocation example (written to task-meta.json usage field after generation):

# Preview (no actual submission)
npx ts-node run.ts --dry-run --amount 1500 --startDate 2026-03-10 ...

# Submit for real
npx ts-node run.ts --amount 1500 --startDate 2026-03-10 --destination "New York" --reason "Client visit"

4. test — Iterative test loop (max 5 rounds)

Run script → check output → if error: diagnose → fix → repeat.

5. run — Execute a registered task

npx ts-node ~/.openclaw/rpa/tasks/<task>/run.ts --param1 value1

6. list — List all tasks

npx ts-node /opt/homebrew/lib/node_modules/openclaw/skills/web-autopilot/scripts/run-task.ts --list

Session & Credential Management

Session (Cookie/Token Storage)

Sessions are cookie-based and work with any login method:

• SSO (OIDC, SAML, CAS, etc.)
• OAuth / OAuth2
• Username + password forms
• Any browser-based authentication

Session files: ~/.openclaw/rpa/sessions/<domain>.session.json

Credentials (Encrypted Storage)

Login credentials are stored encrypted (AES-256-GCM) in a separate file — never stored in plaintext.

File: ~/.openclaw/rpa/credentials.enc

• Encryption key = machine identity (hostname+username) + optional RPA_CREDENTIAL_KEY env var
• File permissions: 0600 (owner only)
• Supports automatic extraction and encrypted storage from recording.json

# Manage credentials
npx ts-node scripts/utils/credentials.ts list                    # List saved domains + usernames
npx ts-node scripts/utils/credentials.ts save <domain> <user> <pass>  # Save manually
npx ts-node scripts/utils/credentials.ts delete <domain>         # Delete
npx ts-node scripts/utils/credentials.ts extract <recording.json> # Extract from recording

Auto-Login Flow

When a session expires, the auto-login flow kicks in:

1. Read encrypted credentials for the target domain from credentials.enc
2. Select login strategy based on loginFlow.type
3. Launch browser (headless if credentials exist, headed if not)
4. Execute login steps → follow redirects → reach target app
5. Capture cookies/tokens → save new session
6. If all else fails → open headed browser for manual login (fallback)

Login Flow Types

When generating scripts, you must identify the login type from the recording and write it to the loginFlow field in task-meta.json:

loginFlow Schema (task-meta.json)

{
  "loginFlow": {
    "type": "api",              // api | form | multi-step | manual-only
    "loginUrl": "https://sso.example.com",
    "loginDomain": "sso.example.com",
    "appDomain": "app.example.com",

    // ── type=api specific fields ──
    "loginApiPath": "/api/sso/login",
    "authType": "passwordAuth",       // Optional, auth type field in API body
    "appId": "1234567890",            // Optional, SSO portal app ID (for forward redirect)
    "appForwardUrl": "...",           // Optional, direct redirect URL (alternative to appId)

    // ── type=form specific fields ──
    "usernameSelector": "input[name='email']",    // Optional, custom selectors
    "passwordSelector": "input[type='password']",
    "submitSelector": "button[type='submit']",

    // ── type=multi-step specific fields ──
    "steps": [
      { "action": "fill", "selector": "input[type=email]", "field": "username" },
      { "action": "click", "selector": "#identifierNext" },
      { "action": "wait", "selector": "input[type=password]", "timeoutMs": 5000 },
      { "action": "fill", "selector": "input[type=password]", "field": "password" },
      { "action": "click", "selector": "#passwordNext" }
    ],

    // ── Common fields ──
    "successIndicator": "url_contains:app.example.com",  // Condition to detect successful login
    "postLoginWaitMs": 3000          // Wait time after login success (for cookies to settle)
  }
}

Login Identification Guide for Analyze Step (MANDATORY)

During the analyze step, you must complete the following login analysis:

1. Extract credentials → credentials.ts extract <recording.json> (auto-detects username/password in POST body)
2. Identify login type → Inspect the login flow in the recording:

- Has a clear POST login/auth API → type = api - Has form fill actions (password type input) on the same page → type = form - Has multiple form fill actions with page navigations in between → type = multi-step - Has CAPTCHA image requests or reCAPTCHA scripts → type = manual-only

1. Document the SSO → app redirect path:

- Does it use an appId forward? - Does it use a redirect_uri callback? - Where is the token — in URL query / response body / cookie?

1. Write loginFlow → Write all fields to task-meta.json
2. Sanitize → Replace passwords in recording.json with [REDACTED]

⚠️ If credentials.ts extract cannot extract credentials (e.g., Google multi-step login), prompt the user to save credentials manually:

npx ts-node scripts/utils/credentials.ts save accounts.google.com user@gmail.com 'password'

Login Code Templates for Script Generation

Choose the auto-login implementation based on loginFlow.type:

type=api (REST API login):

// API login → follow redirects → navigate to app
const resp = await page.evaluate(async (p) => {
  const r = await fetch(p.url, { method: 'POST', headers: {'Content-Type':'application/json'},
    body: JSON.stringify(p.body), credentials: 'include' });
  return { status: r.status, ok: r.ok };
}, { url: loginApiUrl, body: { authType, credential: { username, password } } });

type=form:

await page.fill(loginFlow.usernameSelector || 'input[name="username"]', cred.username);
await page.fill(loginFlow.passwordSelector || 'input[type="password"]', cred.password);
await page.click(loginFlow.submitSelector || 'button[type="submit"]');

type=multi-step:

for (const step of loginFlow.steps) {
  if (step.action === 'fill') {
    const value = step.field === 'username' ? cred.username : cred.password;
    await page.fill(step.selector, value);
  } else if (step.action === 'click') {
    await page.click(step.selector);
  } else if (step.action === 'wait') {
    await page.waitForSelector(step.selector, { timeout: step.timeoutMs || 10000 });
  }
}

type=manual-only:

// Open headed browser, wait for user to complete login manually
const browser = await pw.chromium.launch({ headless: false });
// ... wait for successIndicator

task-meta.json loginFlow example (SSO → enterprise app):

{
  "loginFlow": {
    "type": "api",
    "loginUrl": "https://sso.example.com",
    "loginDomain": "sso.example.com",
    "loginApiPath": "/api/sso/login",
    "authType": "passwordAuth",
    "appId": "1234567890",
    "appDomain": "app.example.com",
    "successIndicator": "url_contains:app.example.com"
  }
}

⚠️ Security Rules (MANDATORY)

1. Passwords in recording.json must be sanitized immediately after analysis (replace with [REDACTED])
2. credentials.enc is an encrypted binary file — do not attempt to read or edit directly
3. credentials.enc and sessions/ directory must never be committed to version control or shared
4. Skill packages (.skill) must not contain any credentials, sessions, or recording data
5. Generated task scripts (run.ts) must never hardcode any passwords

Known Issues & Lessons Learned

🔐 Login flow must match app — don't assume one-size-fits-all

• Current implemented scripts use type=api mode (enterprise SSO → business app)
• Each new app recording must re-identify the login type — do not reuse login logic from old scripts
• Google/Microsoft multi-step logins require type=multi-step + steps sequence
• Sites with CAPTCHA/2FA can only use type=manual-only
• Inlining login logic into run.ts (rather than importing external login.ts) is more stable due to Node ESM/CJS compatibility issues

⚠️ Node v25 ESM compatibility

• Node v25 defaults to ESM, require() is unavailable
• Solution: place tsconfig.json in the task directory to force "module": "commonjs"
• Dependencies like Playwright need full-path require: require('/opt/.../node_modules/playwright')
• Cross-directory .ts imports under ts-node are unstable — recommend inlining critical logic into run.ts

⚠️ Multi-tab traffic capture (fixed)

Some login flows or apps open new tabs. Recorder uses context.on('request/response') to capture ALL tabs.

📋 CSV must include ALL fields with human-readable labels

• Never crop fields — include everything from the API response
• System-generated field names (e.g. field_*, attr_*, custom_*) must be analyzed from sample data
• Create field-mapping.json for every task
• Field order: preserve original order from data, never sort
• Use proper CSV quoting to handle JSON fields with commas

📝 Submit tasks: always confirm field classification before generating

• Never skip the field confirmation step — wrong FIXED/DYNAMIC split breaks every future submission
• Fields that look fixed (e.g. a hardcoded project ID) might actually need to be dynamic in real use
• Always include --dry-run in generated scripts so users can verify the request body before committing
• RELATIONAL fields (e.g. approver ID looked up by name) should be auto-resolved in script, exposed as human-readable params

File Locations