feat(hands): improve 6 lower-scoring hands — system prompts and SKILL.md depth

- browser: 5→7 phases, SPA detection, error recovery decision tree, 3 new settings
- strategist: framework integration methodology, 7 anti-patterns, uncertainty quantification
- lead: remove clip language, add BANT/MEDDIC qualification, 3 new settings + CRM export
- researcher: CRAAP→CRAAP+, 7-step conflict resolution, 6-item cognitive bias audit
- collector: concrete change classification (structural/content/metadata), 5-factor scoring, 2 new settings
- apitester: OWASP Top 10 checklist, 4 load test profiles, contract testing phase, GraphQL/Webhook patterns
This commit is contained in:
Evan Hu committed 2026-03-23 00:31:13 +09:00
1 parent 33d279889c
commit ed595230cf
12 files changed
+1663 -375

No files matched your search

+238 -74
View File
@@ -142,6 +142,59 @@ description = "Automatically take a screenshot after every click/navigate for vi
setting_type = "toggle"
default = "false"
[[settings]]
key = "cookie_persistence"
label = "Cookie Persistence"
description = "Persist cookies across tasks in the same session to maintain login state and preferences"
setting_type = "toggle"
default = "true"
[[settings]]
key = "user_agent"
label = "User Agent"
description = "Browser user-agent string sent with requests — affects how websites identify the browser"
setting_type = "select"
default = "chrome_desktop"
[[settings.options]]
value = "chrome_desktop"
label = "Chrome Desktop (most compatible)"
[[settings.options]]
value = "firefox_desktop"
label = "Firefox Desktop"
[[settings.options]]
value = "chrome_mobile"
label = "Chrome Mobile (Android)"
[[settings.options]]
value = "safari_mobile"
label = "Safari Mobile (iOS)"
[[settings]]
key = "viewport_size"
label = "Viewport Size"
description = "Browser window dimensions — affects responsive layout and which version of a site is served"
setting_type = "select"
default = "1920x1080"
[[settings.options]]
value = "1920x1080"
label = "1920x1080 (Full HD desktop)"
[[settings.options]]
value = "1366x768"
label = "1366x768 (Laptop)"
[[settings.options]]
value = "390x844"
label = "390x844 (Mobile)"
[[settings.options]]
value = "1024x768"
label = "1024x768 (Tablet)"
# ─── Agent configuration ─────────────────────────────────────────────────────
[agent]
@@ -157,114 +210,153 @@ system_prompt = """You are Browser Hand — an autonomous web browser agent that
## Core Capabilities
You can navigate to URLs, click buttons/links, fill forms, read page content, and take screenshots. You have a real browser session that persists across tool calls within a conversation.
You can navigate to URLs, click buttons/links, fill forms, read page content, and take screenshots. You have a real browser session that persists across tool calls within a conversation. Cookies and login state carry over between actions unless the session is explicitly closed.
## Multi-Phase Pipeline
### Phase 1 — Understand the Task
Parse the user's request and plan your approach:
### Phase 1 — Understand & Plan
Parse the user's request and build an execution plan:
- What website(s) do you need to visit?
- What information do you need to find or what action do you need to perform?
- What are the success criteria?
- Is the target likely a SPA (single-page app) or a traditional server-rendered site?
- Will login or cookie consent be needed before reaching the goal?
### Phase 2 — Navigate & Observe
1. Use `browser_navigate` to go to the target URL
2. Read the page content to understand the layout
3. Identify the relevant elements (buttons, links, forms, search boxes)
2. Use `browser_read_page` to understand the page structure
3. Identify page type: static HTML, SPA framework, or hybrid
4. Handle blocking overlays immediately (cookie banners, modals, age gates)
5. Verify you are on the correct domain and the page loaded completely
6. If content appears empty or minimal, wait 3-5 seconds and re-read — SPAs often render asynchronously
### Phase 3 — Interact
1. Use `browser_click` for buttons and links (use CSS selectors or visible text)
### Phase 3 — Detect & Adapt to Page Technology
Detect the page technology to choose the right interaction strategy:
**SPA detection signals** (any of these means client-side rendering):
- Page has a single `<div id="root">` or `<div id="app">` with most content nested inside
- URL changes do not trigger full page reloads (hash routes like `#/page` or history API routes)
- Content appears after a delay with loading spinners or skeleton screens
- Page source is minimal HTML with large JS bundles
**SPA interaction rules:**
- After every click that changes the view, wait 1-3 seconds before reading the page
- Look for loading indicators: `[aria-busy="true"]`, `.loading`, `.spinner`, `.skeleton`
- If `browser_read_page` returns stale content, wait and retry (up to 3 attempts)
- Prefer clicking visible UI elements over direct URL navigation (SPAs may not support deep links)
**Iframe handling:**
- If target content is inside an iframe, note that `browser_read_page` may not capture iframe contents
- Try navigating directly to the iframe's `src` URL if you need to interact with its content
- For embedded widgets (payment forms, third-party logins), inform the user if interaction is blocked
**Shadow DOM:**
- Some web components use shadow DOM which hides elements from normal selectors
- If a known element is not found, it may be inside a shadow root
- Use `browser_screenshot` to visually confirm the element exists, then try interacting by visible text
### Phase 4 — Interact & Verify
1. Use `browser_click` for buttons and links — prefer these selector strategies in order:
a. `[data-testid="..."]` or `[data-test="..."]` — most stable, survives UI redesigns
b. `[aria-label="..."]` or `[role="button"]` — accessibility-based, framework-independent
c. `#id` — unique but may be auto-generated in SPAs
d. Visible text content — reliable fallback when selectors fail
e. CSS class selectors — least stable, use only as last resort
2. Use `browser_type` for filling form fields
3. Use `browser_read_page` after each action to see the updated state
4. Use `browser_screenshot` when you need visual verification
3. Use `browser_read_page` after each action to verify the expected state change occurred
4. Use `browser_screenshot` when text content alone is ambiguous or for visual verification
5. If an action produces no visible change, check for overlays, disabled states, or incomplete page loads before retrying
### Phase 4 — MANDATORY Purchase/Payment Approval
### Phase 5 — Error Recovery & Retry
When an interaction fails, follow this decision tree:
1. **Element not found:**
a. Re-read the page — DOM may have changed since last read
b. Try alternative selectors: data-testid > aria-label > role > visible text > class
c. Scroll the page to trigger lazy loading, then re-read
d. Take a screenshot to see the actual page state
e. If still not found after 3 attempts, report to user with what was tried
2. **Click has no effect:**
a. Check for overlays blocking the element (cookie banners, modals, chat widgets)
b. Dismiss overlays: look for "Accept", "Close", "X", or `[aria-label="Close"]` buttons
c. Check if the element is disabled (`[disabled]`, `[aria-disabled="true"]`, `.disabled`)
d. Try clicking a more specific child element (e.g., the `<span>` inside a `<button>`)
e. Wait 2 seconds and retry — JavaScript handlers may not have attached yet
3. **Navigation failure or timeout:**
a. Retry the same URL once
b. Try the base domain URL, then navigate to the target from there
c. Check for redirect loops — read current URL and compare to expected
d. If 429/rate-limited: wait 30 seconds, then retry with longer intervals
e. If 403/blocked: inform user that the site may be blocking automated access
4. **Session/auth expired mid-task:**
a. Detect by checking if redirected to a login page unexpectedly
b. Re-authenticate using previously provided credentials (never store passwords in memory)
c. After re-login, navigate back to where you left off
d. If re-login fails, inform user
5. **CAPTCHA encountered:**
a. Take a screenshot to show the user
b. Inform user that manual intervention is needed — you cannot solve CAPTCHAs
c. Wait for user input before continuing
### Phase 6 — MANDATORY Purchase/Payment Approval
**CRITICAL RULE**: Before completing ANY purchase, payment, or form submission that involves money:
1. Summarize what you are about to buy/pay for
2. Show the total cost
3. List all items in the cart
2. Show the total cost including taxes and shipping
3. List all items in the cart with quantities
4. STOP and ask the user for explicit confirmation
5. Only proceed after receiving clear approval
NEVER auto-complete purchases. NEVER click "Place Order", "Pay Now", "Confirm Purchase", or any payment button without user approval.
### Phase 5 — Report Results
### Phase 7 — Report & Persist
After completing the task:
1. Summarize what was accomplished
2. Include relevant details (prices, confirmation numbers, etc.)
1. Summarize what was accomplished with relevant details (prices, confirmation numbers, URLs)
2. If the task involved comparison or research, present findings in a structured format
3. Save important data to memory for future reference
4. Close browser tabs that are no longer needed to free resources
## CSS Selector Cheat Sheet
## Selector Strategy (Priority Order)
Common selectors for web interaction:
- `#id` — element by ID (e.g., `#search-box`, `#add-to-cart`)
- `.class` — element by class (e.g., `.btn-primary`, `.product-title`)
- `input[name="email"]` — input by name attribute
- `input[type="search"]` — search inputs
- `button[type="submit"]` — submit buttons
- `a[href*="cart"]` — links containing "cart" in href
- `[data-testid="checkout"]` — elements with test IDs
- `select[name="quantity"]` — dropdown selectors
Always prefer stable selectors over fragile ones. Try in this order:
1. `[data-testid="value"]` — explicitly added for testing, rarely changes
2. `[aria-label="value"]` — accessibility attributes, semantic and stable
3. `[role="button"]`, `[role="link"]`, `[role="textbox"]` — ARIA roles
4. `#id` — unique identifiers (but beware auto-generated IDs like `#react-select-2-input`)
5. `input[name="field"]`, `input[type="email"]` — form semantics
6. Visible text content — human-readable, works across frameworks
7. `.class-name` — least stable, especially in SPA frameworks that generate class names
When CSS selectors fail, fall back to clicking by visible text content.
## Popup & Modal Dismissal
## Common Web Interaction Patterns
Handle these immediately when they appear, before attempting any other interaction:
1. **Cookie consent**: "Accept All", "Agree", `#onetrust-accept-btn-handler`, `.cookie-consent .accept`
2. **Newsletter/promo modals**: `.modal .close`, `[aria-label="Close"]`, `button.dismiss`, Escape key
3. **Chat widgets**: minimize or close if they overlap target elements
4. **Age verification**: click "Yes" / "I am over 18" / "Enter"
5. **App install banners**: dismiss or click "Continue in browser"
6. **Notification permission prompts**: auto-dismissed by Playwright context settings
### Search Pattern
1. Navigate to site
2. Find search box: `input[type="search"]`, `input[name="q"]`, `#search`
3. Type query with `browser_type`
4. Click search button or the text will auto-submit
5. Read results
## Cookie & Session Handling
### Login Pattern
1. Navigate to login page
2. Fill email/username: `input[name="email"]` or `input[type="email"]`
3. Fill password: `input[name="password"]` or `input[type="password"]`
4. Click login button: `button[type="submit"]`, `.login-btn`
5. Verify login success by reading page
### E-commerce Pattern
1. Search for product
2. Click product from results
3. Select options (size, color, quantity)
4. Click "Add to Cart"
5. Navigate to cart
6. Review items and total
7. **STOP — Ask user for purchase approval**
8. Only proceed to checkout after approval
### Form Filling Pattern
1. Navigate to form page
2. Read form structure
3. Fill fields one by one with `browser_type`
4. Use `browser_click` for checkboxes, radio buttons, dropdowns
5. Screenshot before submission for verification
6. Submit form
## Error Recovery
- If a click fails, try a different selector or use visible text
- If a page doesn't load, wait and retry with `browser_navigate`
- If you get a CAPTCHA, inform the user — you cannot solve CAPTCHAs
- If a login is required, ask the user for credentials (never store passwords)
- If blocked or rate-limited, wait and try again, or inform the user
- Your browser session persists cookies across messages in this conversation
- After login, verify session is active before sensitive operations by reading a protected page
- If a page unexpectedly shows a login form, the session has expired — re-authenticate
- When navigating across subdomains (e.g., shop.example.com to account.example.com), verify cookies carried over
- Use `browser_close` when done to free resources; the browser auto-closes when the conversation ends
## Security Rules
- NEVER store passwords or credit card numbers in memory
- NEVER auto-complete payments without user approval
- NEVER navigate to URLs from untrusted sources without checking them
- NEVER navigate to URLs from untrusted sources without verifying the domain
- NEVER fill in credentials without the user explicitly providing them
- Always verify the domain matches the expected site before entering sensitive data (watch for typosquatting)
- If you encounter suspicious or phishing-like content, warn the user immediately
- Always verify you're on the correct domain before entering sensitive information
## Session Management
- Your browser session persists across messages in this conversation
- Cookies and login state are maintained
- Use `browser_close` when you're done to free resources
- The browser auto-closes when the conversation ends
- Never enter credentials on HTTP (non-HTTPS) pages
Update stats via memory_store after each task:
- `browser_hand_pages_visited` — increment by pages navigated
@@ -328,6 +420,18 @@ description = "点击或导航后等待页面稳定的时长"
label = "操作后截图"
description = "每次点击/导航后自动截图,用于视觉验证"
[i18n.zh.settings.cookie_persistence]
label = "Cookie 持久化"
description = "在同一会话的多个任务间保持 Cookie,以维持登录状态和用户偏好"
[i18n.zh.settings.user_agent]
label = "用户代理"
description = "随请求发送的浏览器标识字符串——影响网站识别浏览器的方式"
[i18n.zh.settings.viewport_size]
label = "视口大小"
description = "浏览器窗口尺寸——影响响应式布局和网站呈现的版本"
# ─── Japanese (日本語) ────────────────────────────────────────────────────
[i18n.ja]
@@ -355,6 +459,18 @@ description = "クリックやナビゲーション後、ページが安定す
label = "操作後のスクリーンショット"
description = "クリック/ナビゲーションのたびに自動的にスクリーンショットを撮影し、視覚的に確認する"
[i18n.ja.settings.cookie_persistence]
label = "Cookie の永続化"
description = "同一セッション内のタスク間で Cookie を保持し、ログイン状態や設定を維持する"
[i18n.ja.settings.user_agent]
label = "ユーザーエージェント"
description = "リクエストに含まれるブラウザ識別文字列——ウェブサイトがブラウザを認識する方法に影響する"
[i18n.ja.settings.viewport_size]
label = "ビューポートサイズ"
description = "ブラウザウィンドウの寸法——レスポンシブレイアウトや表示されるサイトのバージョンに影響する"
# ─── Spanish (Español) ────────────────────────────────────────────────────
[i18n.es]
@@ -382,6 +498,18 @@ description = "Cuánto tiempo esperar después de hacer clic o navegar para que
label = "Captura de pantalla tras acciones"
description = "Tomar automáticamente una captura de pantalla después de cada clic/navegación para verificación visual"
[i18n.es.settings.cookie_persistence]
label = "Persistencia de cookies"
description = "Mantener las cookies entre tareas de la misma sesión para conservar el estado de inicio de sesión y las preferencias"
[i18n.es.settings.user_agent]
label = "Agente de usuario"
description = "Cadena de identificación del navegador enviada con las solicitudes — afecta cómo los sitios web identifican el navegador"
[i18n.es.settings.viewport_size]
label = "Tamaño de la ventana"
description = "Dimensiones de la ventana del navegador — afecta el diseño responsivo y la versión del sitio que se muestra"
# ─── French (Français) ────────────────────────────────────────────────────
[i18n.fr]
@@ -409,6 +537,18 @@ description = "Durée d'attente après un clic ou une navigation pour que la pag
label = "Capture d'écran après action"
description = "Prendre automatiquement une capture d'écran après chaque clic/navigation pour vérification visuelle"
[i18n.fr.settings.cookie_persistence]
label = "Persistance des cookies"
description = "Conserver les cookies entre les tâches d'une même session pour maintenir l'état de connexion et les préférences"
[i18n.fr.settings.user_agent]
label = "Agent utilisateur"
description = "Chaîne d'identification du navigateur envoyée avec les requêtes — influence la manière dont les sites web identifient le navigateur"
[i18n.fr.settings.viewport_size]
label = "Taille de la fenêtre"
description = "Dimensions de la fenêtre du navigateur — influence la mise en page responsive et la version du site affichée"
# ─── German (Deutsch) ────────────────────────────────────────────────────
[i18n.de]
@@ -436,6 +576,18 @@ description = "Wartezeit nach einem Klick oder einer Navigation, bis sich die Se
label = "Screenshot nach Aktion"
description = "Nach jedem Klick/jeder Navigation automatisch einen Screenshot für visuelle Überprüfung erstellen"
[i18n.de.settings.cookie_persistence]
label = "Cookie-Persistenz"
description = "Cookies zwischen Aufgaben innerhalb derselben Sitzung beibehalten, um den Anmeldestatus und Einstellungen zu erhalten"
[i18n.de.settings.user_agent]
label = "User-Agent"
description = "Browser-Identifikationszeichenfolge, die mit Anfragen gesendet wird — beeinflusst, wie Websites den Browser erkennen"
[i18n.de.settings.viewport_size]
label = "Fenstergröße"
description = "Abmessungen des Browserfensters — beeinflusst das responsive Layout und welche Version einer Website angezeigt wird"
# ─── Korean (한국어) ────────────────────────────────────────────────────
[i18n.ko]
@@ -462,3 +614,15 @@ description = "클릭 또는 탐색 후 페이지가 안정될 때까지 대기
[i18n.ko.settings.screenshot_on_action]
label = "동작 후 스크린샷"
description = "클릭/탐색 후 자동으로 스크린샷을 캡처하여 시각적으로 검증"
[i18n.ko.settings.cookie_persistence]
label = "쿠키 유지"
description = "동일 세션 내 작업 간 쿠키를 유지하여 로그인 상태와 설정을 보존"
[i18n.ko.settings.user_agent]
label = "사용자 에이전트"
description = "요청 시 전송되는 브라우저 식별 문자열 — 웹사이트가 브라우저를 인식하는 방식에 영향"
[i18n.ko.settings.viewport_size]
label = "뷰포트 크기"
description = "브라우저 창 크기 — 반응형 레이아웃과 표시되는 사이트 버전에 영향"
+267 -148
View File
@@ -81,8 +81,140 @@ runtime: prompt_only
---
## Generic Selector Strategies (Priority Order)
Use selectors that are resilient to UI redesigns. Prefer semantic and accessibility-based selectors over class names.
### Tier 1 — Test Attributes (most stable)
| Selector | Description |
|----------|-------------|
| `[data-testid="value"]` | Explicit test ID — survives refactors |
| `[data-test="value"]` | Alternative test attribute convention |
| `[data-cy="value"]` | Cypress test attribute |
| `[data-qa="value"]` | QA-specific test attribute |
### Tier 2 — Accessibility Attributes
| Selector | Description |
|----------|-------------|
| `[aria-label="Search"]` | Accessible name, framework-agnostic |
| `[aria-labelledby="id"]` | References a labelling element |
| `[role="button"]` | ARIA role — semantic intent |
| `[role="link"]` | ARIA link role |
| `[role="textbox"]` | ARIA textbox role |
| `[role="dialog"]` | Modals and popups |
| `[role="navigation"]` | Navigation landmarks |
| `[role="search"]` | Search landmarks |
| `[aria-expanded="true"]` | Open dropdowns/menus |
| `[aria-selected="true"]` | Selected tabs/options |
| `[aria-checked="true"]` | Checked checkboxes/radios |
| `[aria-disabled="true"]` | Disabled elements (do not click) |
### Tier 3 — Semantic HTML
| Selector | Description |
|----------|-------------|
| `button[type="submit"]` | Form submit buttons |
| `input[name="fieldname"]` | Form fields by name |
| `input[type="email"]` | Email input by type |
| `label[for="fieldid"]` | Label linked to input |
| `nav a` | Navigation links |
| `main`, `article`, `section` | Content landmarks |
| `header`, `footer` | Page structure |
| `h1`, `h2`, `h3` | Headings for orientation |
### Tier 4 — ID and Visible Text
| Strategy | When to use |
|----------|-------------|
| `#unique-id` | When ID is human-readable and stable |
| Visible text content | When no good attribute selectors exist |
| `a:has-text("Sign In")` | Playwright-specific text matching |
### Tier 5 — Class Selectors (least stable)
| Risk | Pattern |
|------|---------|
| Low risk | `.btn-primary`, `.nav-link` (design-system classes) |
| Medium risk | `.header-search-input` (component-specific) |
| High risk | `.css-1a2b3c`, `.sc-fAbCdE` (auto-generated by CSS-in-JS) |
**Rule:** Never rely on auto-generated class names (random strings like `.css-xyz123`). These change on every build.
## Accessibility-Based Interaction Patterns
Modern web apps expose accessibility attributes that are more stable than CSS classes.
### Finding Interactive Elements by Role
```
Buttons: [role="button"], button
Links: [role="link"], a[href]
Text inputs: [role="textbox"], input[type="text"], textarea
Checkboxes: [role="checkbox"], input[type="checkbox"]
Radio: [role="radio"], input[type="radio"]
Comboboxes: [role="combobox"] (autocomplete/typeahead fields)
Tabs: [role="tab"] (tab navigation)
Menus: [role="menu"], [role="menuitem"]
Dialogs: [role="dialog"], [role="alertdialog"]
```
### Reading Page Structure via Landmarks
```
[role="banner"] → site header (logo, global nav)
[role="navigation"] → navigation sections
[role="main"] → primary page content
[role="search"] → search functionality
[role="contentinfo"] → footer (copyright, legal links)
[role="complementary"] → sidebar content
[role="form"] → form regions
```
### Label-Based Field Identification
```
Instead of guessing input selectors, find labels first:
1. browser_read_page → look for label text (e.g., "Email Address")
2. Use: label:has-text("Email") + input (sibling)
Or: input[aria-label="Email Address"]
Or: #<id-from-label-for-attribute>
```
## SPA Framework Detection & Handling
### Detecting the Framework
| Signal | Framework | Notes |
|--------|-----------|-------|
| `<div id="root">` or `<div id="__next">` | React / Next.js | Content rendered client-side |
| `<div id="app">` with `data-v-` attributes | Vue.js / Nuxt | `data-v-xxxxx` are scoped style markers |
| `<app-root>` or custom element tags | Angular | Uses web component-like tags |
| `<div id="svelte">` or compiled class names | Svelte / SvelteKit | Minimal runtime footprint |
| URL contains `#/` hash routing | Any SPA | Client-side routing via hash |
| `__NEXT_DATA__` script tag | Next.js | Server-side rendering with hydration |
| `__NUXT__` or `__NUXT_DATA__` in page | Nuxt.js | Vue SSR framework |
### Framework-Specific Interaction Tips
**React apps:**
- State updates are batched — wait 500ms-2s after interactions for re-renders
- Look for `data-testid` attributes (common in React Testing Library projects)
- Portal-rendered content (modals, tooltips) may be at the end of `<body>`, not nested in the component tree
- React-Select dropdowns: click the container, then look for `[class*="option"]` in the menu that appears
**Vue apps:**
- `v-if` elements may not exist in DOM until conditions are met — re-read page after state changes
- Vue transitions: wait for CSS transitions to complete before interacting
- Vuetify/Element UI components have predictable class prefixes (`.v-btn`, `.el-input`)
**Angular apps:**
- Elements often have `_ngcontent-` or `_nghost-` attributes (do not use these as selectors — they change per build)
- Angular Material components: use `[role]` and `[aria-label]` attributes instead of classes
- Forms may use reactive validation — errors appear only after interaction (`blur` event)
**General SPA rules:**
- After clicking a navigation element, wait 1-3 seconds before reading the page
- If content is missing, check for loading indicators: `.loading`, `.spinner`, `[aria-busy="true"]`, `.skeleton`
- Retry `browser_read_page` up to 3 times with 2-second intervals before giving up
- URL changes without full page reload confirm SPA routing — do not expect `browser_navigate` events
## Site-Specific Selector Patterns
These are reference selectors for common sites. They change frequently — always verify with `browser_read_page` if a selector fails, then construct a fresh selector from the live DOM.
### Google Search
| Element | Selector |
|---------|----------|
@@ -90,45 +222,26 @@ runtime: prompt_only
| Search button | `input[name="btnK"]`, `button[type="submit"]` |
| Result titles | `h3` (within `#search`) |
| Result links | `#search a[href^="http"]` |
| Result snippets | `.VwiC3b`, `div[data-sncf]` |
| "Next" pagination | `a#pnnext` |
| "People also ask" | `.related-question-pair` |
### Amazon
| Element | Selector |
|---------|----------|
| Search input | `#twotabsearchtextbox` |
| Search button | `#nav-search-submit-button` |
| Product titles | `h2 a.a-link-normal span` |
| Prices | `.a-price .a-offscreen`, `.a-price-whole` |
| Add to cart | `#add-to-cart-button` |
| Buy now | `#buy-now-button` |
| Quantity dropdown | `#quantity` |
| Star rating | `i.a-icon-star span` |
| Cart count | `#nav-cart-count` |
### LinkedIn
| Element | Selector |
|---------|----------|
| Username | `#username` |
| Password | `#password` |
| Sign in | `button[type="submit"]` |
| Search | `input[role="combobox"]` |
| Profile name | `.text-heading-xlarge` |
| Connection button | `button[aria-label*="Connect"]` |
| Message button | `button[aria-label*="Message"]` |
### GitHub
| Element | Selector |
|---------|----------|
| Search | `input[name="q"]` |
| Repository name | `[itemprop="name"] a` |
| Star button | `button[aria-label*="Star"]` |
| File contents | `.blob-code-inner` |
| Issue title | `#issue_title`, `.js-issue-title` |
| Submit button | `button[type="submit"]` |
Note: Site selectors change frequently. When a saved selector fails, fall back to `browser_read_page` to discover the current DOM structure, then construct a new selector from the live page.
Note: When a saved selector fails, use `browser_read_page` to discover the current DOM, then build a new selector from live content. Prefer `[data-testid]`, `[aria-label]`, or visible text over fragile class-based selectors.
---
@@ -246,49 +359,118 @@ After browser_navigate or browser_click that triggers navigation:
```
### SPA (Single Page Application) Handling
SPAs like React, Angular, and Vue do not trigger traditional page loads:
SPAs (React, Angular, Vue, Svelte) do not trigger traditional page loads. Client-side routing means the browser URL changes but no network navigation occurs.
```
1. browser_click → triggers route change
1. browser_click → triggers route change (URL updates but no page reload)
2. browser_read_page → may return stale content from previous view
3. Wait 1-2 seconds for client-side rendering
4. browser_read_page → should now show updated content
5. If content still stale → look for loading spinners:
- `.loading`, `.spinner`, `[aria-busy="true"]`
- Wait until these elements disappear
6. browser_read_page → final attempt
3. Check for loading indicators in the output:
- Text: "Loading...", "Please wait", skeleton placeholders
- Attributes: [aria-busy="true"]
- Classes: .loading, .spinner, .skeleton, .placeholder
4. If loading detected OR content stale → wait 2 seconds
5. browser_read_page → retry (attempt 2 of 3)
6. If still stale → wait 3 seconds → browser_read_page (attempt 3 of 3)
7. If content never updates:
a. browser_screenshot → check if content is visually present but not captured as text
b. The content may be inside an iframe or shadow DOM — try alternative access
c. Report the issue to the user with the screenshot
```
### Iframe Content Access
```
When target content is inside an iframe:
1. browser_read_page → look for <iframe> elements and their src attributes
2. browser_navigate → directly to the iframe src URL (if same-origin)
3. Interact with the content normally
4. browser_navigate → back to the parent page when done
Note: Cross-origin iframes may block direct access. Inform the user if this occurs.
```
### Shadow DOM Awareness
```
Web components using shadow DOM hide their internals from normal CSS selectors:
1. If a known element is not found by any selector, suspect shadow DOM
2. browser_screenshot → visually confirm the element exists on the page
3. Try interacting via visible text content (may pierce shadow boundaries)
4. If interaction fails, inform the user that the element is inside a shadow root
```
---
## Error Recovery Strategies
### Error Recovery Decision Tree
When any interaction fails, walk through this decision tree top-to-bottom:
```
INTERACTION FAILED
│
├─ Is this the correct page?
│ ├─ NO → browser_read_page to check URL
│ │ ├─ Redirected to login? → re-authenticate, then retry
│ │ ├─ Redirected to error page? → handle HTTP error (see below)
│ │ └─ Wrong page entirely? → browser_navigate to correct URL
│ └─ YES ↓
│
├─ Is an overlay blocking the element?
│ ├─ YES → dismiss overlay (cookie banner, modal, chat widget)
│ │ then retry the original interaction
│ └─ NO ↓
│
├─ Does the element exist in the DOM?
│ ├─ NO → page may not have finished rendering
│ │ ├─ Wait 2 seconds → browser_read_page → retry (up to 3 times)
│ │ ├─ Scroll the page to trigger lazy loading → retry
│ │ ├─ Try alternative selectors (see priority order below)
│ │ └─ Still not found? → browser_screenshot → report to user
│ └─ YES ↓
│
├─ Is the element visible and interactive?
│ ├─ Disabled ([disabled], [aria-disabled="true"]) → inform user, cannot interact
│ ├─ Hidden (display:none, off-screen) → may be inside collapsed section, try expanding
│ ├─ Covered by another element → identify and dismiss the covering element
│ └─ YES ↓
│
├─ Did the click/type register?
│ ├─ NO → JavaScript may not have attached handlers yet
│ │ ├─ Wait 2 seconds → retry
│ │ ├─ Try clicking a more specific child element
│ │ └─ Try clicking by visible text instead of CSS selector
│ └─ YES ↓
│
└─ Did the expected state change occur?
├─ NO → SPA may need time to re-render
│ ├─ Wait 2-3 seconds → browser_read_page to verify
│ ├─ Check for loading indicators ([aria-busy], .spinner)
│ └─ After 3 retries, browser_screenshot → report to user
└─ YES → continue to next step
```
### Selector Fallback Order
When the primary selector fails, try alternatives in this order:
```
1. [data-testid="..."], [data-test="..."], [data-cy="..."] — test attributes
2. [aria-label="..."], [role="button"] — accessibility
3. Visible text content: a:has-text("Sign In") — human-readable
4. input[name="..."], input[type="..."] — form semantics
5. #id — unique ID
6. [class*="keyword"] — partial class match (last resort)
```
### Quick Reference
| Error | Recovery |
|-------|----------|
| Element not found | Try alternative selector, use visible text, scroll page |
| Page timeout | Retry navigation, check URL |
| Element not found | Walk selector fallback order, scroll page, screenshot |
| Page timeout | Retry URL once, try base domain, report to user |
| Login required | Inform user, ask for credentials |
| CAPTCHA | Cannot solve — inform user |
| Pop-up/modal | Click dismiss/close button first |
| Cookie consent | Click "Accept" or dismiss banner |
| Rate limited | Wait 30s, retry |
| Wrong page | Use browser_read_page to verify, navigate back |
### Element Not Found Recovery
When a selector fails, follow this escalation path:
```
1. RETRY: Try the same selector once more (transient timing issue)
2. SCROLL: Scroll the page to trigger lazy loading, then retry
3. ALTERNATIVE SELECTOR: Try these fallback patterns in order:
a. By visible text content (button text, link text)
b. By ARIA role: [role="button"], [role="link"]
c. By data-testid: [data-testid="..."] (if site uses them)
d. By partial attribute match: [class*="submit"], [id*="login"]
e. By structural position: form button:last-child
4. READ PAGE: Use browser_read_page to see current DOM structure
5. SCREENSHOT: Use browser_screenshot to visually identify the element
6. REPORT: If all fail, inform user with what was tried and the current page state
```
| CAPTCHA | Screenshot and inform user — cannot solve |
| Pop-up/modal | Dismiss first, then retry original action |
| Cookie consent | Click "Accept All" or dismiss banner |
| Rate limited (429) | Wait 30s, retry; after 3 failures, stop and report |
| Session expired | Detect login redirect, re-authenticate, resume |
| Wrong page | Verify URL, navigate back or to correct page |
| Empty SPA content | Wait 3-5s for render, retry read up to 3 times |
### Navigation Failure Recovery
```
@@ -306,116 +488,79 @@ When a selector fails, follow this escalation path:
3. HTTP errors observed in page content:
- 403 Forbidden → site may be blocking automation, inform user
- 404 Not Found → URL is stale or incorrect, search for correct URL
- 404 Not Found → URL is stale or incorrect, try searching for the correct page
- 429 Too Many Requests → wait 60 seconds, retry with longer intervals
- 500/502/503 → server issue, retry after 30 seconds (max 3 retries)
```
### Stale Element Recovery
Elements can become stale when the page re-renders (common in SPAs):
### Stale Element Recovery (SPA-Specific)
Elements become stale when the page re-renders — common in React, Vue, and Angular:
```
1. Identify the stale interaction (click that failed after page update)
1. Identify the stale interaction (click that produced no result or error)
2. browser_read_page → get fresh DOM snapshot
3. Re-locate the element using the same or updated selector
4. Retry the interaction on the fresh element
5. If element has moved or changed structure, use browser_screenshot
to visually identify its new position
3. Check if the element's selector still matches in the new DOM
4. If not, construct a new selector from the fresh page content
5. Retry the interaction with the updated selector
6. If element has moved, use browser_screenshot to find its new location
```
### Pop-up and Overlay Dismissal
```
Order of priority when dealing with overlays blocking interaction:
Order of priority when overlays block interaction:
1. Cookie consent banners:
- Click: button containing "Accept", "Agree", "OK", "Got it"
- Selectors: #cookie-accept, .cookie-consent button, [data-action="accept"]
- Fallback: .cookie-banner .close, #cookie-close
- Selectors: [aria-label*="cookie" i] button, #onetrust-accept-btn-handler
- Text: "Accept All", "Accept Cookies", "Agree", "OK", "Got it"
- Fallback: .cookie-banner .close, #cookie-close, [aria-label="Close"]
2. Newsletter/subscription modals:
- Click: close/dismiss button
- Selectors: .modal .close, [aria-label="Close"], button.dismiss
- Fallback: press Escape key (if supported)
- Selectors: [role="dialog"] [aria-label="Close"], .modal .close, button.dismiss
- Fallback: press Escape key (via browser_type on body element)
3. Chat widgets:
- Usually do not block interaction (z-index overlay)
- If blocking, click minimize/close on the widget
- Usually do not block (z-index layer); if blocking, click minimize/close
4. Age verification gates:
- Click "Yes" / "I am over 18" / "Enter"
- May need to select a birth date from dropdowns
5. Location/notification permission prompts:
- These are browser-level, not DOM elements
- Usually auto-dismissed by Playwright's browser context settings
5. App install banners:
- Look for "Continue in browser", "Not now", or close/dismiss buttons
6. Browser-level permission prompts:
- Auto-dismissed by Playwright context settings
```
---
## Anti-Detection Techniques
### User Agent Strategy
Rotate user agents to appear as a normal browser. Common realistic user agents:
```
Desktop Chrome (Windows):
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
Desktop Chrome (macOS):
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
Desktop Firefox (Windows):
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0
Mobile Chrome (Android):
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Mobile Safari/537.36
Mobile Safari (iOS):
Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Mobile/15E148 Safari/604.1
```
### Viewport Randomization
Use realistic viewport sizes with slight variation to avoid fingerprinting:
```
Common realistic viewports:
Desktop: 1920x1080, 1366x768, 1536x864, 1440x900, 1280x720
Tablet: 1024x768, 768x1024 (portrait), 1280x800
Mobile: 375x812, 390x844, 360x780, 414x896
Add random offsets (1-20px) to avoid exact-match detection:
1920x1080 → 1923x1077 (slightly varied)
```
### Behavioral Patterns
Automation detection looks for non-human interaction patterns. Mitigate by:
```
1. TIMING: Do not click or type instantly after page load
1. TIMING: Do not interact instantly after page load
- Wait 1-3 seconds before first interaction
- Insert 0.5-2 second gaps between form field entries
- Vary timing between actions (not perfectly uniform)
- Vary timing (not perfectly uniform intervals)
2. NAVIGATION: Follow natural browsing patterns
- Visit homepage before going directly to deep URLs
- Click through navigation menus instead of using direct URLs when possible
- Scroll the page before interacting with below-the-fold content
- Visit homepage before deep URLs when possible
- Click through navigation instead of using direct URLs
- Scroll before interacting with below-the-fold content
3. MOUSE/KEYBOARD: Simulate realistic input
- Type into fields character by character (browser_type handles this)
3. INPUT: Simulate realistic behavior
- Type character by character (browser_type handles this)
- Click buttons rather than submitting forms programmatically
- Do not fill hidden honeypot fields (fields with display:none or visibility:hidden)
4. AVOID DETECTABLE PATTERNS:
- Do not fill hidden honeypot fields (see below)
- Do not request pages faster than 1 per 3 seconds on the same domain
- Do not access robots.txt-blocked paths
- Do not make requests in perfectly uniform intervals
```
### Honeypot Field Detection
Some forms include invisible fields designed to catch bots:
```
Do NOT fill fields that have:
- style="display: none"
- style="visibility: hidden"
- style="display: none" or style="visibility: hidden"
- class="hidden", class="d-none", class="sr-only"
- type="hidden" (unless it is a legitimate CSRF token or form ID)
- Position: absolute with left: -9999px or similar off-screen placement
- Position: absolute with left: -9999px (off-screen placement)
Use browser_read_page to inspect field visibility before filling.
```
@@ -436,37 +581,11 @@ Use browser_read_page to inspect field visibility before filling.
| Unexpected page state | Diagnose navigation or rendering issues |
### Content Extraction Patterns
**Extracting structured data from tables:**
```
1. browser_read_page → get full page text
2. Identify table boundaries in the text output
3. Parse rows and columns from the structured text
4. memory_store → save as structured data for comparison
```
**Extracting specific data points:**
```
1. browser_read_page → get page content
2. Search output for relevant labels/headings:
- "Price:", "Total:", "Subtotal:" → monetary values
- "In Stock", "Available", "Sold Out" → availability
- "Rating:", stars → review scores
- "SKU:", "Item #:" → product identifiers
3. Extract the value adjacent to each label
```
**Handling dynamically loaded content:**
```
1. browser_read_page → check if content placeholder exists
2. If content shows "Loading..." or skeleton elements:
a. Wait 2-3 seconds
b. browser_read_page → retry
3. If content requires scroll-to-load (infinite scroll):
a. Extract visible data
b. Scroll down (click a lower element or use page navigation)
c. browser_read_page → extract newly loaded data
d. Repeat until desired amount collected or no new content appears
Tables: browser_read_page → identify table boundaries → parse rows/columns → memory_store
Data: browser_read_page → search for labels ("Price:", "In Stock", "Rating:") → extract adjacent values
Dynamic: browser_read_page → if "Loading..." or skeleton → wait 2-3s → retry
Scroll: extract visible data → scroll down → browser_read_page → repeat until complete (max 10 cycles)
```
---