--- name: browser-automation version: "1.0.0" description: Playwright-based browser automation patterns for autonomous web interaction author: LibreFang tags: [browser, automation, playwright, web, scraping] tools: [browser_navigate, browser_click, browser_type, browser_screenshot, browser_read_page, browser_close] runtime: prompt_only --- # Browser Automation Skill ## Playwright CSS Selector Reference ### Basic Selectors | Selector | Description | Example | |----------|-------------|---------| | `#id` | By ID | `#checkout-btn` | | `.class` | By class | `.add-to-cart` | | `tag` | By element | `button`, `input` | | `[attr=val]` | By attribute | `[data-testid="submit"]` | | `tag.class` | Combined | `button.primary` | | `parent child` | Descendant | `div.container button` | | `parent > child` | Direct child | `ul > li` | | `:nth-child(n)` | Nth element | `li:nth-child(2)` | | `:first-child` | First element | `ul > li:first-child` | | `:last-child` | Last element | `ul > li:last-child` | | `[attr*=val]` | Attribute contains | `[class*="price"]` | | `[attr^=val]` | Attribute starts with | `[href^="https"]` | | `[attr$=val]` | Attribute ends with | `[href$=".pdf"]` | | `:not(sel)` | Negation | `button:not(.disabled)` | | `sel1, sel2` | Multiple selectors | `#submit, button[type="submit"]` | ### Form Selectors | Selector | Use Case | |----------|----------| | `input[type="email"]` | Email fields | | `input[type="password"]` | Password fields | | `input[type="search"]` | Search boxes | | `input[name="q"]` | Google/search query | | `textarea` | Multi-line text areas | | `select[name="country"]` | Dropdown menus | | `input[type="checkbox"]` | Checkboxes | | `input[type="radio"]` | Radio buttons | | `button[type="submit"]` | Submit buttons | | `input[type="file"]` | File upload fields | | `input[type="date"]` | Date pickers | | `input[type="tel"]` | Phone number fields | | `input[autocomplete="cc-number"]` | Credit card fields | | `[contenteditable="true"]` | Rich text editors | ### Navigation Selectors | Selector | Use Case | |----------|----------| | `a[href*="cart"]` | Cart links | | `a[href*="checkout"]` | Checkout links | | `a[href*="login"]` | Login links | | `nav a` | Navigation menu links | | `.breadcrumb a` | Breadcrumb links | | `[role="navigation"] a` | ARIA nav links | | `a[href*="account"]` | Account/profile links | | `a[href*="register"], a[href*="signup"]` | Registration links | | `header a[href="/"]` | Logo/home link | | `footer a` | Footer links | ### E-commerce Selectors | Selector | Use Case | |----------|----------| | `.product-price`, `[data-price]` | Product prices | | `.add-to-cart`, `#add-to-cart` | Add to cart buttons | | `.cart-total`, `.order-total` | Cart total | | `.quantity`, `input[name="quantity"]` | Quantity selectors | | `.checkout-btn`, `#checkout` | Checkout buttons | | `[data-product-id]` | Product identifiers | | `.product-title`, `h1.product-name` | Product names | | `.product-image img`, `[data-zoom-image]` | Product images | | `.star-rating`, `[data-rating]` | Review ratings | | `.in-stock`, `.availability` | Stock status | | `select[name="size"], .size-selector` | Size selectors | | `[data-variant], .color-swatch` | Variant selectors | --- ## Generic Selector Strategies (Priority Order) Use selectors that are resilient to UI redesigns. Prefer semantic and accessibility-based selectors over class names. ### Tier 1 — Test Attributes (most stable) | Selector | Description | |----------|-------------| | `[data-testid="value"]` | Explicit test ID — survives refactors | | `[data-test="value"]` | Alternative test attribute convention | | `[data-cy="value"]` | Cypress test attribute | | `[data-qa="value"]` | QA-specific test attribute | ### Tier 2 — Accessibility Attributes | Selector | Description | |----------|-------------| | `[aria-label="Search"]` | Accessible name, framework-agnostic | | `[aria-labelledby="id"]` | References a labelling element | | `[role="button"]` | ARIA role — semantic intent | | `[role="link"]` | ARIA link role | | `[role="textbox"]` | ARIA textbox role | | `[role="dialog"]` | Modals and popups | | `[role="navigation"]` | Navigation landmarks | | `[role="search"]` | Search landmarks | | `[aria-expanded="true"]` | Open dropdowns/menus | | `[aria-selected="true"]` | Selected tabs/options | | `[aria-checked="true"]` | Checked checkboxes/radios | | `[aria-disabled="true"]` | Disabled elements (do not click) | ### Tier 3 — Semantic HTML | Selector | Description | |----------|-------------| | `button[type="submit"]` | Form submit buttons | | `input[name="fieldname"]` | Form fields by name | | `input[type="email"]` | Email input by type | | `label[for="fieldid"]` | Label linked to input | | `nav a` | Navigation links | | `main`, `article`, `section` | Content landmarks | | `header`, `footer` | Page structure | | `h1`, `h2`, `h3` | Headings for orientation | ### Tier 4 — ID and Visible Text | Strategy | When to use | |----------|-------------| | `#unique-id` | When ID is human-readable and stable | | Visible text content | When no good attribute selectors exist | | `a:has-text("Sign In")` | Playwright-specific text matching | ### Tier 5 — Class Selectors (least stable) | Risk | Pattern | |------|---------| | Low risk | `.btn-primary`, `.nav-link` (design-system classes) | | Medium risk | `.header-search-input` (component-specific) | | High risk | `.css-1a2b3c`, `.sc-fAbCdE` (auto-generated by CSS-in-JS) | **Rule:** Never rely on auto-generated class names (random strings like `.css-xyz123`). These change on every build. ## Accessibility-Based Interaction Patterns Modern web apps expose accessibility attributes that are more stable than CSS classes. ### Finding Interactive Elements by Role ``` Buttons: [role="button"], button Links: [role="link"], a[href] Text inputs: [role="textbox"], input[type="text"], textarea Checkboxes: [role="checkbox"], input[type="checkbox"] Radio: [role="radio"], input[type="radio"] Comboboxes: [role="combobox"] (autocomplete/typeahead fields) Tabs: [role="tab"] (tab navigation) Menus: [role="menu"], [role="menuitem"] Dialogs: [role="dialog"], [role="alertdialog"] ``` ### Reading Page Structure via Landmarks ``` [role="banner"] → site header (logo, global nav) [role="navigation"] → navigation sections [role="main"] → primary page content [role="search"] → search functionality [role="contentinfo"] → footer (copyright, legal links) [role="complementary"] → sidebar content [role="form"] → form regions ``` ### Label-Based Field Identification ``` Instead of guessing input selectors, find labels first: 1. browser_read_page → look for label text (e.g., "Email Address") 2. Use: label:has-text("Email") + input (sibling) Or: input[aria-label="Email Address"] Or: # ``` ## SPA Framework Detection & Handling ### Detecting the Framework | Signal | Framework | Notes | |--------|-----------|-------| | `
` or `
` | React / Next.js | Content rendered client-side | | `
` with `data-v-` attributes | Vue.js / Nuxt | `data-v-xxxxx` are scoped style markers | | `` or custom element tags | Angular | Uses web component-like tags | | `
` or compiled class names | Svelte / SvelteKit | Minimal runtime footprint | | URL contains `#/` hash routing | Any SPA | Client-side routing via hash | | `__NEXT_DATA__` script tag | Next.js | Server-side rendering with hydration | | `__NUXT__` or `__NUXT_DATA__` in page | Nuxt.js | Vue SSR framework | ### Framework-Specific Interaction Tips **React apps:** - State updates are batched — wait 500ms-2s after interactions for re-renders - Look for `data-testid` attributes (common in React Testing Library projects) - Portal-rendered content (modals, tooltips) may be at the end of ``, not nested in the component tree - React-Select dropdowns: click the container, then look for `[class*="option"]` in the menu that appears **Vue apps:** - `v-if` elements may not exist in DOM until conditions are met — re-read page after state changes - Vue transitions: wait for CSS transitions to complete before interacting - Vuetify/Element UI components have predictable class prefixes (`.v-btn`, `.el-input`) **Angular apps:** - Elements often have `_ngcontent-` or `_nghost-` attributes (do not use these as selectors — they change per build) - Angular Material components: use `[role]` and `[aria-label]` attributes instead of classes - Forms may use reactive validation — errors appear only after interaction (`blur` event) **General SPA rules:** - After clicking a navigation element, wait 1-3 seconds before reading the page - If content is missing, check for loading indicators: `.loading`, `.spinner`, `[aria-busy="true"]`, `.skeleton` - Retry `browser_read_page` up to 3 times with 2-second intervals before giving up - URL changes without full page reload confirm SPA routing — do not expect `browser_navigate` events ## Site-Specific Selector Patterns These are reference selectors for common sites. They change frequently — always verify with `browser_read_page` if a selector fails, then construct a fresh selector from the live DOM. ### Google Search | Element | Selector | |---------|----------| | Search input | `input[name="q"]`, `textarea[name="q"]` | | Search button | `input[name="btnK"]`, `button[type="submit"]` | | Result titles | `h3` (within `#search`) | | Result links | `#search a[href^="http"]` | | "Next" pagination | `a#pnnext` | ### Amazon | Element | Selector | |---------|----------| | Search input | `#twotabsearchtextbox` | | Search button | `#nav-search-submit-button` | | Add to cart | `#add-to-cart-button` | | Quantity dropdown | `#quantity` | | Cart count | `#nav-cart-count` | ### GitHub | Element | Selector | |---------|----------| | Search | `input[name="q"]` | | Repository name | `[itemprop="name"] a` | | Star button | `button[aria-label*="Star"]` | | Submit button | `button[type="submit"]` | Note: When a saved selector fails, use `browser_read_page` to discover the current DOM, then build a new selector from live content. Prefer `[data-testid]`, `[aria-label]`, or visible text over fragile class-based selectors. --- ## Common Workflows ### Product Search & Purchase ``` 1. browser_navigate → store homepage 2. browser_type → search box with product name 3. browser_click → search button or press Enter 4. browser_read_page → scan results 5. browser_click → desired product 6. browser_read_page → verify product details & price 7. browser_click → "Add to Cart" 8. browser_navigate → cart page 9. browser_read_page → verify cart contents & total 10. STOP → Report to user, wait for approval 11. browser_click → "Proceed to Checkout" (only after approval) ``` ### Account Login ``` 1. browser_navigate → login page 2. browser_read_page → identify form fields and any CAPTCHA 3. browser_type → email/username field 4. browser_type → password field 5. browser_click → login/submit button 6. browser_read_page → verify successful login (check for dashboard/profile elements) 7. If MFA required → inform user, wait for code input ``` ### Form Submission ``` 1. browser_navigate → form page 2. browser_read_page → understand form structure 3. browser_type → fill each field sequentially 4. browser_click → checkboxes/radio buttons as needed 5. browser_screenshot → visual verification before submit 6. browser_click → submit button 7. browser_read_page → verify confirmation ``` ### Price Comparison ``` 1. For each store: a. browser_navigate → store URL b. browser_type → search query c. browser_read_page → extract prices d. memory_store → save price data 2. memory_recall → compare all prices 3. Report findings to user ``` ### Multi-Page Data Extraction ``` 1. browser_navigate → starting page 2. browser_read_page → extract data from current page 3. memory_store → save extracted data 4. Check for pagination: a. browser_click → "Next" button or page number link b. browser_read_page → verify new page loaded (check for changed content) c. Repeat from step 2 5. If no more pages → compile and report results ``` ### Account Registration ``` 1. browser_navigate → registration page 2. browser_read_page → identify required fields 3. browser_type → fill name, email, password fields sequentially 4. browser_click → accept terms checkbox (if required) 5. browser_screenshot → verify all fields before submission 6. browser_click → submit/register button 7. browser_read_page → check for: - Success page → registration complete - Email verification prompt → inform user - Validation errors → read errors, correct fields, retry ``` ### File Download Monitoring ``` 1. browser_navigate → page with download link 2. browser_read_page → identify download button/link 3. browser_click → download trigger 4. browser_read_page → check for download confirmation or redirect 5. If download requires additional steps (accept terms, choose format): a. browser_click → required selections b. browser_click → final download button 6. Report download status to user ``` --- ## Wait Strategies & Timing ### When to Wait Proper waiting prevents most automation failures. Never use fixed sleep times when a condition-based wait is possible. | Scenario | Strategy | Notes | |----------|----------|-------| | Page navigation | Wait for load event | `browser_navigate` handles this automatically | | After clicking link | Read page to confirm new content | Check for expected elements on destination | | AJAX/dynamic content | Re-read page after delay | Some SPAs load content asynchronously | | Form submission | Read page for confirmation | Check for success message or redirect | | Slow networks | Retry with backoff | 3s, 6s, 12s intervals | | Animation/transition | Brief pause before interaction | Modal fade-in, dropdown expansion | ### Detecting Page Load Completion ``` After browser_navigate or browser_click that triggers navigation: 1. browser_read_page → check if expected content is present 2. If content missing → wait 2-3 seconds → browser_read_page again 3. If still missing after 3 retries → page may have changed structure 4. Use browser_screenshot to visually confirm page state ``` ### SPA (Single Page Application) Handling SPAs (React, Angular, Vue, Svelte) do not trigger traditional page loads. Client-side routing means the browser URL changes but no network navigation occurs. ``` 1. browser_click → triggers route change (URL updates but no page reload) 2. browser_read_page → may return stale content from previous view 3. Check for loading indicators in the output: - Text: "Loading...", "Please wait", skeleton placeholders - Attributes: [aria-busy="true"] - Classes: .loading, .spinner, .skeleton, .placeholder 4. If loading detected OR content stale → wait 2 seconds 5. browser_read_page → retry (attempt 2 of 3) 6. If still stale → wait 3 seconds → browser_read_page (attempt 3 of 3) 7. If content never updates: a. browser_screenshot → check if content is visually present but not captured as text b. The content may be inside an iframe or shadow DOM — try alternative access c. Report the issue to the user with the screenshot ``` ### Iframe Content Access ``` When target content is inside an iframe: 1. browser_read_page → look for