diff --git a/hands/browser/SKILL.md b/hands/browser/SKILL.md index b8d8d6b..df8170a 100644 --- a/hands/browser/SKILL.md +++ b/hands/browser/SKILL.md @@ -20,6 +20,16 @@ runtime: prompt_only | `tag` | By element | `button`, `input` | | `[attr=val]` | By attribute | `[data-testid="submit"]` | | `tag.class` | Combined | `button.primary` | +| `parent child` | Descendant | `div.container button` | +| `parent > child` | Direct child | `ul > li` | +| `:nth-child(n)` | Nth element | `li:nth-child(2)` | +| `:first-child` | First element | `ul > li:first-child` | +| `:last-child` | Last element | `ul > li:last-child` | +| `[attr*=val]` | Attribute contains | `[class*="price"]` | +| `[attr^=val]` | Attribute starts with | `[href^="https"]` | +| `[attr$=val]` | Attribute ends with | `[href$=".pdf"]` | +| `:not(sel)` | Negation | `button:not(.disabled)` | +| `sel1, sel2` | Multiple selectors | `#submit, button[type="submit"]` | ### Form Selectors | Selector | Use Case | @@ -33,6 +43,11 @@ runtime: prompt_only | `input[type="checkbox"]` | Checkboxes | | `input[type="radio"]` | Radio buttons | | `button[type="submit"]` | Submit buttons | +| `input[type="file"]` | File upload fields | +| `input[type="date"]` | Date pickers | +| `input[type="tel"]` | Phone number fields | +| `input[autocomplete="cc-number"]` | Credit card fields | +| `[contenteditable="true"]` | Rich text editors | ### Navigation Selectors | Selector | Use Case | @@ -43,6 +58,10 @@ runtime: prompt_only | `nav a` | Navigation menu links | | `.breadcrumb a` | Breadcrumb links | | `[role="navigation"] a` | ARIA nav links | +| `a[href*="account"]` | Account/profile links | +| `a[href*="register"], a[href*="signup"]` | Registration links | +| `header a[href="/"]` | Logo/home link | +| `footer a` | Footer links | ### E-commerce Selectors | Selector | Use Case | @@ -52,6 +71,66 @@ runtime: prompt_only | `.cart-total`, `.order-total` | Cart total | | `.quantity`, `input[name="quantity"]` | Quantity selectors | | `.checkout-btn`, `#checkout` | Checkout buttons | +| `[data-product-id]` | Product identifiers | +| `.product-title`, `h1.product-name` | Product names | +| `.product-image img`, `[data-zoom-image]` | Product images | +| `.star-rating`, `[data-rating]` | Review ratings | +| `.in-stock`, `.availability` | Stock status | +| `select[name="size"], .size-selector` | Size selectors | +| `[data-variant], .color-swatch` | Variant selectors | + +--- + +## Site-Specific Selector Patterns + +### Google Search +| Element | Selector | +|---------|----------| +| Search input | `input[name="q"]`, `textarea[name="q"]` | +| Search button | `input[name="btnK"]`, `button[type="submit"]` | +| Result titles | `h3` (within `#search`) | +| Result links | `#search a[href^="http"]` | +| Result snippets | `.VwiC3b`, `div[data-sncf]` | +| "Next" pagination | `a#pnnext` | +| "People also ask" | `.related-question-pair` | + +### Amazon +| Element | Selector | +|---------|----------| +| Search input | `#twotabsearchtextbox` | +| Search button | `#nav-search-submit-button` | +| Product titles | `h2 a.a-link-normal span` | +| Prices | `.a-price .a-offscreen`, `.a-price-whole` | +| Add to cart | `#add-to-cart-button` | +| Buy now | `#buy-now-button` | +| Quantity dropdown | `#quantity` | +| Star rating | `i.a-icon-star span` | +| Cart count | `#nav-cart-count` | + +### LinkedIn +| Element | Selector | +|---------|----------| +| Username | `#username` | +| Password | `#password` | +| Sign in | `button[type="submit"]` | +| Search | `input[role="combobox"]` | +| Profile name | `.text-heading-xlarge` | +| Connection button | `button[aria-label*="Connect"]` | +| Message button | `button[aria-label*="Message"]` | + +### GitHub +| Element | Selector | +|---------|----------| +| Search | `input[name="q"]` | +| Repository name | `[itemprop="name"] a` | +| Star button | `button[aria-label*="Star"]` | +| File contents | `.blob-code-inner` | +| Issue title | `#issue_title`, `.js-issue-title` | +| Submit button | `button[type="submit"]` | + +Note: Site selectors change frequently. When a saved selector fails, fall back to `browser_read_page` to discover the current DOM structure, then construct a new selector from the live page. + +--- ## Common Workflows @@ -73,10 +152,12 @@ runtime: prompt_only ### Account Login ``` 1. browser_navigate → login page -2. browser_type → email/username field -3. browser_type → password field -4. browser_click → login/submit button -5. browser_read_page → verify successful login +2. browser_read_page → identify form fields and any CAPTCHA +3. browser_type → email/username field +4. browser_type → password field +5. browser_click → login/submit button +6. browser_read_page → verify successful login (check for dashboard/profile elements) +7. If MFA required → inform user, wait for code input ``` ### Form Submission @@ -101,8 +182,87 @@ runtime: prompt_only 3. Report findings to user ``` +### Multi-Page Data Extraction +``` +1. browser_navigate → starting page +2. browser_read_page → extract data from current page +3. memory_store → save extracted data +4. Check for pagination: + a. browser_click → "Next" button or page number link + b. browser_read_page → verify new page loaded (check for changed content) + c. Repeat from step 2 +5. If no more pages → compile and report results +``` + +### Account Registration +``` +1. browser_navigate → registration page +2. browser_read_page → identify required fields +3. browser_type → fill name, email, password fields sequentially +4. browser_click → accept terms checkbox (if required) +5. browser_screenshot → verify all fields before submission +6. browser_click → submit/register button +7. browser_read_page → check for: + - Success page → registration complete + - Email verification prompt → inform user + - Validation errors → read errors, correct fields, retry +``` + +### File Download Monitoring +``` +1. browser_navigate → page with download link +2. browser_read_page → identify download button/link +3. browser_click → download trigger +4. browser_read_page → check for download confirmation or redirect +5. If download requires additional steps (accept terms, choose format): + a. browser_click → required selections + b. browser_click → final download button +6. Report download status to user +``` + +--- + +## Wait Strategies & Timing + +### When to Wait +Proper waiting prevents most automation failures. Never use fixed sleep times when a condition-based wait is possible. + +| Scenario | Strategy | Notes | +|----------|----------|-------| +| Page navigation | Wait for load event | `browser_navigate` handles this automatically | +| After clicking link | Read page to confirm new content | Check for expected elements on destination | +| AJAX/dynamic content | Re-read page after delay | Some SPAs load content asynchronously | +| Form submission | Read page for confirmation | Check for success message or redirect | +| Slow networks | Retry with backoff | 3s, 6s, 12s intervals | +| Animation/transition | Brief pause before interaction | Modal fade-in, dropdown expansion | + +### Detecting Page Load Completion +``` +After browser_navigate or browser_click that triggers navigation: +1. browser_read_page → check if expected content is present +2. If content missing → wait 2-3 seconds → browser_read_page again +3. If still missing after 3 retries → page may have changed structure +4. Use browser_screenshot to visually confirm page state +``` + +### SPA (Single Page Application) Handling +SPAs like React, Angular, and Vue do not trigger traditional page loads: +``` +1. browser_click → triggers route change +2. browser_read_page → may return stale content from previous view +3. Wait 1-2 seconds for client-side rendering +4. browser_read_page → should now show updated content +5. If content still stale → look for loading spinners: + - `.loading`, `.spinner`, `[aria-busy="true"]` + - Wait until these elements disappear +6. browser_read_page → final attempt +``` + +--- + ## Error Recovery Strategies +### Quick Reference | Error | Recovery | |-------|----------| | Element not found | Try alternative selector, use visible text, scroll page | @@ -114,11 +274,543 @@ runtime: prompt_only | Rate limited | Wait 30s, retry | | Wrong page | Use browser_read_page to verify, navigate back | +### Element Not Found Recovery +When a selector fails, follow this escalation path: +``` +1. RETRY: Try the same selector once more (transient timing issue) +2. SCROLL: Scroll the page to trigger lazy loading, then retry +3. ALTERNATIVE SELECTOR: Try these fallback patterns in order: + a. By visible text content (button text, link text) + b. By ARIA role: [role="button"], [role="link"] + c. By data-testid: [data-testid="..."] (if site uses them) + d. By partial attribute match: [class*="submit"], [id*="login"] + e. By structural position: form button:last-child +4. READ PAGE: Use browser_read_page to see current DOM structure +5. SCREENSHOT: Use browser_screenshot to visually identify the element +6. REPORT: If all fail, inform user with what was tried and the current page state +``` + +### Navigation Failure Recovery +``` +1. Timeout on browser_navigate: + a. Retry the same URL once + b. If still fails, check if URL is valid (no typos, correct protocol) + c. Try simplified URL (remove query params, try base domain) + d. Report connectivity issue to user + +2. Unexpected redirect: + a. browser_read_page → check current URL and content + b. If redirected to login → handle login flow + c. If redirected to error page → report the error code and message + d. If redirected to different page → assess if it is relevant, otherwise navigate back + +3. HTTP errors observed in page content: + - 403 Forbidden → site may be blocking automation, inform user + - 404 Not Found → URL is stale or incorrect, search for correct URL + - 429 Too Many Requests → wait 60 seconds, retry with longer intervals + - 500/502/503 → server issue, retry after 30 seconds (max 3 retries) +``` + +### Stale Element Recovery +Elements can become stale when the page re-renders (common in SPAs): +``` +1. Identify the stale interaction (click that failed after page update) +2. browser_read_page → get fresh DOM snapshot +3. Re-locate the element using the same or updated selector +4. Retry the interaction on the fresh element +5. If element has moved or changed structure, use browser_screenshot + to visually identify its new position +``` + +### Pop-up and Overlay Dismissal +``` +Order of priority when dealing with overlays blocking interaction: +1. Cookie consent banners: + - Click: button containing "Accept", "Agree", "OK", "Got it" + - Selectors: #cookie-accept, .cookie-consent button, [data-action="accept"] + - Fallback: .cookie-banner .close, #cookie-close + +2. Newsletter/subscription modals: + - Click: close/dismiss button + - Selectors: .modal .close, [aria-label="Close"], button.dismiss + - Fallback: press Escape key (if supported) + +3. Chat widgets: + - Usually do not block interaction (z-index overlay) + - If blocking, click minimize/close on the widget + +4. Age verification gates: + - Click "Yes" / "I am over 18" / "Enter" + - May need to select a birth date from dropdowns + +5. Location/notification permission prompts: + - These are browser-level, not DOM elements + - Usually auto-dismissed by Playwright's browser context settings +``` + +--- + +## Anti-Detection Techniques + +### User Agent Strategy +Rotate user agents to appear as a normal browser. Common realistic user agents: +``` +Desktop Chrome (Windows): + Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 + +Desktop Chrome (macOS): + Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 + +Desktop Firefox (Windows): + Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0 + +Mobile Chrome (Android): + Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Mobile Safari/537.36 + +Mobile Safari (iOS): + Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Mobile/15E148 Safari/604.1 +``` + +### Viewport Randomization +Use realistic viewport sizes with slight variation to avoid fingerprinting: +``` +Common realistic viewports: + Desktop: 1920x1080, 1366x768, 1536x864, 1440x900, 1280x720 + Tablet: 1024x768, 768x1024 (portrait), 1280x800 + Mobile: 375x812, 390x844, 360x780, 414x896 + +Add random offsets (1-20px) to avoid exact-match detection: + 1920x1080 → 1923x1077 (slightly varied) +``` + +### Behavioral Patterns +Automation detection looks for non-human interaction patterns. Mitigate by: +``` +1. TIMING: Do not click or type instantly after page load + - Wait 1-3 seconds before first interaction + - Insert 0.5-2 second gaps between form field entries + - Vary timing between actions (not perfectly uniform) + +2. NAVIGATION: Follow natural browsing patterns + - Visit homepage before going directly to deep URLs + - Click through navigation menus instead of using direct URLs when possible + - Scroll the page before interacting with below-the-fold content + +3. MOUSE/KEYBOARD: Simulate realistic input + - Type into fields character by character (browser_type handles this) + - Click buttons rather than submitting forms programmatically + - Do not fill hidden honeypot fields (fields with display:none or visibility:hidden) + +4. AVOID DETECTABLE PATTERNS: + - Do not request pages faster than 1 per 3 seconds on the same domain + - Do not access robots.txt-blocked paths + - Do not make requests in perfectly uniform intervals +``` + +### Honeypot Field Detection +Some forms include invisible fields designed to catch bots: +``` +Do NOT fill fields that have: + - style="display: none" + - style="visibility: hidden" + - class="hidden", class="d-none", class="sr-only" + - type="hidden" (unless it is a legitimate CSRF token or form ID) + - Position: absolute with left: -9999px or similar off-screen placement + +Use browser_read_page to inspect field visibility before filling. +``` + +--- + +## Screenshot & Content Extraction + +### When to Take Screenshots +| Situation | Purpose | +|-----------|---------| +| Before form submission | Visual verification of filled data | +| After login attempt | Confirm success or capture error state | +| When element not found | See actual page state for debugging | +| Price/product comparison | Visual record for user | +| CAPTCHA encountered | Show user what needs solving | +| Before financial transaction | Proof of cart/payment details | +| Unexpected page state | Diagnose navigation or rendering issues | + +### Content Extraction Patterns + +**Extracting structured data from tables:** +``` +1. browser_read_page → get full page text +2. Identify table boundaries in the text output +3. Parse rows and columns from the structured text +4. memory_store → save as structured data for comparison +``` + +**Extracting specific data points:** +``` +1. browser_read_page → get page content +2. Search output for relevant labels/headings: + - "Price:", "Total:", "Subtotal:" → monetary values + - "In Stock", "Available", "Sold Out" → availability + - "Rating:", stars → review scores + - "SKU:", "Item #:" → product identifiers +3. Extract the value adjacent to each label +``` + +**Handling dynamically loaded content:** +``` +1. browser_read_page → check if content placeholder exists +2. If content shows "Loading..." or skeleton elements: + a. Wait 2-3 seconds + b. browser_read_page → retry +3. If content requires scroll-to-load (infinite scroll): + a. Extract visible data + b. Scroll down (click a lower element or use page navigation) + c. browser_read_page → extract newly loaded data + d. Repeat until desired amount collected or no new content appears +``` + +--- + +## Form Filling & Interaction Sequences + +### Dropdown / Select Menus +``` +Standard HTML element to open it + browser_click → the