Using Replit AI as an Browse Scraping Agent

Firefox Browser Automation - Direct Command Execution

Browser Launch Mouse/Keyboard Tab Management Advanced Examples Troubleshooting

Screenshot of Replit Agent


This document provides examples of browser automation using direct command execution with xdotool. These commands can be executed directly in the terminal for browser automation without scripts.

Key Benefit: Using direct command execution allows for quick automation without frameworks like Selenium or Playwright, and provides immediate feedback during execution.

1. Browser Launch and Navigation

Task Command
Launch Firefox with a URL firefox https://www.google.com &
Get Firefox window ID xdotool search --name "Firefox" | head -1
Get window name from ID xdotool getwindowname WINDOW_ID
Activate window by ID xdotool windowactivate WINDOW_ID
Navigate to URL (address bar) xdotool key ctrl+l && xdotool type "https://example.com" && xdotool key Return

2. Mouse and Keyboard Interaction

Task Command
Click at coordinates xdotool mousemove X Y click 1
Right-click at coordinates xdotool mousemove X Y click 3
Type text xdotool type "text to type"
Press keyboard shortcut xdotool key ctrl+t
Press multiple keys xdotool key ctrl+shift+i

3. Tab Management

Task Command
Open new tab xdotool key ctrl+t
Switch to specific tab xdotool key ctrl+1 (for first tab, ctrl+2 for second, etc.)
Close current tab xdotool key ctrl+w
Switch to next tab xdotool key ctrl+Tab
Switch to previous tab xdotool key ctrl+shift+Tab

4. Screenshot Capture

Task Command
Take full screenshot gnome-screenshot -f /tmp/screenshot.png
Take screenshot with delay sleep 3 && gnome-screenshot -f /tmp/screenshot.png
Take screenshot of window gnome-screenshot -w -f /tmp/window.png

5. Advanced Interactions

Task Command
Open context menu and select item xdotool mousemove X Y click 3 && sleep 1 && xdotool key Down Down Return
Page scroll down xdotool key Page_Down
Open developer tools xdotool key F12
Reload page xdotool key ctrl+r
Search on page xdotool key ctrl+f && xdotool type "search term" && xdotool key Return
Navigate back in history xdotool key alt+Left
Navigate forward in history xdotool key alt+Right

6. Working with Context Menu and Clipboard

Task Command
Open context menu xdotool click 3
Select all text xdotool key ctrl+a
Copy selected text xdotool key ctrl+c
Paste from clipboard xdotool key ctrl+v
Cut selected text xdotool key ctrl+x
Save page as xdotool key ctrl+s
Print page xdotool key ctrl+p

7. Handling Browser Dialogs

Task Command
Accept alert dialog xdotool key Return
Dismiss confirmation dialog xdotool key Tab Return
Cancel prompt dialog xdotool key Escape
Handle authentication dialog xdotool type "username" && xdotool key Tab && xdotool type "password" && xdotool key Return
Close a tab with unsaved changes xdotool key ctrl+w && sleep 1 && xdotool key Tab Return

8. Example Workflows

Google Search Workflow

# Launch Firefox with Google
firefox https://www.google.com &

# Wait for Firefox to load and get window ID
sleep 5
FIREFOX_ID=$(xdotool search --name "Firefox" | head -1)

# Focus the window
xdotool windowactivate $FIREFOX_ID

# Click on search box
xdotool mousemove 500 300 click 1

# Type search query
xdotool type "browser automation examples"

# Submit search
xdotool key Return

# Take screenshot of results
sleep 3
gnome-screenshot -f /tmp/search_results.png

Form Submission Workflow

# Launch Firefox with form page
firefox https://httpbin.org/forms/post &

# Wait for Firefox to load and get window ID
sleep 5
FIREFOX_ID=$(xdotool search --onlyvisible --name "Firefox" | head -1)

# Focus the window
xdotool windowactivate $FIREFOX_ID

# Wait for page to fully load
sleep 3

# Fill out Customer Name
xdotool key Tab Tab && sleep 0.5
xdotool type "John Doe"

# Select Pizza Size (Medium)
xdotool key Tab && sleep 0.5
xdotool key Down

# Select Pizza Topping (Onion)
xdotool key Tab Tab && sleep 0.5
xdotool key Down Down && sleep 0.5
xdotool key space

# Best Pizza dropdown - select "New Haven Style"
xdotool key Tab && sleep 0.5
xdotool key Down Down

# Enter comment in textarea
xdotool key Tab && sleep 0.5
xdotool type "This is a complete form submission test."

# Take screenshot before submission
gnome-screenshot -f /tmp/form_filled.png

# Tab to the Submit button and click it
xdotool key Tab Tab && sleep 0.5
xdotool key space

9. Troubleshooting Notes

  • If XGetWindowProperty[_NET_WM_DESKTOP] failed (code=1) appears, it's usually harmless and can be ignored
  • For more reliable window identification, use --onlyvisible with xdotool search
  • Allow sufficient time between commands with sleep to ensure the browser has time to respond
  • Screen coordinates are relative to the current display resolution
  • For better reliability, combine commands with && to ensure sequential execution
  • If website structure changes, tab navigation sequences may need adjustment
  • Browser version differences may affect keyboard shortcuts and dialog behavior
  • Context menu structures vary by browser, OS, and website
Important: Always include sufficient delay (sleep) between commands when automating browser actions. Browser response times can vary based on page complexity and system load.

For more detailed examples and workflows, see the GitHub repository or refer to the direct_commands.md file.

Comments

Post a Comment

Popular posts from this blog

local LLM runners like Ollama, GPT4All, and LMStudio

Image
This is a guide for comparing local LLM runners like Ollama , GPT4All , and LMStudio for running models on an NVIDIA GeForce RTX 4090, here’s a breakdown of the options: 1. Ollama • Pros : • Excellent for macOS and Apple Silicon (M1/M2), but less optimized for NVIDIA GPUs . • Focuses on a user-friendly interface and pre-configured models. • Limited support for CUDA-based acceleration. • Cons : • No deep customization or optimization for high-end GPUs like the RTX 4090. • Slower compared to other runners optimized for NVIDIA GPUs. • Best For : • Users with minimal technical experience who prioritize ease of use. 2. GPT4All • Pros : • Supports NVIDIA GPUs with CUDA acceleration . • Works with a variety of quantized models (e.g., 4-bit, 8-bit). • Lightweight and user-friendly, with CLI and GUI options. • Supports LLaMA, Falcon, and GPT-J family models. • Cons : • Performance may not fully utilize the RTX 4090’s capabilities...
Read more

Understanding Radix UI, shadcn/ui, and Component Architecture in Modern Web Development

Understanding Radix UI, shadcn/ui, and Component Architecture in Modern Web Development The Component Ecosystem Modern web development relies heavily on reusable components. But there's often confusion about different types of component libraries and their purposes. Let's break down the ecosystem using Radix UI and shadcn/ui as perfect examples. The Three Layers of Components Primitive Components (e.g., Radix UI) Raw, unstyled functionality Accessibility built-in Focus on behavior and interactions Example: A dropdown that handles keyboard navigation but has no visual styling 2.  Styled Components (e.g., shadcn/ui) Built on top of primitives Add visual design and styling Implement specific design systems Example: A beautifully styled dropdown using Radix UI's primitive underneath Application Components Built for specific use cases Combine multiple styled components Include business logic Example: A user settings menu u...
Read more

Supabase Storage Image Uploader Guide (Agentic Oriented)

Image
Supabase Storage Image Uploader Guide (cURL-Based Approach) This guide provides comprehensive instructions for working with Supabase Storage using cURL commands. This approach allows an agent to perform all operations directly through the command line. 1. Authentication All Supabase API requests require authentication. You'll need your Supabase project URL and API key: # Set environment variables for easier reuse export SUPABASE_URL= "https://YOUR_PROJECT_REF.supabase.co" export SUPABASE_KEY= "YOUR_SUPABASE_KEY" # Use anon key for client operations or service_role key for admin operations 2. Creating and Managing Buckets Create a Storage Bucket: curl -X POST " ${SUPABASE_URL} /storage/v1/bucket" \ -H "Authorization: Bearer ${SUPABASE_KEY} " \ -H "Content-Type: application/json" \ -d '{ "name" : "images" , "public" : false }' List All Buckets: curl -X GET " ${SUPAB...
Read more