mirror of
https://github.com/browser-use/browser-use.git
synced 2026-10-02 04:04:36 +08:00
Update docs
This commit is contained in:
Magnus Müller
@@ -20,8 +20,8 @@ At every step, your input will consist of:
|
||||
1. <agent_history>: A chronological event stream including your previous actions and their results.
|
||||
2. <agent_state>: Current <user_request>, summary of <file_system>, <todo_contents>, and <step_info>.
|
||||
3. <browser_state>: Current URL, open tabs, interactive elements indexed for actions, and visible page content.
|
||||
4. <browser_vision>: Screenshot of the browser with bounding boxes around interactive elements. If you used take_screenshot before, this will contain a screenshot.
|
||||
5. <read_state> This will be displayed only if your previous action was extract_structured_data or read_file. This data is only shown in the current step.
|
||||
4. <browser_vision>: Screenshot of the browser with bounding boxes around interactive elements. If you used screenshot before, this will contain a screenshot.
|
||||
5. <read_state> This will be displayed only if your previous action was extract or read_file. This data is only shown in the current step.
|
||||
</input>
|
||||
|
||||
<agent_history>
|
||||
@@ -61,14 +61,14 @@ Examples:
|
||||
Note that:
|
||||
- Only elements with numeric indexes in [] are interactive
|
||||
- (stacked) indentation (with \t) is important and means that the element is a (html) child of the element above (with a lower index)
|
||||
- Elements tagged with a star `*[` are the new interactive elements that appeared on the website since the last step - if url has not changed. Your previous actions caused that change. Think if you need to interact with them, e.g. after input_text you might need to select the right option from the list.
|
||||
- Elements tagged with a star `*[` are the new interactive elements that appeared on the website since the last step - if url has not changed. Your previous actions caused that change. Think if you need to interact with them, e.g. after input you might need to select the right option from the list.
|
||||
- Pure text elements without [] are not interactive.
|
||||
</browser_state>
|
||||
|
||||
<browser_vision>
|
||||
If you used take_screenshot before, you will be provided with a screenshot of the current page with bounding boxes around interactive elements. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
|
||||
If you used screenshot before, you will be provided with a screenshot of the current page with bounding boxes around interactive elements. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
|
||||
If an interactive index inside your browser_state does not have text information, then the interactive index is written at the top center of it's element in the screenshot.
|
||||
Use take_screenshot if you are unsure or simply want more information.
|
||||
Use screenshot if you are unsure or simply want more information.
|
||||
</browser_vision>
|
||||
|
||||
<browser_rules>
|
||||
@@ -78,18 +78,18 @@ Strictly follow these rules while using the browser and navigating the web:
|
||||
- If research is needed, open a **new tab** instead of reusing the current one.
|
||||
- If the page changes after, for example, an input text action, analyse if you need to interact with new elements, e.g. selecting the right option from the list.
|
||||
- By default, only elements in the visible viewport are listed. Use scrolling tools if you suspect relevant content is offscreen which you need to interact with. Scroll ONLY if there are more pixels below or above the page.
|
||||
- You can scroll by a specific number of pages using the num_pages parameter (e.g., 0.5 for half page, 2.0 for two pages).
|
||||
- You can scroll by a specific number of pages using the pages parameter (e.g., 0.5 for half page, 2.0 for two pages).
|
||||
- If a captcha appears, attempt solving it if possible. If not, use fallback strategies (e.g., alternative site, backtrack).
|
||||
- If expected elements are missing, try refreshing, scrolling, or navigating back.
|
||||
- If the page is not fully loaded, use the wait action.
|
||||
- You can call extract_structured_data on specific pages to gather structured semantic information from the entire page, including parts not currently visible.
|
||||
- Call extract_structured_data only if the information you are looking for is not visible in your <browser_state> otherwise always just use the needed text from the <browser_state>.
|
||||
- Calling the extract_structured_data tool is expensive! DO NOT query the same page with the same extract_structured_data query multiple times. Make sure that you are on the page with relevant information based on the screenshot before calling this tool.
|
||||
- You can call extract on specific pages to gather structured semantic information from the entire page, including parts not currently visible.
|
||||
- Call extract only if the information you are looking for is not visible in your <browser_state> otherwise always just use the needed text from the <browser_state>.
|
||||
- Calling the extract tool is expensive! DO NOT query the same page with the same extract query multiple times. Make sure that you are on the page with relevant information based on the screenshot before calling this tool.
|
||||
- If you fill an input field and your action sequence is interrupted, most often something changed e.g. suggestions popped up under the field.
|
||||
- If the action sequence was interrupted in previous step due to page changes, make sure to complete any remaining actions that were not executed. For example, if you tried to input text and click a search button but the click was not executed because the page changed, you should retry the click action in your next step.
|
||||
- If the <user_request> includes specific page information such as product type, rating, price, location, etc., try to apply filters to be more efficient.
|
||||
- The <user_request> is the ultimate goal. If the user specifies explicit steps, they have always the highest priority.
|
||||
- If you input_text into a field, you might need to press enter, click the search button, or select from dropdown for completion.
|
||||
- If you input into a field, you might need to press enter, click the search button, or select from dropdown for completion.
|
||||
- Don't login into a page if you don't have to. Don't login if you don't have the credentials.
|
||||
- There are 2 types of tasks always first think which type of request you are dealing with:
|
||||
1. Very specific step by step instructions:
|
||||
@@ -101,7 +101,7 @@ Strictly follow these rules while using the browser and navigating the web:
|
||||
|
||||
<file_system>
|
||||
- You have access to a persistent file system which you can use to track progress, store results, and manage long tasks.
|
||||
- Your file system is initialized with a `todo.md`: Use this to keep a checklist for known subtasks. Use `replace_file_str` tool to update markers in `todo.md` as first action whenever you complete an item. This file should guide your step-by-step execution when you have a long running task.
|
||||
- Your file system is initialized with a `todo.md`: Use this to keep a checklist for known subtasks. Use `replace_file` tool to update markers in `todo.md` as first action whenever you complete an item. This file should guide your step-by-step execution when you have a long running task.
|
||||
- If you are writing a `csv` file, make sure to use double quotes if cell elements contain commas.
|
||||
- If the file is too large, you are only given a preview of your file. Use `read_file` to see the full content if necessary.
|
||||
- If exists, <available_file_paths> includes files you have downloaded or uploaded by the user. You can only read or upload these files but you don't have write access.
|
||||
@@ -138,17 +138,17 @@ If you are allowed multiple actions, you can specify multiple actions in the lis
|
||||
You can output multiple actions in one step. Try to be efficient where it makes sense. Do not predict actions which do not make sense for the current page.
|
||||
|
||||
**Recommended Action Combinations:**
|
||||
- `input_text` + `click_element_by_index` → Fill form field and submit/search in one step
|
||||
- `input_text` + `input_text` → Fill multiple form fields
|
||||
- `click_element_by_index` + `click_element_by_index` → Navigate through multi-step flows (when the page does not navigate between clicks)
|
||||
- `scroll` with num_pages 10 + `extract_structured_data` → Scroll to the bottom of the page to load more content before extracting structured data
|
||||
- File operations + browser actions
|
||||
- `input` + `click` → Fill form field and submit/search in one step
|
||||
- `input` + `input` → Fill multiple form fields
|
||||
- `click` + `click` → Navigate through multi-step flows (when the page does not navigate between clicks)
|
||||
- `scroll` with pages 10 + `extract` → Scroll to the bottom of the page to load more content before extracting structured data
|
||||
- File operations + browser actions
|
||||
|
||||
Do not try multiple different paths in one step. Always have one clear goal per step.
|
||||
Its important that you see in the next step if your action was successful, so do not chain actions which change the browser state multiple times, e.g.
|
||||
- do not use click_element_by_index and then navigate, because you would not see if the click was successful or not.
|
||||
- or do not use switch_tab and switch_tab together, because you would not see the state in between.
|
||||
- do not use input_text and then scroll, because you would not see if the input text was successful or not.
|
||||
Do not try multiple different paths in one step. Always have one clear goal per step.
|
||||
Its important that you see in the next step if your action was successful, so do not chain actions which change the browser state multiple times, e.g.
|
||||
- do not use click and then navigate, because you would not see if the click was successful or not.
|
||||
- or do not use switch and switch together, because you would not see the state in between.
|
||||
- do not use input and then scroll, because you would not see if the input was successful or not.
|
||||
</efficiency_guidelines>
|
||||
|
||||
<reasoning_rules>
|
||||
|
||||
@@ -20,8 +20,8 @@ At every step, your input will consist of:
|
||||
1. <agent_history>: A chronological event stream including your previous actions and their results.
|
||||
2. <agent_state>: Current <user_request>, summary of <file_system>, <todo_contents>, and <step_info>.
|
||||
3. <browser_state>: Current URL, open tabs, interactive elements indexed for actions, and visible page content.
|
||||
4. <browser_vision>: Screenshot of the browser with bounding boxes around interactive elements. If you used take_screenshot before, this will contain a screenshot.
|
||||
5. <read_state> This will be displayed only if your previous action was extract_structured_data or read_file. This data is only shown in the current step.
|
||||
4. <browser_vision>: Screenshot of the browser with bounding boxes around interactive elements. If you used screenshot before, this will contain a screenshot.
|
||||
5. <read_state> This will be displayed only if your previous action was extract or read_file. This data is only shown in the current step.
|
||||
</input>
|
||||
|
||||
<agent_history>
|
||||
@@ -61,14 +61,14 @@ Examples:
|
||||
Note that:
|
||||
- Only elements with numeric indexes in [] are interactive
|
||||
- (stacked) indentation (with \t) is important and means that the element is a (html) child of the element above (with a lower index)
|
||||
- Elements tagged with a star `*[` are the new interactive elements that appeared on the website since the last step - if url has not changed. Your previous actions caused that change. Think if you need to interact with them, e.g. after input_text you might need to select the right option from the list.
|
||||
- Elements tagged with a star `*[` are the new interactive elements that appeared on the website since the last step - if url has not changed. Your previous actions caused that change. Think if you need to interact with them, e.g. after input you might need to select the right option from the list.
|
||||
- Pure text elements without [] are not interactive.
|
||||
</browser_state>
|
||||
|
||||
<browser_vision>
|
||||
If you used take_screenshot before, you will be provided with a screenshot of the current page with bounding boxes around interactive elements. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
|
||||
If you used screenshot before, you will be provided with a screenshot of the current page with bounding boxes around interactive elements. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
|
||||
If an interactive index inside your browser_state does not have text information, then the interactive index is written at the top center of it's element in the screenshot.
|
||||
Use take_screenshot if you are unsure or simply want more information.
|
||||
Use screenshot if you are unsure or simply want more information.
|
||||
</browser_vision>
|
||||
|
||||
<browser_rules>
|
||||
@@ -78,18 +78,18 @@ Strictly follow these rules while using the browser and navigating the web:
|
||||
- If research is needed, open a **new tab** instead of reusing the current one.
|
||||
- If the page changes after, for example, an input text action, analyse if you need to interact with new elements, e.g. selecting the right option from the list.
|
||||
- By default, only elements in the visible viewport are listed. Use scrolling tools if you suspect relevant content is offscreen which you need to interact with. Scroll ONLY if there are more pixels below or above the page.
|
||||
- You can scroll by a specific number of pages using the num_pages parameter (e.g., 0.5 for half page, 2.0 for two pages).
|
||||
- You can scroll by a specific number of pages using the pages parameter (e.g., 0.5 for half page, 2.0 for two pages).
|
||||
- If a captcha appears, attempt solving it if possible. If not, use fallback strategies (e.g., alternative site, backtrack).
|
||||
- If expected elements are missing, try refreshing, scrolling, or navigating back.
|
||||
- If the page is not fully loaded, use the wait action.
|
||||
- You can call extract_structured_data on specific pages to gather structured semantic information from the entire page, including parts not currently visible.
|
||||
- Call extract_structured_data only if the information you are looking for is not visible in your <browser_state> otherwise always just use the needed text from the <browser_state>.
|
||||
- Calling the extract_structured_data tool is expensive! DO NOT query the same page with the same extract_structured_data query multiple times. Make sure that you are on the page with relevant information based on the screenshot before calling this tool.
|
||||
- You can call extract on specific pages to gather structured semantic information from the entire page, including parts not currently visible.
|
||||
- Call extract only if the information you are looking for is not visible in your <browser_state> otherwise always just use the needed text from the <browser_state>.
|
||||
- Calling the extract tool is expensive! DO NOT query the same page with the same extract query multiple times. Make sure that you are on the page with relevant information based on the screenshot before calling this tool.
|
||||
- If you fill an input field and your action sequence is interrupted, most often something changed e.g. suggestions popped up under the field.
|
||||
- If the action sequence was interrupted in previous step due to page changes, make sure to complete any remaining actions that were not executed. For example, if you tried to input text and click a search button but the click was not executed because the page changed, you should retry the click action in your next step.
|
||||
- If the <user_request> includes specific page information such as product type, rating, price, location, etc., try to apply filters to be more efficient.
|
||||
- The <user_request> is the ultimate goal. If the user specifies explicit steps, they have always the highest priority.
|
||||
- If you input_text into a field, you might need to press enter, click the search button, or select from dropdown for completion.
|
||||
- If you input into a field, you might need to press enter, click the search button, or select from dropdown for completion.
|
||||
- Don't login into a page if you don't have to. Don't login if you don't have the credentials.
|
||||
- There are 2 types of tasks always first think which type of request you are dealing with:
|
||||
1. Very specific step by step instructions:
|
||||
@@ -101,7 +101,7 @@ Strictly follow these rules while using the browser and navigating the web:
|
||||
|
||||
<file_system>
|
||||
- You have access to a persistent file system which you can use to track progress, store results, and manage long tasks.
|
||||
- Your file system is initialized with a `todo.md`: Use this to keep a checklist for known subtasks. Use `replace_file_str` tool to update markers in `todo.md` as first action whenever you complete an item. This file should guide your step-by-step execution when you have a long running task.
|
||||
- Your file system is initialized with a `todo.md`: Use this to keep a checklist for known subtasks. Use `replace_file` tool to update markers in `todo.md` as first action whenever you complete an item. This file should guide your step-by-step execution when you have a long running task.
|
||||
- If you are writing a `csv` file, make sure to use double quotes if cell elements contain commas.
|
||||
- If the file is too large, you are only given a preview of your file. Use `read_file` to see the full content if necessary.
|
||||
- If exists, <available_file_paths> includes files you have downloaded or uploaded by the user. You can only read or upload these files but you don't have write access.
|
||||
@@ -137,17 +137,17 @@ If you are allowed multiple actions, you can specify multiple actions in the lis
|
||||
You can output multiple actions in one step. Try to be efficient where it makes sense. Do not predict actions which do not make sense for the current page.
|
||||
|
||||
**Recommended Action Combinations:**
|
||||
- `input_text` + `click_element_by_index` → Fill form field and submit/search in one step
|
||||
- `input_text` + `input_text` → Fill multiple form fields
|
||||
- `click_element_by_index` + `click_element_by_index` → Navigate through multi-step flows (when the page does not navigate between clicks)
|
||||
- `scroll` with num_pages 10 + `extract_structured_data` → Scroll to the bottom of the page to load more content before extracting structured data
|
||||
- `input` + `click` → Fill form field and submit/search in one step
|
||||
- `input` + `input` → Fill multiple form fields
|
||||
- `click` + `click` → Navigate through multi-step flows (when the page does not navigate between clicks)
|
||||
- `scroll` with pages 10 + `extract` → Scroll to the bottom of the page to load more content before extracting structured data
|
||||
- File operations + browser actions
|
||||
|
||||
Do not try multiple different paths in one step. Always have one clear goal per step.
|
||||
Its important that you see in the next step if your action was successful, so do not chain actions which change the browser state multiple times, e.g.
|
||||
- do not use click_element_by_index and then navigate, because you would not see if the click was successful or not.
|
||||
- or do not use switch_tab and switch_tab together, because you would not see the state in between.
|
||||
- do not use input_text and then scroll, because you would not see if the input text was successful or not.
|
||||
- do not use click and then navigate, because you would not see if the click was successful or not.
|
||||
- or do not use switch and switch together, because you would not see the state in between.
|
||||
- do not use input and then scroll, because you would not see if the input was successful or not.
|
||||
</efficiency_guidelines>
|
||||
|
||||
<reasoning_rules>
|
||||
|
||||
@@ -327,12 +327,10 @@ class AgentHistory(BaseModel):
|
||||
if self.model_output:
|
||||
action_dump = [action.model_dump(exclude_none=True) for action in self.model_output.action]
|
||||
|
||||
# Filter sensitive data only from input_text action parameters if sensitive_data is provided
|
||||
# Filter sensitive data only from input action parameters if sensitive_data is provided
|
||||
if sensitive_data:
|
||||
action_dump = [
|
||||
self._filter_sensitive_data_from_dict(action, sensitive_data)
|
||||
if action.get('name') == 'input_text'
|
||||
else action
|
||||
self._filter_sensitive_data_from_dict(action, sensitive_data) if action.get('name') == 'input' else action
|
||||
for action in action_dump
|
||||
]
|
||||
|
||||
|
||||
@@ -273,7 +273,7 @@ class Tools(Generic[Context]):
|
||||
except BrowserError as e:
|
||||
if 'Cannot click on <select> elements.' in str(e):
|
||||
try:
|
||||
return await get_dropdown_options(
|
||||
return await dropdown_options(
|
||||
params=GetDropdownOptionsAction(index=params.index), browser_session=browser_session
|
||||
)
|
||||
except Exception as dropdown_error:
|
||||
|
||||
@@ -12,7 +12,7 @@ Prompting can trasticly improve performance and solve existing limitations of th
|
||||
```python
|
||||
task = """
|
||||
1. Go to https://quotes.toscrape.com/
|
||||
2. Use extract_structured_data action with the query "first 3 quotes with their authors"
|
||||
2. Use extract action with the query "first 3 quotes with their authors"
|
||||
3. Save results to quotes.csv using write_file action
|
||||
4. Do a google search for the first quote and find when it was written
|
||||
"""
|
||||
@@ -30,11 +30,11 @@ When you know exactly what the agent should do, reference actions by name:
|
||||
```python
|
||||
task = """
|
||||
1. Use search action to find "Python tutorials"
|
||||
2. Use click_element_by_index to open first result in a new tab
|
||||
2. Use click to open first result in a new tab
|
||||
3. Use scroll action to scroll down 2 pages
|
||||
4. Use extract_structured_data to extract the names of the first 5 items
|
||||
4. Use extract to extract the names of the first 5 items
|
||||
5. Wait for 2 seconds if the page is not loaded, refresh it and wait 10 sec
|
||||
6. Use send_keys action with "Tab Tab ArrowDown Enter"
|
||||
6. Use send_keys action with "Tab Tab ArrowDown Enter"
|
||||
"""
|
||||
```
|
||||
|
||||
|
||||
@@ -9,40 +9,40 @@ mode: "wide"
|
||||
|
||||
|
||||
### Navigation & Browser Control
|
||||
- **`search`** - Search queries in Google
|
||||
- **`navigate`** - Navigate to URLs
|
||||
- **`search`** - Search queries (DuckDuckGo, Google, Bing)
|
||||
- **`navigate`** - Navigate to URLs
|
||||
- **`go_back`** - Go back in browser history
|
||||
- **`wait`** - Wait for specified seconds
|
||||
|
||||
### Page Interaction
|
||||
- **`click_element_by_index`** - Click elements by their index
|
||||
- **`input_text`** - Input text into form fields
|
||||
- **`upload_file_to_element`** - Upload files to file inputs
|
||||
- **`click`** - Click elements by their index
|
||||
- **`input`** - Input text into form fields
|
||||
- **`upload_file`** - Upload files to file inputs
|
||||
- **`scroll`** - Scroll the page up/down
|
||||
- **`scroll_to_text`** - Scroll to specific text on page
|
||||
- **`find_text`** - Scroll to specific text on page
|
||||
- **`send_keys`** - Send special keys (Enter, Escape, etc.)
|
||||
|
||||
### JavaScript Execution
|
||||
- **`execute_js`** - Execute custom JavaScript code on the page (for advanced interactions, shadow DOM, custom selectors, data extraction)
|
||||
- **`evaluate`** - Execute custom JavaScript code on the page (for advanced interactions, shadow DOM, custom selectors, data extraction)
|
||||
|
||||
### Tab Management
|
||||
- **`switch_tab`** - Switch between browser tabs
|
||||
- **`close_tab`** - Close browser tabs
|
||||
### Tab Management
|
||||
- **`switch`** - Switch between browser tabs
|
||||
- **`close`** - Close browser tabs
|
||||
|
||||
### Content Extraction
|
||||
- **`extract_structured_data`** - Extract data from webpages using LLM
|
||||
- **`extract`** - Extract data from webpages using LLM
|
||||
|
||||
### Visual Analysis
|
||||
- **`take_screenshot`** - Request a screenshot in your next browser state for visual confirmation
|
||||
- **`screenshot`** - Request a screenshot in your next browser state for visual confirmation
|
||||
|
||||
### Form Controls
|
||||
- **`get_dropdown_options`** - Get dropdown option values
|
||||
- **`select_dropdown_option`** - Select dropdown options
|
||||
- **`dropdown_options`** - Get dropdown option values
|
||||
- **`select_dropdown`** - Select dropdown options
|
||||
|
||||
### File Operations
|
||||
- **`write_file`** - Write content to files
|
||||
- **`read_file`** - Read file contents
|
||||
- **`replace_file_str`** - Replace text in files
|
||||
- **`read_file`** - Read file contents
|
||||
- **`replace_file`** - Replace text in files
|
||||
|
||||
### Task Completion
|
||||
- **`done`** - Complete the task (always available)
|
||||
|
||||
@@ -24,7 +24,7 @@ Then, use append_file to add the first sentence of the article to "data.md"
|
||||
Then, read the file to see its content and make sure it's correct.
|
||||
Finally, share the file with me.
|
||||
|
||||
NOTE: DO NOT USE extract_structured_data action - everything is visible in browser state.
|
||||
NOTE: DO NOT USE extract action - everything is visible in browser state.
|
||||
""".strip('\n')
|
||||
|
||||
llm = ChatOpenAI(model='gpt-4.1-mini')
|
||||
|
||||
Reference in New Issue
Block a user