202 Commits
Author SHA1 Message Date
W fb26a7ab91 fix(shell): stop dropping cookies whose values SimpleCookie rejects (#541)
* fix(shell): stop dropping cookies whose values SimpleCookie rejects

_CookieParser fed the cookie string to http.cookies.SimpleCookie, which
stops at the first value it considers illegal (a space, a double quote,
...) and silently discards that cookie and everything after it. Cookie
strings copied from DevTools contain such values regularly, so curl
commands parsed by uncurl/curl2fetcher and `extract --cookies` lost
cookies without any warning.

Split the string on ";" instead and keep each value as sent, including
surrounding quotes, which SimpleCookie used to strip. Set-Cookie
attribute names are still skipped, as before.

* fix(shell): keep request cookies named partitioned
2026-11-04 16:08:55 +02:00
Jay Hemnani ead76eadde fix(shell): accept curl's -L flag (#554) 2026-11-04 16:04:36 +02:00
Karim shoair c83d10bdd1 feat(mcp): replace one mouse drag action with two mouse control actions
Making it able to solve all slider captchas, tested on all datadome and geetest CAPTCHAs I can find
2026-11-03 18:52:25 +02:00
Karim shoair a16d0cf8f0 fix(mcp): rewrite docstrings to save tokens 2026-11-03 18:52:25 +02:00
Karim shoair 37f448d689 feat(mcp): add browser observe tool to record the browser. 2026-11-03 18:52:25 +02:00
Karim shoair b02a20af68 feat(fetchers)!: move remote browsers sessions to separate locations with more control 2026-10-31 16:39:10 +02:00
Karim shoair fbc2465abc feat(mcp): add mouse drag-and-drop action 2026-10-19 18:42:44 +03:00
Huang Qi 5e1383df3a fix: disambiguate generated selectors for first same-tag siblings (#495)
Count all siblings while retaining the target's same-tag position so
first siblings and ancestors receive a positional qualifier when needed.
Add 40 parameterized regression cases for CSS and XPath round trips.
2026-10-16 03:02:23 +03:00
Suliman Abdulrazzaq bd90176c66 fix(shell): apply curl -A and pass -x proxies as a URL string (#490) 2026-10-16 01:10:40 +03:00
Suliman Abdulrazzaq 71f567e806 fix(cli): write md and txt extract output as UTF-8 (#488) 2026-10-16 00:48:47 +03:00
Karim shoair 0f27a77ac0 refactor(mcp): move tools descriptions to the definition itself to save tokens on definition 2026-10-15 20:34:58 +03:00
Karim shoair 3c3e3f95ff perf(mcp): Make session creating return shorter output 2026-10-14 20:42:26 +03:00
Karim shoair c54966a8c7 refactor(fetchers)!: Remove dynamic sessions and dynamic fetcher 2026-10-08 23:28:12 +03:00
Karim shoair 99a2281923 feat(mcp): Add new tools to set and load browser session state 2026-10-08 23:28:12 +03:00
Karim shoair 890d0c08c1 feat/refactor(mcp)!: Renamed browser_snapshot to browser_extract while adding more options
This gives the agent more control
2026-10-07 19:37:56 +03:00
Karim shoair 7d97d07276 feat(mcp): Add a dialog handling action 2026-10-07 02:25:18 +03:00
Karim shoair 5c39feb851 feat(mcp): Add snapshot searching 2026-10-07 02:10:19 +03:00
Karim shoair 2541e6b7fb Merge branch 'dev' of https://github.com/D4Vinci/Scrapling into dev 2026-10-07 02:01:50 +03:00
Yiğit ERDOĞAN e88029347a fix(ai): strip every noise tag, not just the ones before a nested one (#474)
_strip_noise_tags dropped elements while iterating over them. When a
dropped tag held another noise tag (<noscript><style> in <head>,
<svg><style> in the body), lxml's iterator continued inside the detached
subtree and ended there, so every later script/style/noscript/svg
reached the html and Markdown output. Collect the matches first.
2026-10-07 01:33:45 +03:00
Karim shoair 1c3e355c9f perf(mcp)!: Combine selector and ref into one field called target
Save some tokens in the tools definition and make it simpler to use the server
2026-10-07 00:44:48 +03:00
Karim shoair 7372361683 perf(mcp): lower the tokens used by tools definition more. 2026-10-07 00:27:45 +03:00
Karim shoair d82f835d31 refactor(mcp): Move all AI related files to one submodule 2026-10-06 23:47:58 +03:00
Karim shoair 54c3a54360 perf(mcp): Optimize the tokens a bit more. 2026-10-06 21:31:31 +03:00
Karim shoair 84ddced8b2 refactor(mcp)!: Remove all bulk tools to lower tools definition tokens and more optimizations 2026-10-06 17:45:46 +03:00
Karim shoair 6f21ba42ce fix(mcp): raise detailed mcp errors on newer versions 2026-10-04 22:47:32 +03:00
Karim shoair f1cd65751e Merge branch 'dev' into v5-dev 2026-10-04 22:14:07 +03:00
Karim shoair 7c2ff106ab refactor(fetchers)!: Remove capture_xhr parameter while we have record_requests parameter for sessions 2026-10-04 22:01:16 +03:00
Karim shoair be1874eb9e feat(mcp): Add new tools to list and view browser background requests 2026-10-04 20:03:04 +03:00
Karim shoair 197e573f2d feat(browsers): catch background requests per session + improve encoding detection 2026-10-04 18:21:26 +03:00
Karim shoair 8366d0722d style(mcp): move browser actions to a separate file 2026-09-29 01:36:09 +03:00
Karim shoair 2a65a0b9da refactor(mcp)!: Make stealth browser the only browser used 2026-09-29 01:29:17 +03:00
Karim shoair 768e615df6 feat(mcp): Add a new tool to execute JS on page 2026-09-27 18:17:59 +03:00
Karim shoair f6f8d77493 feat(mcp): Make the screenshot tool can select an element instead of full page 2026-09-26 22:23:06 +03:00
Karim shoair 5bbc6842bb fix(mcp)!: Make the screenshot tool doesn't reload the page 2026-09-26 21:58:31 +03:00
Karim shoair 9e36038ca9 feat(mcp)!: Combine all four browser action tools into one chain-able tool 2026-09-26 21:44:45 +03:00
Yiğit ERDOĞANandKarim shoair 11e0935f6b fix(ai): match inline style values instead of substrings when sanitizing (#460)
Co-authored-by: Karim shoair <D4Vinci@users.noreply.github.com>
2026-09-26 21:09:50 +03:00
Karim shoair b68157f3f8 feat(mcp): Add a new tool to wait for several conditions (Chainable) 2026-09-26 21:04:25 +03:00
Karim shoair 69bdfd5335 refactor(mcp)!: Combine all mouse tools into one mouse actions tool to chain actions 2026-09-26 20:12:21 +03:00
Karim shoair 427f00ffea feat(mcp)!: Replace browser type with fields filling tool to save tokens 2026-09-26 19:07:00 +03:00
Karim shoair 6ed271d33f feat(mcp): add a tool to fill all kinds of inputs in a page 2026-09-26 18:30:20 +03:00
Karim shoair 3cc50aa9f3 feat(mcp): Add tool to chain keyboard keys 2026-09-26 17:19:46 +03:00
Karim shoair e512bc0bc4 feat(mcp): Add tool to control mouse wheel 2026-09-26 16:13:50 +03:00
Karim shoair 230ac0995e feat(mcp): Add tool to write text 2026-09-25 22:59:15 +03:00
Karim shoair a049dd70c8 feat(mcp): Add hover to the mouse movement tool 2026-09-25 02:57:38 +03:00
Karim shoair 7b6757ef24 feat(mcp): Add new tools to move and click with the mouse 2026-09-25 02:57:38 +03:00
Karim shoair a90d7dcf55 refactor(mcp)!: Use a clearer naming for the browser one-shot tools 2026-09-25 02:57:38 +03:00
Karim shoair 8fdbdeae02 fix(mcp): Make all MCP tools titles clearer 2026-09-25 02:57:38 +03:00
Karim shoair d9e153c671 fix(mcp): Shorten all tools' docstrings to consume less tokens 2026-09-25 02:57:38 +03:00
Karim shoair c8cc8977ff fix(mcp)!: Rename all browser based tools and move session validation inside session look up function 2026-09-25 02:57:38 +03:00
Karim shoair 31ba8e7986 feat(mcp): Add tool to take Aria Snapshot of the page for AI 2026-09-25 02:57:37 +03:00