W
fb26a7ab91
fix(shell): stop dropping cookies whose values SimpleCookie rejects ( #541 )
...
* fix(shell): stop dropping cookies whose values SimpleCookie rejects
_CookieParser fed the cookie string to http.cookies.SimpleCookie, which
stops at the first value it considers illegal (a space, a double quote,
...) and silently discards that cookie and everything after it. Cookie
strings copied from DevTools contain such values regularly, so curl
commands parsed by uncurl/curl2fetcher and `extract --cookies` lost
cookies without any warning.
Split the string on ";" instead and keep each value as sent, including
surrounding quotes, which SimpleCookie used to strip. Set-Cookie
attribute names are still skipped, as before.
* fix(shell): keep request cookies named partitioned
2026-11-04 16:08:55 +02:00
Jay Hemnani
ead76eadde
fix(shell): accept curl's -L flag ( #554 )
2026-11-04 16:04:36 +02:00
Karim shoair
c83d10bdd1
feat(mcp): replace one mouse drag action with two mouse control actions
...
Making it able to solve all slider captchas, tested on all datadome and geetest CAPTCHAs I can find
2026-11-03 18:52:25 +02:00
Karim shoair
a16d0cf8f0
fix(mcp): rewrite docstrings to save tokens
2026-11-03 18:52:25 +02:00
Karim shoair
37f448d689
feat(mcp): add browser observe tool to record the browser.
2026-11-03 18:52:25 +02:00
Karim shoair
b02a20af68
feat(fetchers)!: move remote browsers sessions to separate locations with more control
2026-10-31 16:39:10 +02:00
Karim shoair
fbc2465abc
feat(mcp): add mouse drag-and-drop action
2026-10-19 18:42:44 +03:00
Huang Qi
5e1383df3a
fix: disambiguate generated selectors for first same-tag siblings ( #495 )
...
Count all siblings while retaining the target's same-tag position so
first siblings and ancestors receive a positional qualifier when needed.
Add 40 parameterized regression cases for CSS and XPath round trips.
2026-10-16 03:02:23 +03:00
Suliman Abdulrazzaq
bd90176c66
fix(shell): apply curl -A and pass -x proxies as a URL string ( #490 )
2026-10-16 01:10:40 +03:00
Suliman Abdulrazzaq
71f567e806
fix(cli): write md and txt extract output as UTF-8 ( #488 )
2026-10-16 00:48:47 +03:00
Karim shoair
0f27a77ac0
refactor(mcp): move tools descriptions to the definition itself to save tokens on definition
2026-10-15 20:34:58 +03:00
Karim shoair
3c3e3f95ff
perf(mcp): Make session creating return shorter output
2026-10-14 20:42:26 +03:00
Karim shoair
c54966a8c7
refactor(fetchers)!: Remove dynamic sessions and dynamic fetcher
2026-10-08 23:28:12 +03:00
Karim shoair
99a2281923
feat(mcp): Add new tools to set and load browser session state
2026-10-08 23:28:12 +03:00
Karim shoair
890d0c08c1
feat/refactor(mcp)!: Renamed browser_snapshot to browser_extract while adding more options
...
This gives the agent more control
2026-10-07 19:37:56 +03:00
Karim shoair
7d97d07276
feat(mcp): Add a dialog handling action
2026-10-07 02:25:18 +03:00
Karim shoair
5c39feb851
feat(mcp): Add snapshot searching
2026-10-07 02:10:19 +03:00
Karim shoair
2541e6b7fb
Merge branch 'dev' of https://github.com/D4Vinci/Scrapling into dev
2026-10-07 02:01:50 +03:00
Yiğit ERDOĞAN
e88029347a
fix(ai): strip every noise tag, not just the ones before a nested one ( #474 )
...
_strip_noise_tags dropped elements while iterating over them. When a
dropped tag held another noise tag (<noscript><style> in <head>,
<svg><style> in the body), lxml's iterator continued inside the detached
subtree and ended there, so every later script/style/noscript/svg
reached the html and Markdown output. Collect the matches first.
2026-10-07 01:33:45 +03:00
Karim shoair
1c3e355c9f
perf(mcp)!: Combine selector and ref into one field called target
...
Save some tokens in the tools definition and make it simpler to use the server
2026-10-07 00:44:48 +03:00
Karim shoair
7372361683
perf(mcp): lower the tokens used by tools definition more.
2026-10-07 00:27:45 +03:00
Karim shoair
d82f835d31
refactor(mcp): Move all AI related files to one submodule
2026-10-06 23:47:58 +03:00
Karim shoair
54c3a54360
perf(mcp): Optimize the tokens a bit more.
2026-10-06 21:31:31 +03:00
Karim shoair
84ddced8b2
refactor(mcp)!: Remove all bulk tools to lower tools definition tokens and more optimizations
2026-10-06 17:45:46 +03:00
Karim shoair
6f21ba42ce
fix(mcp): raise detailed mcp errors on newer versions
2026-10-04 22:47:32 +03:00
Karim shoair
f1cd65751e
Merge branch 'dev' into v5-dev
2026-10-04 22:14:07 +03:00
Karim shoair
7c2ff106ab
refactor(fetchers)!: Remove capture_xhr parameter while we have record_requests parameter for sessions
2026-10-04 22:01:16 +03:00
Karim shoair
be1874eb9e
feat(mcp): Add new tools to list and view browser background requests
2026-10-04 20:03:04 +03:00
Karim shoair
197e573f2d
feat(browsers): catch background requests per session + improve encoding detection
2026-10-04 18:21:26 +03:00
Karim shoair
8366d0722d
style(mcp): move browser actions to a separate file
2026-09-29 01:36:09 +03:00
Karim shoair
2a65a0b9da
refactor(mcp)!: Make stealth browser the only browser used
2026-09-29 01:29:17 +03:00
Karim shoair
768e615df6
feat(mcp): Add a new tool to execute JS on page
2026-09-27 18:17:59 +03:00
Karim shoair
f6f8d77493
feat(mcp): Make the screenshot tool can select an element instead of full page
2026-09-26 22:23:06 +03:00
Karim shoair
5bbc6842bb
fix(mcp)!: Make the screenshot tool doesn't reload the page
2026-09-26 21:58:31 +03:00
Karim shoair
9e36038ca9
feat(mcp)!: Combine all four browser action tools into one chain-able tool
2026-09-26 21:44:45 +03:00
Yiğit ERDOĞAN and Karim shoair
11e0935f6b
fix(ai): match inline style values instead of substrings when sanitizing ( #460 )
...
Co-authored-by: Karim shoair <D4Vinci@users.noreply.github.com >
2026-09-26 21:09:50 +03:00
Karim shoair
b68157f3f8
feat(mcp): Add a new tool to wait for several conditions (Chainable)
2026-09-26 21:04:25 +03:00
Karim shoair
69bdfd5335
refactor(mcp)!: Combine all mouse tools into one mouse actions tool to chain actions
2026-09-26 20:12:21 +03:00
Karim shoair
427f00ffea
feat(mcp)!: Replace browser type with fields filling tool to save tokens
2026-09-26 19:07:00 +03:00
Karim shoair
6ed271d33f
feat(mcp): add a tool to fill all kinds of inputs in a page
2026-09-26 18:30:20 +03:00
Karim shoair
3cc50aa9f3
feat(mcp): Add tool to chain keyboard keys
2026-09-26 17:19:46 +03:00
Karim shoair
e512bc0bc4
feat(mcp): Add tool to control mouse wheel
2026-09-26 16:13:50 +03:00
Karim shoair
230ac0995e
feat(mcp): Add tool to write text
2026-09-25 22:59:15 +03:00
Karim shoair
a049dd70c8
feat(mcp): Add hover to the mouse movement tool
2026-09-25 02:57:38 +03:00
Karim shoair
7b6757ef24
feat(mcp): Add new tools to move and click with the mouse
2026-09-25 02:57:38 +03:00
Karim shoair
a90d7dcf55
refactor(mcp)!: Use a clearer naming for the browser one-shot tools
2026-09-25 02:57:38 +03:00
Karim shoair
8fdbdeae02
fix(mcp): Make all MCP tools titles clearer
2026-09-25 02:57:38 +03:00
Karim shoair
d9e153c671
fix(mcp): Shorten all tools' docstrings to consume less tokens
2026-09-25 02:57:38 +03:00
Karim shoair
c8cc8977ff
fix(mcp)!: Rename all browser based tools and move session validation inside session look up function
2026-09-25 02:57:38 +03:00
Karim shoair
31ba8e7986
feat(mcp): Add tool to take Aria Snapshot of the page for AI
2026-09-25 02:57:37 +03:00