758 Commits
Author SHA1 Message Date
Karim shoair 66b885d811 build: pump version up and deps 2026-11-14 19:24:39 +02:00
mschoi0411 63a040f601 fix(spiders): interpret asctime Retry-After dates as UTC (#561) 2026-11-14 18:27:06 +02:00
Karim shoair 193bd48776 fix(spider)!: make follow function doesn't inherit any request data from parent
Fixes #558 #587
Avoids the whole bug class.
Closes #559
2026-11-14 18:23:25 +02:00
SirTeggunandKarim shoair 822bb5f80c fix: deny file URL schema in LinkExtractor to prevent local file disclosure (#542)
Co-authored-by: Karim shoair <D4Vinci@users.noreply.github.com>
2026-11-14 16:24:31 +02:00
W fb26a7ab91 fix(shell): stop dropping cookies whose values SimpleCookie rejects (#541)
* fix(shell): stop dropping cookies whose values SimpleCookie rejects

_CookieParser fed the cookie string to http.cookies.SimpleCookie, which
stops at the first value it considers illegal (a space, a double quote,
...) and silently discards that cookie and everything after it. Cookie
strings copied from DevTools contain such values regularly, so curl
commands parsed by uncurl/curl2fetcher and `extract --cookies` lost
cookies without any warning.

Split the string on ";" instead and keep each value as sent, including
surrounding quotes, which SimpleCookie used to strip. Set-Cookie
attribute names are still skipped, as before.

* fix(shell): keep request cookies named partitioned
2026-11-04 16:08:55 +02:00
Jay Hemnani ead76eadde fix(shell): accept curl's -L flag (#554) 2026-11-04 16:04:36 +02:00
Jay Hemnani 7ac09b6519 fix(parser): return the value of XPath expressions that do not select nodes (#552)
string(), count(), boolean() and the like return one value, not a list of nodes.
xpath() walked it as a list: a string became one Selector per character and failed
with "Type 'str' cannot be serialized" on .getall(), and a number raised
"'float' object is not iterable".
2026-11-04 16:03:06 +02:00
Lykin 12c855d829 fix(spiders): handle CR record separators in CSV feeds (#551) 2026-11-04 15:56:27 +02:00
Wu Shuwen f0a54ac9b5 fix(spiders): skip malformed links during extraction (#548) 2026-11-03 22:21:08 +02:00
ZHANG YU fc6c7e62c7 fix(spiders): isolate temporary files for concurrent cache writes (#546) 2026-11-03 22:18:16 +02:00
kevin c2bd467ad7 fix(fetchers): decode percent-encoded credentials in proxy URLs (#545) 2026-11-03 22:07:15 +02:00
Karim shoair c83d10bdd1 feat(mcp): replace one mouse drag action with two mouse control actions
Making it able to solve all slider captchas, tested on all datadome and geetest CAPTCHAs I can find
2026-11-03 18:52:25 +02:00
Karim shoair a16d0cf8f0 fix(mcp): rewrite docstrings to save tokens 2026-11-03 18:52:25 +02:00
Karim shoair 37f448d689 feat(mcp): add browser observe tool to record the browser. 2026-11-03 18:52:25 +02:00
Nor.na 104294c2d8 fix(parser): honor zero similarity threshold without reference attributes (#539) 2026-11-02 22:16:22 +02:00
Rayen Mlayeh 49eaff11da fix(spiders): fetch robots.txt once per domain under concurrent requests (#538) 2026-11-02 22:07:57 +02:00
Karim shoair e3005a1d40 feat(spiders): add logging file encoding parameter
Better fix than #538
2026-11-02 21:50:48 +02:00
rushilnimbalker250 254b31322b fix(parser): handle empty HTML roots (#536) 2026-11-02 20:37:23 +02:00
cangen185-sudo 37950a6304 style(spiders): raise error on truncated and invalid gzip in templates (#522) 2026-11-02 20:34:22 +02:00
postoso d0639cddcb fix(cli): make --ai-targeted enable ad blocking on fetch (#527) 2026-10-31 18:11:16 +02:00
Karim shoair b02a20af68 feat(fetchers)!: move remote browsers sessions to separate locations with more control 2026-10-31 16:39:10 +02:00
Karim shoair 9ac748c52a build: pump version up and deps 2026-10-19 18:42:44 +03:00
Karim shoair fbc2465abc feat(mcp): add mouse drag-and-drop action 2026-10-19 18:42:44 +03:00
Karim shoair e60ea58610 fix(solver): fix detection of delayed Cloudflare Turnstile widgets
Fixes #509
2026-10-17 22:01:42 +03:00
Karim shoair c0634a0e90 fix(cf solver): avoid missing-widget waits and bound Cloudflare solver loops
Fixes #468
2026-10-17 17:55:24 +03:00
Karim shoair 1f02e3c89a fix(fetchers): preserve body precedence and None values in form encoding
A follow-up fix for #485
2026-10-17 16:50:51 +03:00
Suliman Abdulrazzaq 2dbeb48e38 fix(fetchers): encode list values and drop None values in dict data and params (#485) 2026-10-17 16:47:23 +03:00
Karim shoair 0449bd366e fix(fetcher): solve running browser from elevated shell on Windows
Fixes #463
2026-10-16 03:17:59 +03:00
Huang Qi 5e1383df3a fix: disambiguate generated selectors for first same-tag siblings (#495)
Count all siblings while retaining the target's same-tag position so
first siblings and ancestors receive a positional qualifier when needed.
Add 40 parameterized regression cases for CSS and XPath round trips.
2026-10-16 03:02:23 +03:00
Karim shoair e09f595e66 fix(parser)!: remove any left traces of keep_cdata parameter
The parts that didn't get covered in #493
2026-10-16 01:51:32 +03:00
SuyadiandKarim shoair 609f519e6b fix: remove no-op strip_cdata kwarg from lxml HTMLParser to silence DeprecationWarning (#493)
Co-authored-by: Karim shoair <D4Vinci@users.noreply.github.com>
2026-10-16 01:38:34 +03:00
Suliman Abdulrazzaq bd90176c66 fix(shell): apply curl -A and pass -x proxies as a URL string (#490) 2026-10-16 01:10:40 +03:00
Suliman Abdulrazzaq 71f567e806 fix(cli): write md and txt extract output as UTF-8 (#488) 2026-10-16 00:48:47 +03:00
Suliman Abdulrazzaq 71b57943fd fix(fetchers): let request headers override session headers regardless of case (#487) 2026-10-15 23:02:12 +03:00
Suliman Abdulrazzaq 15373320e6 fix(spiders): keep the checkpoint when a crawl ends with an exception (#486) 2026-10-15 20:36:33 +03:00
Suliman Abdulrazzaq b3d0dc70c1 fix(cli): keep falsy json bodies in extract commandline requests (#484) 2026-10-15 20:36:00 +03:00
Suliman Abdulrazzaq 353fc707b2 fix(spiders): let LinkExtractor restrict selectors match the link element itself (#483) 2026-10-15 20:35:16 +03:00
Karim shoair 0f27a77ac0 refactor(mcp): move tools descriptions to the definition itself to save tokens on definition 2026-10-15 20:34:58 +03:00
Karim shoair 3c3e3f95ff perf(mcp): Make session creating return shorter output 2026-10-14 20:42:26 +03:00
Karim shoair 7183891c31 feat(CrawlSpider): Add crawl strategy with values BFS or DFS 2026-10-13 01:39:11 +03:00
Karim shoair 9cf428038b refactor(rag): Renamed SiteToMarkdownSpider to ContentSpider 2026-10-13 01:08:54 +03:00
Karim shoair c54966a8c7 refactor(fetchers)!: Remove dynamic sessions and dynamic fetcher 2026-10-08 23:28:12 +03:00
Karim shoair 99a2281923 feat(mcp): Add new tools to set and load browser session state 2026-10-08 23:28:12 +03:00
Karim shoair d5b1995ce6 feat(fetchers): add functions to save browser session state and load it 2026-10-08 23:28:12 +03:00
Alex 7831c19eb3 fix(spiders): preserve falsy JSON bodies in request fingerprints (#480) 2026-10-08 02:10:10 +03:00
Yuval DinodiaandKarim shoair 1ac8e52fb1 fix: stop response.follow() inheriting the method, params and body (#462)
* fix: stop response.follow() inheriting the method, params and body

* fix: drop inherited Content-Length in response.follow()

---------

Co-authored-by: Karim shoair <D4Vinci@users.noreply.github.com>
2026-10-08 02:06:50 +03:00
Karim shoair 890d0c08c1 feat/refactor(mcp)!: Renamed browser_snapshot to browser_extract while adding more options
This gives the agent more control
2026-10-07 19:37:56 +03:00
Karim shoair 7d97d07276 feat(mcp): Add a dialog handling action 2026-10-07 02:25:18 +03:00
Karim shoair 5c39feb851 feat(mcp): Add snapshot searching 2026-10-07 02:10:19 +03:00
Karim shoair 2541e6b7fb Merge branch 'dev' of https://github.com/D4Vinci/Scrapling into dev 2026-10-07 02:01:50 +03:00