mschoi0411
63a040f601
fix(spiders): interpret asctime Retry-After dates as UTC ( #561 )
2026-11-14 18:27:06 +02:00
Karim shoair
193bd48776
fix(spider)!: make follow function doesn't inherit any request data from parent
...
Fixes #558 #587
Avoids the whole bug class.
Closes #559
2026-11-14 18:23:25 +02:00
SirTeggun and Karim shoair
822bb5f80c
fix: deny file URL schema in LinkExtractor to prevent local file disclosure ( #542 )
...
Co-authored-by: Karim shoair <D4Vinci@users.noreply.github.com >
2026-11-14 16:24:31 +02:00
Karim shoair
28bdfbb2a1
test: fix repeated issue on GitHub Actions
2026-11-04 18:06:10 +02:00
W
fb26a7ab91
fix(shell): stop dropping cookies whose values SimpleCookie rejects ( #541 )
...
* fix(shell): stop dropping cookies whose values SimpleCookie rejects
_CookieParser fed the cookie string to http.cookies.SimpleCookie, which
stops at the first value it considers illegal (a space, a double quote,
...) and silently discards that cookie and everything after it. Cookie
strings copied from DevTools contain such values regularly, so curl
commands parsed by uncurl/curl2fetcher and `extract --cookies` lost
cookies without any warning.
Split the string on ";" instead and keep each value as sent, including
surrounding quotes, which SimpleCookie used to strip. Set-Cookie
attribute names are still skipped, as before.
* fix(shell): keep request cookies named partitioned
2026-11-04 16:08:55 +02:00
Jay Hemnani
ead76eadde
fix(shell): accept curl's -L flag ( #554 )
2026-11-04 16:04:36 +02:00
Jay Hemnani
7ac09b6519
fix(parser): return the value of XPath expressions that do not select nodes ( #552 )
...
string(), count(), boolean() and the like return one value, not a list of nodes.
xpath() walked it as a list: a string became one Selector per character and failed
with "Type 'str' cannot be serialized" on .getall(), and a number raised
"'float' object is not iterable".
2026-11-04 16:03:06 +02:00
Lykin
12c855d829
fix(spiders): handle CR record separators in CSV feeds ( #551 )
2026-11-04 15:56:27 +02:00
Wu Shuwen
f0a54ac9b5
fix(spiders): skip malformed links during extraction ( #548 )
2026-11-03 22:21:08 +02:00
ZHANG YU
fc6c7e62c7
fix(spiders): isolate temporary files for concurrent cache writes ( #546 )
2026-11-03 22:18:16 +02:00
kevin
c2bd467ad7
fix(fetchers): decode percent-encoded credentials in proxy URLs ( #545 )
2026-11-03 22:07:15 +02:00
Karim shoair
f35fa0b93c
test: update tests with recent changes
2026-11-03 18:52:25 +02:00
Karim shoair
22318c34a6
test: add tests for the latest changes
2026-11-03 18:52:25 +02:00
Nor.na
104294c2d8
fix(parser): honor zero similarity threshold without reference attributes ( #539 )
2026-11-02 22:16:22 +02:00
Rayen Mlayeh
49eaff11da
fix(spiders): fetch robots.txt once per domain under concurrent requests ( #538 )
2026-11-02 22:07:57 +02:00
rushilnimbalker250
254b31322b
fix(parser): handle empty HTML roots ( #536 )
2026-11-02 20:37:23 +02:00
cangen185-sudo
37950a6304
style(spiders): raise error on truncated and invalid gzip in templates ( #522 )
2026-11-02 20:34:22 +02:00
Maulana
41d88c0696
test: fix flaky Cloudflare readiness test ( #515 )
2026-11-02 20:29:17 +02:00
postoso
d0639cddcb
fix(cli): make --ai-targeted enable ad blocking on fetch ( #527 )
2026-10-31 18:11:16 +02:00
Karim shoair
acc03fcee5
test: update to the current changes
2026-10-31 16:50:39 +02:00
Karim shoair
fbc2465abc
feat(mcp): add mouse drag-and-drop action
2026-10-19 18:42:44 +03:00
Karim shoair
e60ea58610
fix(solver): fix detection of delayed Cloudflare Turnstile widgets
...
Fixes #509
2026-10-17 22:01:42 +03:00
Karim shoair
c0634a0e90
fix(cf solver): avoid missing-widget waits and bound Cloudflare solver loops
...
Fixes #468
2026-10-17 17:55:24 +03:00
Karim shoair
1f02e3c89a
fix(fetchers): preserve body precedence and None values in form encoding
...
A follow-up fix for #485
2026-10-17 16:50:51 +03:00
Suliman Abdulrazzaq
2dbeb48e38
fix(fetchers): encode list values and drop None values in dict data and params ( #485 )
2026-10-17 16:47:23 +03:00
Huang Qi
5e1383df3a
fix: disambiguate generated selectors for first same-tag siblings ( #495 )
...
Count all siblings while retaining the target's same-tag position so
first siblings and ancestors receive a positional qualifier when needed.
Add 40 parameterized regression cases for CSS and XPath round trips.
2026-10-16 03:02:23 +03:00
Karim shoair
e09f595e66
fix(parser)!: remove any left traces of keep_cdata parameter
...
The parts that didn't get covered in #493
2026-10-16 01:51:32 +03:00
Suliman Abdulrazzaq
bd90176c66
fix(shell): apply curl -A and pass -x proxies as a URL string ( #490 )
2026-10-16 01:10:40 +03:00
Suliman Abdulrazzaq
71f567e806
fix(cli): write md and txt extract output as UTF-8 ( #488 )
2026-10-16 00:48:47 +03:00
Suliman Abdulrazzaq
71b57943fd
fix(fetchers): let request headers override session headers regardless of case ( #487 )
2026-10-15 23:02:12 +03:00
Suliman Abdulrazzaq
15373320e6
fix(spiders): keep the checkpoint when a crawl ends with an exception ( #486 )
2026-10-15 20:36:33 +03:00
Suliman Abdulrazzaq
b3d0dc70c1
fix(cli): keep falsy json bodies in extract commandline requests ( #484 )
2026-10-15 20:36:00 +03:00
Suliman Abdulrazzaq
353fc707b2
fix(spiders): let LinkExtractor restrict selectors match the link element itself ( #483 )
2026-10-15 20:35:16 +03:00
Karim shoair
0f27a77ac0
refactor(mcp): move tools descriptions to the definition itself to save tokens on definition
2026-10-15 20:34:58 +03:00
Karim shoair
1a9a8f4b18
test: fixes for the mcp errors
2026-10-15 02:52:39 +03:00
Karim shoair
3c3e3f95ff
perf(mcp): Make session creating return shorter output
2026-10-14 20:42:26 +03:00
Karim shoair
7183891c31
feat(CrawlSpider): Add crawl strategy with values BFS or DFS
2026-10-13 01:39:11 +03:00
Karim shoair
9cf428038b
refactor(rag): Renamed SiteToMarkdownSpider to ContentSpider
2026-10-13 01:08:54 +03:00
Karim shoair
335beb932a
test: update tests accordingly
2026-10-08 23:28:12 +03:00
Karim shoair
99a2281923
feat(mcp): Add new tools to set and load browser session state
2026-10-08 23:28:12 +03:00
Karim shoair
d5b1995ce6
feat(fetchers): add functions to save browser session state and load it
2026-10-08 23:28:12 +03:00
Alex
7831c19eb3
fix(spiders): preserve falsy JSON bodies in request fingerprints ( #480 )
2026-10-08 02:10:10 +03:00
Yuval Dinodia and Karim shoair
1ac8e52fb1
fix: stop response.follow() inheriting the method, params and body ( #462 )
...
* fix: stop response.follow() inheriting the method, params and body
* fix: drop inherited Content-Length in response.follow()
---------
Co-authored-by: Karim shoair <D4Vinci@users.noreply.github.com >
2026-10-08 02:06:50 +03:00
Karim shoair
890d0c08c1
feat/refactor(mcp)!: Renamed browser_snapshot to browser_extract while adding more options
...
This gives the agent more control
2026-10-07 19:37:56 +03:00
Karim shoair
7d97d07276
feat(mcp): Add a dialog handling action
2026-10-07 02:25:18 +03:00
Karim shoair
5c39feb851
feat(mcp): Add snapshot searching
2026-10-07 02:10:19 +03:00
Karim shoair
2541e6b7fb
Merge branch 'dev' of https://github.com/D4Vinci/Scrapling into dev
2026-10-07 02:01:50 +03:00
Yiğit ERDOĞAN
06ae92f17f
fix(spiders): pass stealthy_headers to the session instead of curl_cffi ( #475 )
...
The public FetcherSession methods pop `stealthy_headers` and hand it to
`_make_request` as `stealth`. SessionManager.fetch forwarded the request
kwargs untouched, so `Request(url, stealthy_headers=False)` reached
curl_cffi as an unknown argument, and the request failed with a TypeError
that the default `on_error` swallows. Map it the same way here.
2026-10-07 01:37:09 +03:00
Yiğit ERDOĞAN
e88029347a
fix(ai): strip every noise tag, not just the ones before a nested one ( #474 )
...
_strip_noise_tags dropped elements while iterating over them. When a
dropped tag held another noise tag (<noscript><style> in <head>,
<svg><style> in the body), lxml's iterator continued inside the detached
subtree and ended there, so every later script/style/noscript/svg
reached the html and Markdown output. Collect the matches first.
2026-10-07 01:33:45 +03:00
Sarwadnya Mahajan
4b97bfe4a3
fix: make request fingerprint independent of header order ( #466 )
...
- Sort processed headers tuple when generating request fingerprint
- Add a regression test in tests/spiders/test_request.py
- Fixes #465
2026-10-07 01:25:32 +03:00