6 Commits
Author SHA1 Message Date
Kuang-xianxinandKuang-xianxin ce294063e9 fix(spiders): cache responses only after blocked-response checks (#446)
* fix(spiders): avoid replaying blocked responses from development cache

* fix(spiders): leave development cache replay unchanged

---------

Co-authored-by: Kuang-xianxin <243476082+Kuang-xianxin@users.noreply.github.com>
2026-09-18 12:35:22 +03:00
Yiğit ERDOĞAN 63cdc99cfd fix(spiders): keep the request meta on cached responses (#419) 2026-08-23 21:58:21 +03:00
Karim shoair ca51962421 feat(spiders): Auto throttle based on website speed/protection 2026-07-26 22:54:36 +03:00
Amit Vijapur 3ed4fde93a fix(spiders): preserve cookies for browser-engine responses in cache (#376)
ResponseCacheManager.put() only recognized the flat `dict` shape that
static-engine responses use for `Response.cookies`. Browser engines
(Playwright) populate cookies as a `tuple` of full cookie dicts, so the
isinstance(dict) guard fell through to the `else {}` branch and silently
discarded every cookie before it ever reached the cache file. Replaying a
cached browser-engine response then rebuilt it with cookies={}.

Preserve whichever shape the cookies are in instead of collapsing anything
non-dict to {}: serialize a tuple as a JSON array and a dict as a JSON
object, then restore the tuple shape on read since JSON arrays deserialize
back as `list`.

Add regression tests covering both the browser-engine (tuple) and
static-engine (dict) cookie round-trip through put()/get().
2026-07-21 00:38:48 +08:00
Ahmed Elshahat f0db3d7d14 fix(spiders): use os.replace for atomic checkpoint/cache writes on Windows
CheckpointManager.save() and ResponseCacheManager.put() write to a temp
file and then move it into place with Path.rename(). On Windows, os.rename
cannot overwrite an existing destination and raises FileExistsError
(WinError 183), so every write after the first one fails: checkpoint
saving raises and breaks resume, while the development response cache
swallows the error and keeps returning the stale entry.

Path.replace() (os.replace) overwrites the destination atomically on every
platform and behaves identically to rename() on POSIX, so this is a no-op
on Linux and macOS and only fixes the broken overwrite on Windows.

Add a regression test for the cache overwrite path; the checkpoint
overwrite is already covered by test_multiple_saves_overwrite.
2026-06-07 04:24:12 +03:00
Karim shoair d1baf1fc46 feat(spiders): add a development mode 2026-04-07 04:08:54 +02:00