30 Commits
Author SHA1 Message Date
Karim shoair 193bd48776 fix(spider)!: make follow function doesn't inherit any request data from parent
Fixes #558 #587
Avoids the whole bug class.
Closes #559
2026-11-14 18:23:25 +02:00
Karim shoair e3005a1d40 feat(spiders): add logging file encoding parameter
Better fix than #538
2026-11-02 21:50:48 +02:00
Karim shoair d62e5b9854 docs(spiders): document checkpoint data requirements in all languages
Better workaround for #489
2026-10-17 17:06:30 +03:00
Karim shoair c8568a4149 docs: update all pages to be clearer, fix mistakes, and improve for SEO 2026-10-15 02:52:19 +03:00
Karim shoair 7183891c31 feat(CrawlSpider): Add crawl strategy with values BFS or DFS 2026-10-13 01:39:11 +03:00
Karim shoair 9cf428038b refactor(rag): Renamed SiteToMarkdownSpider to ContentSpider 2026-10-13 01:08:54 +03:00
Karim shoair 2228316b40 docs: update the docs accordingly with the latest breaking changes 2026-10-08 23:28:12 +03:00
Yuval DinodiaandKarim shoair 1ac8e52fb1 fix: stop response.follow() inheriting the method, params and body (#462)
* fix: stop response.follow() inheriting the method, params and body

* fix: drop inherited Content-Length in response.follow()

---------

Co-authored-by: Karim shoair <D4Vinci@users.noreply.github.com>
2026-10-08 02:06:50 +03:00
Karim shoair e45311e2c1 docs!: update pages and skill to cover the recent changes 2026-10-04 22:11:08 +03:00
Karim shoair 6ca35d0448 feat(spiders): Adding easy interface for building RAG systems 2026-08-23 16:26:15 +03:00
Karim shoair 9bbd7c8b25 feat(spiders): Add XML/CSV feed templates + docs 2026-08-09 20:44:12 +03:00
Karim shoair 87e010a2e8 docs: fix 2026-07-27 12:38:38 +03:00
Karim shoair 0a7fec8f86 feat(spiders): Add exporters to CSV/XML 2026-07-26 23:13:35 +03:00
Karim shoair 347db4c76b docs: add the auto throttle option 2026-07-26 23:04:28 +03:00
Karim shoair b1c250fb29 docs: add a new page for the Shopify template 2026-07-12 18:45:18 +03:00
Karim shoair 6a641912cd docs: add docs for the new features 2026-05-11 02:58:00 +03:00
Karim shoair 88d0459cd2 docs: adding the new development mode 2026-04-07 05:30:51 +02:00
Karim shoair 4c3ff2f0f1 docs: updating pages with the new feature 2026-04-05 05:27:07 +02:00
Karim shoair a403156b2e docs: style adjustment 2026-03-30 03:53:56 +02:00
Karim shoair 61cda587be docs: update pages with the XHR feature 2026-03-29 23:12:56 +02:00
Karim shoair 7b8fc911e1 docs: add a page for spiders advanced usage 2026-02-15 02:40:55 +02:00
Karim shoair 7b4523af8e docs: update the titles for all files 2026-02-15 02:35:39 +02:00
Karim shoair 77e825e7be docs(spider): add a new page for proxy management & blocking 2026-02-15 01:01:43 +02:00
Karim shoair c94ebd2fa9 docs(spiders): add a page to explain sessions logic 2026-02-14 17:19:55 +02:00
Karim shoair a8f701f8ab docs(spiders): add a new page for requests & responses 2026-02-14 04:04:18 +02:00
Karim shoair 1c2c7c55bf docs: use a better looking architecture image instead of mermaid 2026-02-12 15:17:58 +02:00
Karim shoair 14143dd0f0 docs: add getting started page to the spiders section 2026-02-12 04:45:39 +02:00
Karim shoair bbb8c26df0 docs: Improve spider architecture page 2026-02-12 04:30:16 +02:00
Karim shoair d0b2cec849 docs: simplify the architecture mermaid 2026-02-12 03:30:14 +02:00
Karim shoair 10cd46ee23 docs: add the first page of the spiders section 2026-02-11 03:55:23 +02:00