Karim shoair
193bd48776
fix(spider)!: make follow function doesn't inherit any request data from parent
...
Fixes #558 #587
Avoids the whole bug class.
Closes #559
2026-11-14 18:23:25 +02:00
Karim shoair
e3005a1d40
feat(spiders): add logging file encoding parameter
...
Better fix than #538
2026-11-02 21:50:48 +02:00
Karim shoair
d62e5b9854
docs(spiders): document checkpoint data requirements in all languages
...
Better workaround for #489
2026-10-17 17:06:30 +03:00
Karim shoair
c8568a4149
docs: update all pages to be clearer, fix mistakes, and improve for SEO
2026-10-15 02:52:19 +03:00
Karim shoair
7183891c31
feat(CrawlSpider): Add crawl strategy with values BFS or DFS
2026-10-13 01:39:11 +03:00
Karim shoair
9cf428038b
refactor(rag): Renamed SiteToMarkdownSpider to ContentSpider
2026-10-13 01:08:54 +03:00
Karim shoair
2228316b40
docs: update the docs accordingly with the latest breaking changes
2026-10-08 23:28:12 +03:00
Yuval Dinodia and Karim shoair
1ac8e52fb1
fix: stop response.follow() inheriting the method, params and body ( #462 )
...
* fix: stop response.follow() inheriting the method, params and body
* fix: drop inherited Content-Length in response.follow()
---------
Co-authored-by: Karim shoair <D4Vinci@users.noreply.github.com >
2026-10-08 02:06:50 +03:00
Karim shoair
e45311e2c1
docs!: update pages and skill to cover the recent changes
2026-10-04 22:11:08 +03:00
Karim shoair
6ca35d0448
feat(spiders): Adding easy interface for building RAG systems
2026-08-23 16:26:15 +03:00
Karim shoair
9bbd7c8b25
feat(spiders): Add XML/CSV feed templates + docs
2026-08-09 20:44:12 +03:00
Karim shoair
87e010a2e8
docs: fix
2026-07-27 12:38:38 +03:00
Karim shoair
0a7fec8f86
feat(spiders): Add exporters to CSV/XML
2026-07-26 23:13:35 +03:00
Karim shoair
347db4c76b
docs: add the auto throttle option
2026-07-26 23:04:28 +03:00
Karim shoair
b1c250fb29
docs: add a new page for the Shopify template
2026-07-12 18:45:18 +03:00
Karim shoair
6a641912cd
docs: add docs for the new features
2026-05-11 02:58:00 +03:00
Karim shoair
88d0459cd2
docs: adding the new development mode
2026-04-07 05:30:51 +02:00
Karim shoair
4c3ff2f0f1
docs: updating pages with the new feature
2026-04-05 05:27:07 +02:00
Karim shoair
a403156b2e
docs: style adjustment
2026-03-30 03:53:56 +02:00
Karim shoair
61cda587be
docs: update pages with the XHR feature
2026-03-29 23:12:56 +02:00
Karim shoair
7b8fc911e1
docs: add a page for spiders advanced usage
2026-02-15 02:40:55 +02:00
Karim shoair
7b4523af8e
docs: update the titles for all files
2026-02-15 02:35:39 +02:00
Karim shoair
77e825e7be
docs(spider): add a new page for proxy management & blocking
2026-02-15 01:01:43 +02:00
Karim shoair
c94ebd2fa9
docs(spiders): add a page to explain sessions logic
2026-02-14 17:19:55 +02:00
Karim shoair
a8f701f8ab
docs(spiders): add a new page for requests & responses
2026-02-14 04:04:18 +02:00
Karim shoair
1c2c7c55bf
docs: use a better looking architecture image instead of mermaid
2026-02-12 15:17:58 +02:00
Karim shoair
14143dd0f0
docs: add getting started page to the spiders section
2026-02-12 04:45:39 +02:00
Karim shoair
bbb8c26df0
docs: Improve spider architecture page
2026-02-12 04:30:16 +02:00
Karim shoair
d0b2cec849
docs: simplify the architecture mermaid
2026-02-12 03:30:14 +02:00
Karim shoair
10cd46ee23
docs: add the first page of the spiders section
2026-02-11 03:55:23 +02:00