string(), count(), boolean() and the like return one value, not a list of nodes.
xpath() walked it as a list: a string became one Selector per character and failed
with "Type 'str' cannot be serialized" on .getall(), and a number raised
"'float' object is not iterable".
Count all siblings while retaining the target's same-tag position so
first siblings and ancestors receive a positional qualifier when needed.
Add 40 parameterized regression cases for CSS and XPath round trips.
Candidates with fewer attributes than the original got inflated scores because the denominator counted candidate attributes only, while the extra-attributes penalty direction worked as intended. Using `max()` on both counts fixes the inflation while keeping the penalty.
Closes#322
When `css()`/`xpath()` are called with both `adaptive=True` and
`auto_save=True`, the relocation branch guarded the re-save with
`if elements is not None`. However `relocate()` returns an empty
list (never `None`) when no candidate clears the `percentage`
threshold, so the guard always passed and `self.save(elements[0], ...)`
raised `IndexError: list index out of range`.
This crashes exactly when adaptive resilience is needed most: the page
structure changed enough that nothing matches above the threshold.
Fix: use a truthiness check (`if elements and auto_save`) so the
re-save is skipped when relocation yields nothing. The successful
relocation path (which re-saves the relocated element) is unchanged.
Added a regression test that fails before the fix (IndexError) and
passes after.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address review feedback: In full-path XPath generation, elements with
IDs were producing bare predicates like `[@id='x']` which creates
invalid XPath steps like `//body/[@id='main']`. Now emits `*[@id='x']`
for full-path mode (e.g. `//body/*[@id='main']/*[@id='target']`).
Short-path XPath mode unchanged — still uses `//*[@id='x']` prefix.
Also added XPath evaluation assertion to the regression test to verify
the generated selector actually selects the correct element.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When generating full-path CSS/XPath selectors, elements with id
attributes had their selector appended twice — once in the id branch
(line 30) and again unconditionally (line 50).
This produced selectors like 'body > #main > #main > #target > #target'
instead of the correct 'body > #main > #target'.
Move the append into the else branch so it only fires for elements
without an id (elements with id already append in the if branch).
Includes 2 regression tests.
Several critical code paths in custom_types.py lacked test coverage:
- TextHandler.re(check_match=True): returns bool, not TextHandlers
- TextHandler.re(replace_entities=False): entity preservation path
- TextHandler.re() with capture groups: flatten behavior
- TextHandler.re_first() default value when no match
- TextHandler.clean(remove_entities=True): entity replacement path
- TextHandler.json() valid and invalid input
- TextHandlers.re(): list-level regex with result flattening
- TextHandlers.extract()/get_all(): identity return
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove all tests for the old encoding logic
- Stop using the `nopecha` test page to test Cloudflare solver
- Remove useless tests like testing for infinite timeout
- Fixes to make the code compatible with new changes
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
- Renamed both `css_selector` and `xpath_selector` both to `generate_css_selector` and `generate_xpath_selector` for clarity and not interrupting the auto-completion while coding.
- Added `generate_full_css_selector` and `generate_full_xpath_selector` methods