Commit Graph
211 Commits
Author SHA1 Message Date
Karl Seguin 90302b3c1a fix another slow CI failure 2026-11-14 11:39:59 +08:00
Adrià Arrufat 89d82c586b tools: rename jev.zig to decisions.zig
The module serves either decision model now, so it takes the name of
the zenai layer it wraps; MockJev becomes MockDecisions to match.
2026-11-11 21:50:25 +01:00
Adrià Arrufat 21c8c8e037 tools: ask Gemini Decisions or TypeSafe Jev
classify, acceptCookies and login now go through zenai.decisions, which
asks whichever decision model has a key: Gemini's Decisions API
(GOOGLE_API_KEY/GEMINI_API_KEY) or TypeSafe System One
(TYPESAFE_API_KEY). Gemini is preferred when both are set;
LIGHTPANDA_DECISION_PROVIDER=gemini|typesafe picks one.

Messages, tool descriptions and the REPL's no-key fallback name both
keys. The test mock serves Gemini's endpoint too, every mock test names
its provider so a developer's own Gemini key never reaches Google, and
the acceptCookies and login happy paths run against both.
2026-11-11 21:48:54 +01:00
Karl Seguin ee91afe861 Merge pull request #4363 from lightpanda-io/speed-up-slow-tests
test: speed up two slow tests
2026-11-02 23:42:40 +08:00
Karl Seguin a9037dd921 test: speed up two slow tests
- Use a signal (atomic bool) to signal load no longer  needs to be held
- check for completion more frequently
2026-11-02 10:54:02 +08:00
Karl Seguin 64c44720ca chore: remove dead code, remove 'pub' where it isn't needed 2026-11-02 10:11:47 +08:00
Adrià Arrufat 3a0be0a59c browser: ask Jev from a helper thread, pumping the page meanwhile
`askPage` called the pool inline, so the browser thread sat in a blocking
HTTPS read for the length of the answer — 200ms on a good connection, up to
the 10s request timeout. `search` has wrapped its own HTTP in
`Session.runPumped` from the start, for the reason its comment gives, and
this is the same read.

Thread-per-session in both servers (`server/Server.zig:593`,
`mcp/HttpServer.zig:299`) means no other session was ever affected. What
stalled was the calling session's own page: no Runner tick, so its timers and
queued work stop and the curl multi driving its transfers goes unpolled,
while `CURLOPT_TIMEOUT_MS` (15s) keeps counting against them in wall-clock.
That is the shape of the MCP idle-transfer bug and of the agent's blocking
model call, both already fixed this way. The watchdog is not in play at its
30s default, but `--watchdog_ms` under 10s would be.

`std.Io.Mutex` under `Io.Threaded` is a futex mutex that blocks the calling
OS thread rather than parking a task, so the pool is safe to take from a
helper thread and zenai needs no change; `init_single_threaded` constrains
its task pool, not other threads. zenai already drives `lp.io` timers from a
thread it spawns per request, and `buildState` stays on the browser thread,
so only the HTTP leaves it.

Pumping lets the page's own scripts run during the ask, which a blocked
thread never allowed. A navigation among them swaps the root frame and tears
down the Page that the candidate elements and node ids point into, so the ask
now reports that rather than handing the caller an answer about a page that
has gone — the `late_redirect.html` test navigates mid-ask and reaches it.
Detached-but-not-freed nodes within one document stay readable, and a
selector built from one simply fails to resolve, which every caller already
handles.

The mock can now answer slowly, which is what makes the stall observable: a
20ms interval on the page advances across a 400ms answer (a turn per 250ms
quiet slice) where blocking gave it none at all.
2026-10-30 11:14:54 +01:00
Adrià Arrufat 78428e5ce1 browser: close what the review found in the login and consent tools
The credential guards ran on the first decision only. An identifier-first
login submits, asks again on the page that comes back, and then filled the
password into whatever that second answer named — a plain `input[type=text]`,
or a form posting to `http://`. They are one predicate now, asked of both
answers. `replayLogin` already checked a selector a caller handed it, which
is what made the gap visible.

A recorded two-step login could not replay at all: `--save` writes both
selectors into one `login(...)` call, and the password's field is not on the
first page, so resolving it returned `NodeNotFound` and the tool failed
outright. It repeats the submit that raised the field now, as the decision
did. A replay given only `usernameSelector` says so, rather than reporting
"filled the fields given" for a password it never typed.

`choose.ranked` measured candidates against `none` alone, so an `extra`
option winning the argmax left the elements below it in play: a login whose
answer was "press Enter" could click an in-form "Create an account" instead,
since the submit control is only checked for sharing the field's form. An
answer that is not a candidate drops them all, which is what its doc comment
already promised.

A verification ask that failed threw away the outcome it was verifying. Both
the consent loop and `verifyLogin` returned the raw API failure, so a 500
after `login` had submitted real credentials told the caller "ApiError" and
nothing about the submit, the selectors or the replay arguments.

`classify` was annotated `read_only`, which claims `openWorldHint = false`
and `idempotentHint = true` for a paid call to a third-party model that
answers differently twice. `search`, the other external-API tool, already
leaves `openWorldHint` alone.

The two-step fixtures need a mock that decides there is a later step, so it
reads that off the one thing it can see — a request that offered no field
looking like a password — and fails the verification re-read for a key a test
sets. One doc comment the marshal move had split in two is whole again.
2026-10-30 11:14:54 +01:00
Adrià Arrufat 0e9e36b5df browser: move the argument marshalling into tools/marshal.zig
From the JSON a caller sent to the struct a tool reads: the schema's casing,
the `$LP_*` placeholders, and what a malformed call is told. Five of these
were already public and called from `mcp/tools.zig`, `agent/Agent.zig` and
`script/Recorder.zig` — they lived in tools.zig because the tools were their
first caller, not because they belong to it. The names callers use are
re-exported, so nothing outside changes.

`marshal`, not `args`: Zig refuses the import alias the basename would give,
because `const args = try parseArgs(...)` is the first line of nearly every
`exec*`. The repo already calls this marshalling (`Runtime.marshalArgs`).

`lpEnvNames` and `lookupLpEnv` go with it. They are the `LP_` namespace, read
by the substitution as much as by the `getEnv` tool, and the substitution is
here now.

One duplicate gone on the way: the `setenv`/`unsetenv` externs are declared
once, in the test module both files already import, instead of a copy per
file.
2026-10-30 11:14:54 +01:00
Adrià Arrufat b97f5c592c browser: take the Jev plumbing from zenai
The question DSL and the client pool had no browser knowledge in them, so
they are zenai's now (lightpanda-io/zenai#44).

classify.zig keeps what is actually the browser's: the page presets a caller
can name instead of wording a question, the wording of the category question,
and what an empty questions object means. It goes from 454 lines to 110, and
its question-shape tests move to zenai with the code.

App keeps a zenai.typesafe.Pool rather than its own idle list. The pool is
handed lp.environ rather than a snapshot of it, because setenv can move the
block an Environ points into — and because it retires a client whose
credentials the environment no longer agrees with, the test scaffold no
longer has to drop idle clients around every Jev test.
2026-10-30 11:14:54 +01:00
Adrià Arrufat c2eb9f2792 browser: apply the simplify pass
The mock answers through std.json.Stringify instead of hand-built JSON with
comma flags, keys come from the constants the real code exports rather than
copies of them, and one enum says which of the three request shapes it got
instead of two flags and an inert count guard.

The REPL's two prompt paths share one helper; fromToolCall gets its doc
comment back, which an insert had handed to its neighbour; four identical
error switches become one function; LineCollector drops an allocator field
its tree already carried; and the TypeSafe test preamble, written four times,
is one scaffold beside the test server that owns the port it needs.
2026-10-30 11:14:54 +01:00
Adrià Arrufat 97b43872f6 browser: use zenai's ErrorDetail for API tool failures
The search tool kept its own copy of zenai.http.ErrorDetail and duped
the message by hand; ErrorDetail.clone does that. The classify tests
take the environment variable names from zenai.
2026-10-30 11:14:49 +01:00
Lucien Coffe d8fa225b8e cdp: Network.loadingFinished reports the bytes received for the response
encodedDataLength was the Content-Length header: 0 for a chunked or
streamed response, the compressed length without any header byte for a
gzipped one. Chrome reports the bytes the response put on the wire, its
headers and its body as received (still compressed).

A response fetched over the network now reports libcurl's header size
(CURLINFO_HEADER_SIZE) plus its body bytes as received
(CURLINFO_SIZE_DOWNLOAD_T, after chunk framing is removed, before
decompression). A redirect counts its final hop, a revalidated 304 its
own bytes rather than the cached body it unlocks, and a partial fetch
that stops early what arrived before it stopped. A cache hit, a
fulfilled or a synthetic response keeps reporting its body length.

Network.responseReceived's encodedDataLength is unchanged.
2026-10-23 19:19:10 -07:00
Karl Seguin 7dcf8ef38e dev: groundwork for cross validating our unit tests in chrome
Will rely on an external driver (in demo) to run our page tests in Chrome and
catch unintentional divergence.

Initially this involves:

1 - Flagging every test with a marker (`<meta name=lp-mode content=test>`) so
    that the external runner knows this is a test file to be run, and not
    some support file.

2 - Extracting the TestHTTPServer and TestWSServer so that they can be run from
    a new (very simple) build step (`zig build test-server`) so that the
    test files and test endpoints (e.g. `/redirect_cross_echo_headers`) are
    accessible

3 - Starting to tweak testing.js so that it runs better in Chrome. Namely
    removing some of the attribution (for module scripts) which depend on our
    non-compliant document.currentScript behavior.

4 - Flagging a few tests and fixing them, as proof of concept.
2026-10-23 07:07:02 +08:00
Lucien Coffe fdb3bf2471 cdp: report a non-UTF-8 status text and response headers as strings
The reason phrase and the response header names and values are the
server's raw octets. std.json writes a slice that is not valid UTF-8 as
an array of numbers, so a Latin-1 byte (e.g. 0xA0) turned
Fetch.requestPaused's responseStatusText and responseHeaders, and
Network.Response.statusText, into arrays. A client expecting strings
cannot decode the event; at the Fetch response stage it never answers
the pause and the request hangs until it times out.

They now go through SafeString, like Network's headers already do:
non-UTF-8 bytes are read as Latin-1 and transcoded, as Chrome does.
2026-10-20 22:44:18 +02:00
Karl Seguin 375af14aa6 Rename option from --socket to --unix-socket
Allow WebDriver protocol (yes, we might advertise an invalid session URL, but
I think if someone uses --unix-socket, they'll be able to deal with that (at
least we can give them the chance to))

Dedupe json/version payload generation

Centralize all flag validation in Config.zig
2026-10-18 12:59:04 +08:00
Adrià Arrufat b65c6caa06 Store a script's resolved encoding and let V8 validate UTF-8
The Content-Type charset is now kept as the encoding it names, which
lives in encoding_rs's static tables, so recording it no longer
allocates or fails. Classic scripts in UTF-8 skip the validation pass:
V8 already replaces malformed sequences as decoding would, so only a
UTF-16 BOM needs the decoder.

The encoding helpers become EncodingInfo.forLabel/utf8/isUtf8, and
asUtf8 is renamed validUtf8 to set it apart from decodeToUtf8.
2026-10-12 22:01:42 +02:00
Adrià Arrufat 697331a3b8 Decode classic scripts with their charset
External scripts reached V8 as raw bytes, read as UTF-8, so every
non-ASCII character of a script served in Latin-1, Shift_JIS, GBK or
windows-1251 became U+FFFD.

Decode classic scripts as the HTML spec specifies: a BOM, then the
Content-Type charset, then the element's charset attribute, then the
document's encoding. The Content-Type charset is recorded in the header
callback for async, defer and preloaded scripts, and from the copied
headers for sync ones. Module scripts stay UTF-8, as the spec requires.
2026-10-12 21:53:57 +02:00
Adrià Arrufat de6917d08e css: decode fetched stylesheets before tokenizing
External stylesheets reached the tokenizer as raw response bytes, but
the tokenizer assumes valid UTF-8: it skips as many bytes as a lead
byte claims. A single Latin-1 byte swallowed the closing quote or */
after it, so the rest of the sheet became one string or comment, and a
truncated sequence at the end of a sheet read past the input.

Pick the encoding as css-syntax specifies (BOM, Content-Type charset,
@charset, then the document's encoding) and decode with encoding_rs,
which turns malformed sequences into U+FFFD. Valid UTF-8 without a BOM
is passed through without a copy.
2026-10-12 21:10:46 +02:00
Karl Seguin cca1fc1736 extend time abort timing for slow CI 2026-10-10 09:33:13 +08:00
Adrià Arrufat 834b58b2ae css: trim stylesheet cap comments 2026-10-09 23:24:15 +02:00
Adrià Arrufat bba49da1b6 css: raise the external stylesheet cap to 4 MiB
caltrain.com ships its Drupal theme as one 2.35 MB sheet, which the 2 MiB
cap rejected. Bootstrap's `.dropdown-menu` and `.tab-pane` display:none
rules live in it, so collapsed menus and inactive tab panels stayed in
computed style and innerText.
2026-10-09 23:02:18 +02:00
Karl Seguin 90d11a79e4 Merge pull request #3871 from lightpanda-io/test-handler-thread-safety2
ci: remove thread-unsafe allocator in test http handler
2026-10-09 18:02:59 +08:00
Karl Seguin 5d0e30aa4e ci: remove thread-unsafe allocator in test http handler
https://github.com/lightpanda-io/browser/pull/3801 removed the use of a non-
thread-safe allocator in tests. https://github.com/lightpanda-io/browser/pull/3742
which overlapped (more or less) continued to use the unsafe approach. This
fixes it.
2026-10-09 13:58:54 +08:00
Adrià Arrufat 4451aca092 test: free the curl route list after the App that uses it 2026-10-08 22:33:03 +02:00
Adrià Arrufat 7a6518c63e test: clean up shard dirs and stop every shard on fail-first
runShards ends in exit(), which skips defers, so its temp dir was never
removed; it now uses tmpDir and cleans up before exiting. With
TEST_FAIL_FIRST a failing shard pushes the shared counter past the last
test so its siblings stop too. Status lines are buffered so long ones
also go out in one write, the Trace test writes under tmpDir, and the
report printing and comments are trimmed.
2026-10-08 17:57:24 +02:00
Adrià Arrufat 7cd4863c48 test: simplify the parallel runner and port routing
Share the bound-address lookup as sys_net.boundAddress, set the test
connect-to routes once per handle in Connection.reset, add a
testing.tmpPath helper, and print the runner Report directly instead of
round-tripping it through SlowTracker. Replace deprecated
std.fmt.allocPrint and std.fs.path uses.
2026-10-08 17:18:03 +02:00
Adrià Arrufat d6dfd23217 test: let several test processes run at once
The test servers bind ephemeral ports instead of 9582-9584. Fixtures and
tests keep addressing the fixed ports: in test builds, libcurl's
CONNECT_TO routes them to the real ones, so URLs and origins are
unchanged. Only the few CDP server tests that dial it, or assert a URL
it advertises, use the real port.

Tests that wrote to fixed paths (/tmp/lightpanda_test.sqlite, /tmp/iid,
.zig-cache/tmp/upload.txt, files in the repo root) now use tmpDir.
2026-10-08 17:08:39 +02:00
Karl Seguin 7a3671c3b3 dev: micro improvement to unit test run
Overall, no real difference, but most small html runner tests go from 40ms ->
20ms. We use to check -> tick -> sleep. So if the tick completed the test, we'd
still speed. Now we tick -> check -> sleep.
2026-10-08 18:34:58 +08:00
Karl Seguin f1c8abb1ce Merge pull request #3742 from lightpanda-io/nikneym/compression-streams
webapi: add Compression Streams API
2026-10-07 10:02:12 +08:00
Halil Durak 54d7902cd2 update tests for brotli coverage 2026-10-06 16:15:20 +03:00
Karl Seguin 4e9f3bf6a2 webapi: XHR response header compliance
They have to be sorted (by uppercase name) and combined. The easiest solution
is to store these sorted so that, on demand, combining them is easy.

Fixes WPT /xhr/getallresponseheaders-after-reuse.html
2026-10-06 18:18:45 +08:00
Karl Seguin a4ee6eb577 ci: remove thread-unsafe allocator in test http handler 2026-10-06 08:52:29 +08:00
Adrià Arrufat aed949481a tools: flag navigations that land on an error page or bot challenge
goto reported "Navigated successfully. HTTP 429 Too Many Requests." and
the model then read a Vercel Security Checkpoint as page content; read
tools navigating by url (markdown, tree, ...) reported no status at all.

call() now prefixes any result whose call navigated (new frame id) onto
a 4xx/5xx or a detected bot challenge (Frame._bot_challenge: Vercel,
Cloudflare, AWS WAF) with a warning. Only model-facing callers (agent,
MCP) opt in through CallOpts.nav_note: scripts parse the text as-is.
goto says "Navigated." instead of "Navigated successfully."
2026-10-05 22:57:44 +02:00
Adrià Arrufat 01a703f7e2 Merge branch 'main' into zig-0.17
# Conflicts:
#	src/browser/StyleManager.zig
2026-10-03 16:12:46 +02:00
Adrià Arrufat c3083725c5 Replace deprecated std.mem.indexOf* with find* 2026-10-03 16:09:42 +02:00
Adrià Arrufat 10609b025a Upgrade to Zig 0.17
Dependencies:
- v8: zig-v8-fork 0.17 branch (lightpanda-io/zig-v8-fork#218).
- sqlite3: build the amalgamation directly; the allyourcodebase wrapper
  has no 0.17 support yet.
- pcre2: master commit with 0.17 support (no release yet).
- translate_c package (2.0.0) replaces the deprecated addTranslateC.
  default_init is set to keep 0.16's zero-initialized struct fields.

Build:
- b.pathFromRoot/build_root/args/sysroot are gone: resolve paths from
  b.root, use addPassthruArgs, declare configure-time file and directory
  dependencies, and poison the configure cache for the git version.
- Drop curl config values the template never used (now an error).
- Drop the 0.16 zip-fetch workaround from the Makefile and CI.
- -Doptimize=ReleaseFast -> -Doptimize=fast.

Language and std:
- @intFromEnum/@enumFromInt -> @backingInt/@fromBackingInt.
- Struct-of-arrays @typeInfo (field_names, field_types, decl_names,
  param_types, error_names).
- `a ** n` removed: @splat for arrays, string.repeat for strings.
- errdefer captures removed: split WebDriver.run and fulfillRedirect.
- @hasDecl only sees pub decls: make StyleManager's Spec.finish pub,
  otherwise it silently stops running.
- SafeAllocator replaces DebugAllocator; the test runner now has to
  initialize std.testing.allocator_instance itself.
- std.fmt.allocPrint/bufPrint -> Allocator.print/std.mem.print,
  dupeZ -> dupeSentinel, std.builtin -> std.lang, builtin.os ->
  builtin.target.os, zon.parse arena API, BufferFirstAllocator, and the
  remaining renames (getLastOrNull, bit_set, ascii.find*, meta.Int).
2026-10-03 16:09:42 +02:00
Karl Seguin fc7935e51a webapi: fetch resolves after header received
We currently reject a fetch() if an error happens at any point during the
request. This is wrong. Once the header is received, the fetch should resolve.
If there was an error after reading the headers, then that should cause the
cause the `response.body()` or `response.json()` (or whatever) to reject).

This adds a new `header_before_body_error: bool = false` to the request which
will execute the header_callback before the error_callback on a post-header
error. fetch can use this to determine how to handle the error_callback.

Fixes a handful of WPT cases, e.g.
/fetch/content-encoding/zstd/bad-zstd-body.https.any.html
2026-10-02 08:24:19 +08:00
Karl Seguin d6de5fee8e Merge headers_only and headers_only_prefix_bytes into a single optional
`partial: ?u32 = null` is now the tristate to capture (1) load everything (null)
(2) load no body (0) and  (3) load N body (N).

partial responses are NOT sent to CDP - the data can be garbage.

read orientation of jpeg and swap height/width as needed
2026-09-30 07:34:38 +08:00
Scott Taylor 009225ace1 Report natural dimensions of fetched images from a bounded header prefix 2026-09-29 14:43:46 -04:00
Karl Seguin 5932638baf Merge pull request #3671 from lightpanda-io/status-text
webapi: xhr/fetch report status reason from server
2026-09-28 22:43:27 +08:00
Pierre Tachoire be65962d68 testing: remove unused func 2026-09-28 14:04:11 +02:00
Karl Seguin f7fbc2f34f webapi: xhr/fetch report status reason from server
If the server gives a status reason, report it as-is. Only default to the
zig code->reason map when one isn't given.

Also, don't force a 407 status code when auth_challenge is present.
2026-09-28 19:01:39 +08:00
Pierre Tachoire cebd8e3690 cors: fix unit tests when enabling cors by default 2026-09-28 12:23:29 +02:00
Scott Taylor 5b1bb1bc78 Respect important declarations in tracked CSS cascade 2026-09-26 23:55:06 -04:00
nikneym 238d4a53a3 cdp: move a kept isolated-world context onto the navigated origin 2026-09-21 16:28:00 +03:00
Karl Seguin d28d37be92 webapi: improve fetch/response correctness with null bodies
Responses with specific status (e.g. 204) should always have a null body. Also
adds validation to Response constructor (e.g. can't provide an invalid status).

Improves a handful of WPT tests:

fetch/api/response/response-error.any.html
fetch/api/response/response-static-json.any.html
2026-09-19 13:38:12 +08:00
Karl Seguin e2299e2c05 dev: speedup unit tests
Directly improve a couple slow tests (crash_handler actually generating crash
dumps on some systems, e.g. mine). The runWebApiTest prefers to poll for work
when possible rather than a blind sleep -> check loop. (e.g. if we have
websocket connections, prefer an http tick).

For me, it's 22s -> 17s.
2026-09-18 15:17:39 +08:00
Karl Seguin dc1ae129e3 webapi: experimental (and useless, for now) ServiceWorker
This adds the shell for ServiceWorker, behind
`--experimental-features serviceworker`.

It's pretty useless as-is. We don't have the CacheStorage API (next) and don't
have the fetch interceptor (next next). But as-is, the change is quite big but
thankfully largely isolated.
2026-09-16 17:49:07 +08:00
Karl Seguin 2e6999f20b chore: make declarations private if they don't need to be public
This change is 99%  s/pub//   + a handful of dead code removal.
2026-09-10 14:42:09 +08:00