classify, acceptCookies and login now go through zenai.decisions, which
asks whichever decision model has a key: Gemini's Decisions API
(GOOGLE_API_KEY/GEMINI_API_KEY) or TypeSafe System One
(TYPESAFE_API_KEY). Gemini is preferred when both are set;
LIGHTPANDA_DECISION_PROVIDER=gemini|typesafe picks one.
Messages, tool descriptions and the REPL's no-key fallback name both
keys. The test mock serves Gemini's endpoint too, every mock test names
its provider so a developer's own Gemini key never reaches Google, and
the acceptCookies and login happy paths run against both.
`askPage` called the pool inline, so the browser thread sat in a blocking
HTTPS read for the length of the answer — 200ms on a good connection, up to
the 10s request timeout. `search` has wrapped its own HTTP in
`Session.runPumped` from the start, for the reason its comment gives, and
this is the same read.
Thread-per-session in both servers (`server/Server.zig:593`,
`mcp/HttpServer.zig:299`) means no other session was ever affected. What
stalled was the calling session's own page: no Runner tick, so its timers and
queued work stop and the curl multi driving its transfers goes unpolled,
while `CURLOPT_TIMEOUT_MS` (15s) keeps counting against them in wall-clock.
That is the shape of the MCP idle-transfer bug and of the agent's blocking
model call, both already fixed this way. The watchdog is not in play at its
30s default, but `--watchdog_ms` under 10s would be.
`std.Io.Mutex` under `Io.Threaded` is a futex mutex that blocks the calling
OS thread rather than parking a task, so the pool is safe to take from a
helper thread and zenai needs no change; `init_single_threaded` constrains
its task pool, not other threads. zenai already drives `lp.io` timers from a
thread it spawns per request, and `buildState` stays on the browser thread,
so only the HTTP leaves it.
Pumping lets the page's own scripts run during the ask, which a blocked
thread never allowed. A navigation among them swaps the root frame and tears
down the Page that the candidate elements and node ids point into, so the ask
now reports that rather than handing the caller an answer about a page that
has gone — the `late_redirect.html` test navigates mid-ask and reaches it.
Detached-but-not-freed nodes within one document stay readable, and a
selector built from one simply fails to resolve, which every caller already
handles.
The mock can now answer slowly, which is what makes the stall observable: a
20ms interval on the page advances across a 400ms answer (a turn per 250ms
quiet slice) where blocking gave it none at all.
The credential guards ran on the first decision only. An identifier-first
login submits, asks again on the page that comes back, and then filled the
password into whatever that second answer named — a plain `input[type=text]`,
or a form posting to `http://`. They are one predicate now, asked of both
answers. `replayLogin` already checked a selector a caller handed it, which
is what made the gap visible.
A recorded two-step login could not replay at all: `--save` writes both
selectors into one `login(...)` call, and the password's field is not on the
first page, so resolving it returned `NodeNotFound` and the tool failed
outright. It repeats the submit that raised the field now, as the decision
did. A replay given only `usernameSelector` says so, rather than reporting
"filled the fields given" for a password it never typed.
`choose.ranked` measured candidates against `none` alone, so an `extra`
option winning the argmax left the elements below it in play: a login whose
answer was "press Enter" could click an in-form "Create an account" instead,
since the submit control is only checked for sharing the field's form. An
answer that is not a candidate drops them all, which is what its doc comment
already promised.
A verification ask that failed threw away the outcome it was verifying. Both
the consent loop and `verifyLogin` returned the raw API failure, so a 500
after `login` had submitted real credentials told the caller "ApiError" and
nothing about the submit, the selectors or the replay arguments.
`classify` was annotated `read_only`, which claims `openWorldHint = false`
and `idempotentHint = true` for a paid call to a third-party model that
answers differently twice. `search`, the other external-API tool, already
leaves `openWorldHint` alone.
The two-step fixtures need a mock that decides there is a later step, so it
reads that off the one thing it can see — a request that offered no field
looking like a password — and fails the verification re-read for a key a test
sets. One doc comment the marshal move had split in two is whole again.
From the JSON a caller sent to the struct a tool reads: the schema's casing,
the `$LP_*` placeholders, and what a malformed call is told. Five of these
were already public and called from `mcp/tools.zig`, `agent/Agent.zig` and
`script/Recorder.zig` — they lived in tools.zig because the tools were their
first caller, not because they belong to it. The names callers use are
re-exported, so nothing outside changes.
`marshal`, not `args`: Zig refuses the import alias the basename would give,
because `const args = try parseArgs(...)` is the first line of nearly every
`exec*`. The repo already calls this marshalling (`Runtime.marshalArgs`).
`lpEnvNames` and `lookupLpEnv` go with it. They are the `LP_` namespace, read
by the substitution as much as by the `getEnv` tool, and the substitution is
here now.
One duplicate gone on the way: the `setenv`/`unsetenv` externs are declared
once, in the test module both files already import, instead of a copy per
file.
The question DSL and the client pool had no browser knowledge in them, so
they are zenai's now (lightpanda-io/zenai#44).
classify.zig keeps what is actually the browser's: the page presets a caller
can name instead of wording a question, the wording of the category question,
and what an empty questions object means. It goes from 454 lines to 110, and
its question-shape tests move to zenai with the code.
App keeps a zenai.typesafe.Pool rather than its own idle list. The pool is
handed lp.environ rather than a snapshot of it, because setenv can move the
block an Environ points into — and because it retires a client whose
credentials the environment no longer agrees with, the test scaffold no
longer has to drop idle clients around every Jev test.
The mock answers through std.json.Stringify instead of hand-built JSON with
comma flags, keys come from the constants the real code exports rather than
copies of them, and one enum says which of the three request shapes it got
instead of two flags and an inert count guard.
The REPL's two prompt paths share one helper; fromToolCall gets its doc
comment back, which an insert had handed to its neighbour; four identical
error switches become one function; LineCollector drops an allocator field
its tree already carried; and the TypeSafe test preamble, written four times,
is one scaffold beside the test server that owns the port it needs.
The search tool kept its own copy of zenai.http.ErrorDetail and duped
the message by hand; ErrorDetail.clone does that. The classify tests
take the environment variable names from zenai.
encodedDataLength was the Content-Length header: 0 for a chunked or
streamed response, the compressed length without any header byte for a
gzipped one. Chrome reports the bytes the response put on the wire, its
headers and its body as received (still compressed).
A response fetched over the network now reports libcurl's header size
(CURLINFO_HEADER_SIZE) plus its body bytes as received
(CURLINFO_SIZE_DOWNLOAD_T, after chunk framing is removed, before
decompression). A redirect counts its final hop, a revalidated 304 its
own bytes rather than the cached body it unlocks, and a partial fetch
that stops early what arrived before it stopped. A cache hit, a
fulfilled or a synthetic response keeps reporting its body length.
Network.responseReceived's encodedDataLength is unchanged.
Will rely on an external driver (in demo) to run our page tests in Chrome and
catch unintentional divergence.
Initially this involves:
1 - Flagging every test with a marker (`<meta name=lp-mode content=test>`) so
that the external runner knows this is a test file to be run, and not
some support file.
2 - Extracting the TestHTTPServer and TestWSServer so that they can be run from
a new (very simple) build step (`zig build test-server`) so that the
test files and test endpoints (e.g. `/redirect_cross_echo_headers`) are
accessible
3 - Starting to tweak testing.js so that it runs better in Chrome. Namely
removing some of the attribution (for module scripts) which depend on our
non-compliant document.currentScript behavior.
4 - Flagging a few tests and fixing them, as proof of concept.
The reason phrase and the response header names and values are the
server's raw octets. std.json writes a slice that is not valid UTF-8 as
an array of numbers, so a Latin-1 byte (e.g. 0xA0) turned
Fetch.requestPaused's responseStatusText and responseHeaders, and
Network.Response.statusText, into arrays. A client expecting strings
cannot decode the event; at the Fetch response stage it never answers
the pause and the request hangs until it times out.
They now go through SafeString, like Network's headers already do:
non-UTF-8 bytes are read as Latin-1 and transcoded, as Chrome does.
Allow WebDriver protocol (yes, we might advertise an invalid session URL, but
I think if someone uses --unix-socket, they'll be able to deal with that (at
least we can give them the chance to))
Dedupe json/version payload generation
Centralize all flag validation in Config.zig
The Content-Type charset is now kept as the encoding it names, which
lives in encoding_rs's static tables, so recording it no longer
allocates or fails. Classic scripts in UTF-8 skip the validation pass:
V8 already replaces malformed sequences as decoding would, so only a
UTF-16 BOM needs the decoder.
The encoding helpers become EncodingInfo.forLabel/utf8/isUtf8, and
asUtf8 is renamed validUtf8 to set it apart from decodeToUtf8.
External scripts reached V8 as raw bytes, read as UTF-8, so every
non-ASCII character of a script served in Latin-1, Shift_JIS, GBK or
windows-1251 became U+FFFD.
Decode classic scripts as the HTML spec specifies: a BOM, then the
Content-Type charset, then the element's charset attribute, then the
document's encoding. The Content-Type charset is recorded in the header
callback for async, defer and preloaded scripts, and from the copied
headers for sync ones. Module scripts stay UTF-8, as the spec requires.
External stylesheets reached the tokenizer as raw response bytes, but
the tokenizer assumes valid UTF-8: it skips as many bytes as a lead
byte claims. A single Latin-1 byte swallowed the closing quote or */
after it, so the rest of the sheet became one string or comment, and a
truncated sequence at the end of a sheet read past the input.
Pick the encoding as css-syntax specifies (BOM, Content-Type charset,
@charset, then the document's encoding) and decode with encoding_rs,
which turns malformed sequences into U+FFFD. Valid UTF-8 without a BOM
is passed through without a copy.
caltrain.com ships its Drupal theme as one 2.35 MB sheet, which the 2 MiB
cap rejected. Bootstrap's `.dropdown-menu` and `.tab-pane` display:none
rules live in it, so collapsed menus and inactive tab panels stayed in
computed style and innerText.
runShards ends in exit(), which skips defers, so its temp dir was never
removed; it now uses tmpDir and cleans up before exiting. With
TEST_FAIL_FIRST a failing shard pushes the shared counter past the last
test so its siblings stop too. Status lines are buffered so long ones
also go out in one write, the Trace test writes under tmpDir, and the
report printing and comments are trimmed.
Share the bound-address lookup as sys_net.boundAddress, set the test
connect-to routes once per handle in Connection.reset, add a
testing.tmpPath helper, and print the runner Report directly instead of
round-tripping it through SlowTracker. Replace deprecated
std.fmt.allocPrint and std.fs.path uses.
The test servers bind ephemeral ports instead of 9582-9584. Fixtures and
tests keep addressing the fixed ports: in test builds, libcurl's
CONNECT_TO routes them to the real ones, so URLs and origins are
unchanged. Only the few CDP server tests that dial it, or assert a URL
it advertises, use the real port.
Tests that wrote to fixed paths (/tmp/lightpanda_test.sqlite, /tmp/iid,
.zig-cache/tmp/upload.txt, files in the repo root) now use tmpDir.
Overall, no real difference, but most small html runner tests go from 40ms ->
20ms. We use to check -> tick -> sleep. So if the tick completed the test, we'd
still speed. Now we tick -> check -> sleep.
They have to be sorted (by uppercase name) and combined. The easiest solution
is to store these sorted so that, on demand, combining them is easy.
Fixes WPT /xhr/getallresponseheaders-after-reuse.html
goto reported "Navigated successfully. HTTP 429 Too Many Requests." and
the model then read a Vercel Security Checkpoint as page content; read
tools navigating by url (markdown, tree, ...) reported no status at all.
call() now prefixes any result whose call navigated (new frame id) onto
a 4xx/5xx or a detected bot challenge (Frame._bot_challenge: Vercel,
Cloudflare, AWS WAF) with a warning. Only model-facing callers (agent,
MCP) opt in through CallOpts.nav_note: scripts parse the text as-is.
goto says "Navigated." instead of "Navigated successfully."
Dependencies:
- v8: zig-v8-fork 0.17 branch (lightpanda-io/zig-v8-fork#218).
- sqlite3: build the amalgamation directly; the allyourcodebase wrapper
has no 0.17 support yet.
- pcre2: master commit with 0.17 support (no release yet).
- translate_c package (2.0.0) replaces the deprecated addTranslateC.
default_init is set to keep 0.16's zero-initialized struct fields.
Build:
- b.pathFromRoot/build_root/args/sysroot are gone: resolve paths from
b.root, use addPassthruArgs, declare configure-time file and directory
dependencies, and poison the configure cache for the git version.
- Drop curl config values the template never used (now an error).
- Drop the 0.16 zip-fetch workaround from the Makefile and CI.
- -Doptimize=ReleaseFast -> -Doptimize=fast.
Language and std:
- @intFromEnum/@enumFromInt -> @backingInt/@fromBackingInt.
- Struct-of-arrays @typeInfo (field_names, field_types, decl_names,
param_types, error_names).
- `a ** n` removed: @splat for arrays, string.repeat for strings.
- errdefer captures removed: split WebDriver.run and fulfillRedirect.
- @hasDecl only sees pub decls: make StyleManager's Spec.finish pub,
otherwise it silently stops running.
- SafeAllocator replaces DebugAllocator; the test runner now has to
initialize std.testing.allocator_instance itself.
- std.fmt.allocPrint/bufPrint -> Allocator.print/std.mem.print,
dupeZ -> dupeSentinel, std.builtin -> std.lang, builtin.os ->
builtin.target.os, zon.parse arena API, BufferFirstAllocator, and the
remaining renames (getLastOrNull, bit_set, ascii.find*, meta.Int).
We currently reject a fetch() if an error happens at any point during the
request. This is wrong. Once the header is received, the fetch should resolve.
If there was an error after reading the headers, then that should cause the
cause the `response.body()` or `response.json()` (or whatever) to reject).
This adds a new `header_before_body_error: bool = false` to the request which
will execute the header_callback before the error_callback on a post-header
error. fetch can use this to determine how to handle the error_callback.
Fixes a handful of WPT cases, e.g.
/fetch/content-encoding/zstd/bad-zstd-body.https.any.html
`partial: ?u32 = null` is now the tristate to capture (1) load everything (null)
(2) load no body (0) and (3) load N body (N).
partial responses are NOT sent to CDP - the data can be garbage.
read orientation of jpeg and swap height/width as needed
If the server gives a status reason, report it as-is. Only default to the
zig code->reason map when one isn't given.
Also, don't force a 407 status code when auth_challenge is present.
Responses with specific status (e.g. 204) should always have a null body. Also
adds validation to Response constructor (e.g. can't provide an invalid status).
Improves a handful of WPT tests:
fetch/api/response/response-error.any.html
fetch/api/response/response-static-json.any.html
Directly improve a couple slow tests (crash_handler actually generating crash
dumps on some systems, e.g. mine). The runWebApiTest prefers to poll for work
when possible rather than a blind sleep -> check loop. (e.g. if we have
websocket connections, prefer an http tick).
For me, it's 22s -> 17s.
This adds the shell for ServiceWorker, behind
`--experimental-features serviceworker`.
It's pretty useless as-is. We don't have the CacheStorage API (next) and don't
have the fetch interceptor (next next). But as-is, the change is quite big but
thankfully largely isolated.