Will rely on an external driver (in demo) to run our page tests in Chrome and
catch unintentional divergence.
Initially this involves:
1 - Flagging every test with a marker (`<meta name=lp-mode content=test>`) so
that the external runner knows this is a test file to be run, and not
some support file.
2 - Extracting the TestHTTPServer and TestWSServer so that they can be run from
a new (very simple) build step (`zig build test-server`) so that the
test files and test endpoints (e.g. `/redirect_cross_echo_headers`) are
accessible
3 - Starting to tweak testing.js so that it runs better in Chrome. Namely
removing some of the attribution (for module scripts) which depend on our
non-compliant document.currentScript behavior.
4 - Flagging a few tests and fixing them, as proof of concept.
`make coverage` (or `zig build coverage`) runs the unit tests under kcov
and writes an HTML report to zig-out/coverage/index.html. F= filters as
with `make test`.
The tests are built with LLVM for this step: kcov can't read line info
from the self-hosted backend's DWARF, so a dev_fast binary reports no
Zig files. The suite runs as a single process because each shard would
be its own report.
The crash_handler signal tests now skip under a tracer. kcov follows
their forked child and intercepts its fatal signal, so a hardware fault
re-executes forever and the run never finishes.
Read browser tool schemas from Schema.all() instead of re-parsing them,
format table rows through Anchor/Cell formatters, share one flag writer
between traits and hints, and use mcp/tools.zig's browser_tool_list
directly. Group headings move next to the generator, and an addGenerator
helper in build.zig drives both the skills and docs steps.
Render MDX partials for the docs site from the registries the runtime
serves, so the browser tools, MCP tools and agent REPL command references
can't drift from the code:
- browser-tools.mdx: every tool, grouped, with its description, an
argument table from its input schema, and its script/navigation/page
state traits.
- mcp-tools.mdx: titles and annotation hints per tool, linking to the
browser tools reference, plus the MCP-only tools and resources.
- agent-commands.mdx: meta, LLM and browser slash commands in the syntax
the REPL parses.
Tool.group() assigns each tool its docs section as an exhaustive switch,
so a new tool can't ship undocumented. The release workflow attaches the
partials to each release as lightpanda-docs.tar.gz for the docs repo to
pull.
Dependencies:
- v8: zig-v8-fork 0.17 branch (lightpanda-io/zig-v8-fork#218).
- sqlite3: build the amalgamation directly; the allyourcodebase wrapper
has no 0.17 support yet.
- pcre2: master commit with 0.17 support (no release yet).
- translate_c package (2.0.0) replaces the deprecated addTranslateC.
default_init is set to keep 0.16's zero-initialized struct fields.
Build:
- b.pathFromRoot/build_root/args/sysroot are gone: resolve paths from
b.root, use addPassthruArgs, declare configure-time file and directory
dependencies, and poison the configure cache for the git version.
- Drop curl config values the template never used (now an error).
- Drop the 0.16 zip-fetch workaround from the Makefile and CI.
- -Doptimize=ReleaseFast -> -Doptimize=fast.
Language and std:
- @intFromEnum/@enumFromInt -> @backingInt/@fromBackingInt.
- Struct-of-arrays @typeInfo (field_names, field_types, decl_names,
param_types, error_names).
- `a ** n` removed: @splat for arrays, string.repeat for strings.
- errdefer captures removed: split WebDriver.run and fulfillRedirect.
- @hasDecl only sees pub decls: make StyleManager's Spec.finish pub,
otherwise it silently stops running.
- SafeAllocator replaces DebugAllocator; the test runner now has to
initialize std.testing.allocator_instance itself.
- std.fmt.allocPrint/bufPrint -> Allocator.print/std.mem.print,
dupeZ -> dupeSentinel, std.builtin -> std.lang, builtin.os ->
builtin.target.os, zon.parse arena API, BufferFirstAllocator, and the
remaining renames (getLastOrNull, bit_set, ascii.find*, meta.Int).
Debug and release builds each compiled their own copy of every C
dependency and of the Rust staticlib, because build.zig threaded the
top-level optimize mode into all of them. Under dev_fast the deps also
picked up the bundled-CRT target query, so even the same mode could not
share objects with a plain build.
Dependencies now build in ReleaseFast for the requested target, the way
the prebuilt V8 archive already works. Debug and release builds share
one set of cached dependency objects, and debug binaries run TLS, HTML
parsing, regex and sqlite optimized. -Ddebug_deps restores the old
behaviour for stepping into a dependency.
The Rust staticlib can only be shared by dropping the Debug-only memstats
feature: its single export, html5ever_get_memory_usage, was declared on
the Zig side but never called, and it pulled a jemalloc build into every
cold debug build. The Makefile override that existed for jemalloc's
nested make goes with it.
Filter lists carry a few hundred rules written as JavaScript regex
literals (24 in EasyList, 165 in uBO's badware list); they parsed but
were dropped as unsupported. PCRE2 reads that syntax as-is, `\/` and
friends included, its compiled patterns are immutable so the one
blocker shared by every HTTP client thread can run them, and 10.48
ships a build.zig for 0.16, so it is wired like sqlite3.
`Regex.Context` routes every PCRE2 allocation through the blocker's
allocator, which puts the compiled patterns under the test runner's
leak detection, and caps match and depth so a broken pattern costs a
false negative rather than a stalled request. As in uBO, a regex
tests the raw URL with the case-insensitive flag unless `$match-case`.
Regex filters are still never tokenized: they ride the fallback bucket.
curl 8.22 moved the resolver sources into lib/vdns/, renamed the QUIC
backends to cf-ngtcp2*/cf-quiche, folded noproxy.c into proxy.c and
added a handful of new files. Regenerate the source list from the
tarball and drop the four config defines the 8.22 template no longer
declares (Zig errors on unused config header values).
https://github.com/lightpanda-io/browser/pull/3271 introduce an orderfile so
that hot sections of code were grouped together in the binary, resulting in more
efficient loading. But, it excluded v8 functions because v8 is compiled with
`-fno-unique-section-names` so each section gets the same name. Because of this
the orderfile can't target specific "hot" or "cold" functions.
As a follow up to 3271, I thought we'd be able to re-compile v8 without that
flag, but:
1 - The flag isn't directly exposed by v8, so we'd need to change v8's own build
BUILD.gn in our build process
2 - It would make the .a file ~ 40MB larger (though the final lightpanda binary
would stay the same size
3 - If we wanted to created two .a files (one with -fno-unique-section-names and
one without), it would require a full rebuild (for both x86-64 and arm).
I didn't love those compromises, so I asked Claude if the sections names could
be generated in a separate file and then that could be used when generating out
lightpanda.ld. What claude came up with was a small Zig script
(mark_hot_sections.zig) which is now run as part of the build. It takes the
v8.a file (which still has `-fno-unique-section-names`), it takes a text list
of hot functions, and it re-generates v8.a with a special .text.hot and
.rodata.hot sections. lightpanda.ld can include these.
This PR depends on https://github.com/lightpanda-io/zig-v8-fork/pull/202 but
202 doesn't require a v8 rebuild. It merely allows prebuilt_v8_path to be
a lazypath, which we need because the prebuilt_v8_path that we pass is now
generated from this build script.
So we're trying to limit glibc to 2.43.0 do to an issue in Zig's self-hosted
compiler. I'm on 2.44.0 and I hard-coded 2.43.0. But for anyone on a version
prior to 2.43.0, that causes issues. So this takes the host version but limits
it to 2.43.0
The explicit version only exists to force Zig's bundled CRT (zig#31272),
but it also selects symbol versions. PR #3231 added tiny-skia, which
calls acosf, and glibc 2.43 re-versioned that symbol, so the binary now
fails to load on hosts with an older glibc:
libm.so.6: version `GLIBC_2.43' not found
Use the detected host glibc instead: still an explicit version, so the
bundled CRT is still used, but never newer than what the binary runs on.
CI build, for e2e-test and nightly now build with:
-Dorderfile/lightpanda.ld
This file informs the build on how to organize the code in the binary, grouping
hot code together so that we have to load less of the binary into memory.
lightpanda.ld will drift: we'll refactor our code, add new features, update
dependencies, update Zig, ... So it has to be re-generated. But we can do that
automatically in the CI (say, before the nightly build). That's for a follow up
PR.
This does not currently cover V8. V8 is being build with
`-no-unique-section-names`, so we don't get names that we can correctly
organize. The real win comes from doing this in V8, since a lot of V8 is cold.
This PR can land as-is, a zig-v8-fork PR will remove that flag, and then we can
have a follow up PR with an lightpanda.ld that includes the v8 symbols.
This is opt-in (via the -Dorderfile flag) because it adds ~20 seconds of
linking time.
Cargo prints its progress to stderr. Zig's build sees the stderr content and
prints it as a "failed command: ...". Capture cargo's stderr to silence this
fake-error. A status != 0 still causes the error to be printed.
Deduplicate the three executable blocks and the vendored C library
module setup, drop vestigial error unions, and replace hand-rolled
formatting with b.fmt/b.pathJoin/std.mem.concat.
The snapshot-creator and skills executables now get the same
sanitize_c/sanitize_thread flags as the browser executable, which
they were silently missing.
Prebuilt V8 discovery now reports every cache miss with the probed
path. Previously a miss on the static archive path fell back to the
source build without a message, and the dev_fast message claimed a
source build was starting when the graph might not compile anything.
Assuming the conditions are met for -dev_fast to work (linux, debug, x86-64
without sanitizers), the -dev_fast now defaults to true.
download-v8 make target always downloads the .a (as before) and on linux x86-64
it also downloads the .so.
Supports prebuild-v8 shared library and adds a `download-v8-shared` make target.
Depends on: https://github.com/lightpanda-io/zig-v8-fork/pull/198
The auto-discovery of the .so is done in the build.zig, which is different than
the auto-discovery of the .a which is done in the Makefile. This is intentional
and if we inverse -Ddev_fast so that it's on by default, the .a's auto discovery
will be moved to build.zig for consistency and explicitness. As-is, we have no
way to tell whether -Dprebuilt_v8_path=v8.a is being injected by Makefile or
is explicit (which isn't a problem so long as we're not doing dev_fast by
default).
Zig 0.17 removes @cImport in favor of translate-c steps in the build
system. Each library's link function now creates an addTranslateC step
exposed as a module import: curl and isocline translate their headers
straight from the dependency tree, while sqlite3 uses the artifact's
emitted include tree since the amalgamation lives in a sub-dependency.
Terminal.zig and prompt_assist.zig previously instantiated two
independent @cImport namespaces for isocline; they now share one
translated module, so its types are identical across both files.
Make sure jemallocator is disabled in Release builds (we only use it in Debug
builds to collect metrics. Supposedly something changed in our 0.16 build that
causes it to be initialized even though it isn't used. Not sure, for this, I'm
just doing what Claude tell me, but disabling something in Release that I know
we aren't using sounds ok).
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
The /save script documentation lived as a hand-maintained string in
Agent.zig whose primitives table drifted whenever a tool changed. Move
it to src/script/skill.zig and render the reference (signatures, option
lists, enums, defaults, per-parameter descriptions) from Schema.all()
at first use, keeping the curated per-tool notes behind an exhaustive
switch on Tool so a new or renamed tool is a compile error until its
doc entry exists.
The rendered skill is shared by three consumers:
- the /save and revision system prompts (built lazily in Agent.zig)
- a new mcp://skill/pandascript resource
- `zig build skills`, which writes zig-out/skills/<name>/SKILL.md with
Claude Code frontmatter via a registry-based generator exe, so
future Lightpanda skills are one registry entry each
Schema.FieldEntry now retains per-parameter schema descriptions, which
previously existed only in the raw JSON.
This is a bit all over the place.
1 - Replace libidn2 with rust-idna. It looks like there are different idna
profiles, and rust-idna (from the servo project) implements the whatwg
one. libidn2 would be too strict in some cases and not strict enough in
others. (Gemini says I could use libidn2 for this, but what it suggested
didn't work, and I couldn't figure it out myself, and claude insisted it
_did not_ have the correct implementation for what we want).
2 - We previously only ran a URL through idna if it wasn't ascii. Turns out
we also need to run it if there's a "xn--" (aka, an IDNA ACE prefix) in
there. This helps us pass hundreds of WPT cases, and it's pretty cheap.
3 - Implement more of the Area WebAPI. Mostly copied from Anchor.
4 - Add username/password accessor to Anchor/Area
5 - window.open validates the URL (i.e. tries to resolve it and handles the
error)
6 - Invalid idna conversion maps to a TypeError
7 - Cleanup closed popups on the next tick (like destroyed pages), rather than
at an interval or on shutdown. This one seems unrelated, but some of these
tests are opening hundreds (thousands?) of popups and then closing them.
AFAIK, these aren't being used and I've personally not have reason to reference
/ look at them for months.
I have no issues though if we want to keep them in.
The build script wrote the version line to stdout, polluting any pipeline
that captures program output via 'zig build run' or similar. Banners
belong on stderr.
Replaces the linenoise library with isocline for improved REPL
functionality. Updates build scripts and moves completion state from
global variables into the Terminal struct.
config.h already has a hand-added _LIBIDN2_LP_DECLS guarded block at
the end declaring extern strverscmp for the same gnulib-overlay reason.
Move the strchrnul prototype into that block (still #ifdef __APPLE__
since glibc Linux already declares it) so all project-added externs
live in one place, the autoconf-generated section stays untouched, and
the precedent for "how this codebase handles missing glibc extensions"
is one consistent pattern instead of two.
Refresh the build.zig and strchrnul.c comments to point at the
relocated declaration.
No functional change — verified across -target x86_64-macos.{13.0,
14.0,15.4} (lookup.c pattern + the three TUs that include both
<config.h> and <string.h>) plus x86_64-linux-gnu, all under
-Wall -Wextra -Werror.