feat(sbom): add SBOM generation, license resolution, and CSV export tooling (#239)

* feat(sbom): add SBOM generation, license resolution, and CSV export tooling

Add mise-integrated SBOM pipeline for container images using Syft.
Includes license resolution via crates.io/npm/PyPI APIs and CycloneDX
JSON to CSV conversion. Adds agent skill for on-demand SBOM operations.

Closes #237

* fix(sbom): chain task dependencies to run generate → resolve → csv sequentially

* fix(sbom): add concurrent license resolution, progress logging, and exclude dev artifacts

* feat(notices): add mise run notices to generate THIRD-PARTY-NOTICES with full license texts

Use cargo-about for Rust crate licenses and pip-licenses for Python
packages. Produces a single attribution file with per-package copyright
notices and full license text for open-source compliance.
This commit is contained in:
John T. Myers
2026-03-11 15:34:00 -07:00
committed by GitHub
parent 329725d1b9
commit 169655a0d8
10 changed files with 17036 additions and 2409 deletions
+105
View File
@@ -0,0 +1,105 @@
---
name: sbom
description: Generate and manage Software Bill of Materials (SBOMs) for the OpenShell project. Covers SBOM generation with Syft, license resolution via public registries, and CSV export for compliance review. Trigger keywords - SBOM, sbom, bill of materials, license audit, license resolution, generate sbom, sbom csv, dependency license, supply chain, license scan.
---
# SBOM Generation and License Resolution
Generate CycloneDX SBOMs, resolve missing licenses, and export to CSV for compliance review.
## Overview
The OpenShell SBOM tooling produces CycloneDX JSON SBOMs using Syft, resolves missing or hash-based licenses by querying public registries (crates.io, npm, PyPI), and exports the results to CSV for stakeholder review.
SBOMs are **release artifacts only** -- they are generated on demand and not committed to the repository. Output lands in `deploy/sbom/output/` (gitignored).
## Prerequisites
- `mise install` has been run (installs Syft and other tools)
- The repository is checked out at the root
## Workflow 1: Full SBOM Generation (One Command)
```bash
mise run sbom
```
This single command chains three stages:
1. **Generate** (`sbom:generate`): Syft scans the workspace source tree and produces a CycloneDX JSON SBOM
2. **Resolve** (`sbom:resolve`): Public registry APIs fill in missing or hash-based licenses in the JSON
3. **CSV** (`sbom:csv`): JSON SBOMs are converted to CSV for review
Output directory: `deploy/sbom/output/`
After running, the user can find:
- `deploy/sbom/output/*.cdx.json` -- full CycloneDX SBOMs
- `deploy/sbom/output/*.csv` -- CSV exports ready for spreadsheet review
## Workflow 2: Individual Stages
Run stages independently when debugging or iterating:
```bash
mise run sbom:generate # Generate JSON SBOMs only (requires Syft)
mise run sbom:resolve # Resolve licenses in existing JSONs (queries APIs)
mise run sbom:csv # Convert existing JSONs to CSV
```
## Workflow 3: License Check (CI Advisory)
```bash
mise run sbom:check
```
Reports unresolved licenses without failing. Intended for PR CI as a non-blocking advisory check. Requires that SBOMs have already been generated (`mise run sbom:generate`).
## Workflow 4: Processing External SBOMs
The Python scripts accept explicit file paths, so they can process SBOMs from any source (e.g., NVIDIA nSpect pipeline output):
```bash
uv run python deploy/sbom/resolve_licenses.py /path/to/external-sbom.json
uv run python deploy/sbom/sbom_to_csv.py /path/to/external-sbom.json
```
## License Resolution Details
The resolver queries these public registries:
| Registry | Package URL prefix | Method |
|----------|-------------------|--------|
| crates.io | `pkg:cargo/*` | REST API |
| npm | `pkg:npm/*` | Registry API |
| PyPI | `pkg:pypi/*` | JSON API |
| Go modules | `pkg:golang/*` | Known license map (no API) |
| Debian/Ubuntu | `pkg:deb/*` | Known license map |
Components from private registries (e.g., `@openclaw/*` npm packages) are not resolved and will appear in the "unresolved" report.
## Output Files
| Pattern | Description |
|---------|-------------|
| `deploy/sbom/output/openshell-source-{version}.cdx.json` | CycloneDX JSON SBOM |
| `deploy/sbom/output/openshell-source-{version}.csv` | CSV export (name, version, type, purl, licenses, bom-ref) |
## Key Files
| File | Purpose |
|------|---------|
| `deploy/sbom/resolve_licenses.py` | License resolution script |
| `deploy/sbom/sbom_to_csv.py` | JSON-to-CSV converter |
| `tasks/sbom.toml` | Mise task definitions |
| `mise.toml` | Syft tool definition (under `[tools]`) |
## Quick Reference
| Task | Command |
|------|---------|
| Full pipeline | `mise run sbom` |
| Generate only | `mise run sbom:generate` |
| Resolve licenses | `mise run sbom:resolve` |
| Export CSV | `mise run sbom:csv` |
| CI license check | `mise run sbom:check` |
| Process external SBOM | `uv run python deploy/sbom/resolve_licenses.py <file>` |
+3
View File
@@ -184,6 +184,9 @@ _build/
# Docker build artifacts (image tarballs, packaged helm charts)
deploy/docker/.build/
# SBOM generated output (JSON, CSV) — release artifacts, not committed
deploy/sbom/output/
# Local mise settings
mise.local.toml
+15969 -2262
View File
File diff suppressed because it is too large Load Diff
+27
View File
@@ -0,0 +1,27 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# cargo-about configuration for generating third-party license notices.
# See https://embarkstudios.github.io/cargo-about/
# Accepted licenses (SPDX expressions). All permissive/weak-copyleft licenses
# found in this workspace's dependency tree.
accepted = [
"0BSD",
"Apache-2.0",
"Apache-2.0 WITH LLVM-exception",
"BSD-1-Clause",
"BSD-2-Clause",
"BSD-3-Clause",
"BSL-1.0",
"CC0-1.0",
"CDLA-Permissive-2.0",
"ISC",
"LGPL-2.1-or-later",
"MIT",
"MIT-0",
"OpenSSL",
"Unicode-3.0",
"Unlicense",
"Zlib",
]
+538
View File
@@ -0,0 +1,538 @@
#!/usr/bin/env python3
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Resolve missing and hash-based licenses in CycloneDX SBOM JSON files.
Queries public registries:
- crates.io (pkg:cargo/*)
- npm (pkg:npm/*)
- PyPI (pkg:pypi/*)
- Known maps (pkg:golang/*, pkg:deb/*, operating-system, application)
Updates the JSON files in-place, then reports what was resolved.
Usage:
python resolve_licenses.py # resolve all *.cdx.json in deploy/sbom/output/
python resolve_licenses.py file1.json ... # resolve specific files
"""
from __future__ import annotations
import json
import sys
import threading
import time
import urllib.error
import urllib.parse
import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
# ---------------------------------------------------------------------------
# Known licenses for packages that registries won't resolve
# ---------------------------------------------------------------------------
KNOWN_LICENSES: dict[str, str] = {
# libxcrypt -- all variants
"libcrypt-dev": "LGPL-2.1-or-later",
"libcrypt1": "LGPL-2.1-or-later",
# python3-defaults -- all variants
"python3": "PSF-2.0",
"python3-minimal": "PSF-2.0",
"libpython3-stdlib": "PSF-2.0",
"python3-venv": "PSF-2.0",
# Go standard library / well-known
"golang.org/x/crypto": "BSD-3-Clause",
"golang.org/x/mod": "BSD-3-Clause",
"golang.org/x/net": "BSD-3-Clause",
"golang.org/x/sync": "BSD-3-Clause",
"golang.org/x/sys": "BSD-3-Clause",
"golang.org/x/term": "BSD-3-Clause",
"golang.org/x/text": "BSD-3-Clause",
"google.golang.org/grpc": "Apache-2.0",
"google.golang.org/protobuf": "BSD-3-Clause",
"google.golang.org/genproto/googleapis/api": "Apache-2.0",
"google.golang.org/genproto/googleapis/rpc": "Apache-2.0",
"gopkg.in/yaml.v3": "MIT",
"go.yaml.in/yaml/v3": "MIT",
"go.opentelemetry.io/otel": "Apache-2.0",
"go.opentelemetry.io/otel/metric": "Apache-2.0",
"go.opentelemetry.io/otel/trace": "Apache-2.0",
"go.opentelemetry.io/auto/sdk": "Apache-2.0",
"go.mongodb.org/mongo-driver": "Apache-2.0",
# Debian / Ubuntu system packages
"debian": "GPL-2.0-only",
"ubuntu": "GPL-2.0-only",
# Application entries without purl
"Simple Launcher": "Proprietary",
"python": "PSF-2.0",
}
# Well-known Go module licenses (GitHub-based)
GO_KNOWN: dict[str, str] = {
"github.com/AlecAivazis/survey": "MIT",
"github.com/MakeNowJust/heredoc": "MIT",
"github.com/Masterminds/goutils": "Apache-2.0",
"github.com/Masterminds/semver": "MIT",
"github.com/Masterminds/sprig": "MIT",
"github.com/alecthomas/chroma": "MIT",
"github.com/asaskevich/govalidator": "MIT",
"github.com/atotto/clipboard": "BSD-3-Clause",
"github.com/aymanbagabas/go-osc52": "MIT",
"github.com/aymerick/douceur": "MIT",
"github.com/blang/semver": "MIT",
"github.com/briandowns/spinner": "Apache-2.0",
"github.com/catppuccin/go": "MIT",
"github.com/cenkalti/backoff": "MIT",
"github.com/charmbracelet/bubbles": "MIT",
"github.com/charmbracelet/bubbletea": "MIT",
"github.com/charmbracelet/colorprofile": "MIT",
"github.com/charmbracelet/glamour": "MIT",
"github.com/charmbracelet/huh": "MIT",
"github.com/charmbracelet/lipgloss": "MIT",
"github.com/charmbracelet/x": "MIT",
"github.com/cli/browser": "BSD-2-Clause",
"github.com/cli/cli": "MIT",
"github.com/cli/go-gh": "MIT",
"github.com/cli/oauth": "MIT",
"github.com/cli/safeexec": "BSD-2-Clause",
"github.com/cli/shurcooL-graphql": "MIT",
"github.com/containerd/stargz-snapshotter": "Apache-2.0",
"github.com/cyberphone/json-canonicalization": "Apache-2.0",
"github.com/davecgh/go-spew": "ISC",
"github.com/digitorus/pkcs7": "MIT",
"github.com/digitorus/timestamp": "MIT",
"github.com/distribution/reference": "Apache-2.0",
"github.com/dlclark/regexp2": "MIT",
"github.com/docker/cli": "Apache-2.0",
"github.com/docker/distribution": "Apache-2.0",
"github.com/docker/docker-credential-helpers": "MIT",
"github.com/dustin/go-humanize": "MIT",
"github.com/fatih/color": "MIT",
"github.com/gabriel-vasile/mimetype": "MIT",
"github.com/gdamore/encoding": "Apache-2.0",
"github.com/gdamore/tcell": "Apache-2.0",
"github.com/go-logr/logr": "Apache-2.0",
"github.com/go-logr/stdr": "Apache-2.0",
"github.com/go-openapi/analysis": "Apache-2.0",
"github.com/go-openapi/errors": "Apache-2.0",
"github.com/go-openapi/jsonpointer": "Apache-2.0",
"github.com/go-openapi/jsonreference": "Apache-2.0",
"github.com/go-openapi/loads": "Apache-2.0",
"github.com/go-openapi/runtime": "Apache-2.0",
"github.com/go-openapi/spec": "Apache-2.0",
"github.com/go-openapi/strfmt": "Apache-2.0",
"github.com/go-openapi/swag": "Apache-2.0",
"github.com/go-openapi/validate": "Apache-2.0",
"github.com/go-viper/mapstructure": "MIT",
"github.com/godbus/dbus": "BSD-2-Clause",
"github.com/golang/snappy": "BSD-3-Clause",
"github.com/google/certificate-transparency-go": "Apache-2.0",
"github.com/google/go-containerregistry": "Apache-2.0",
"github.com/google/shlex": "Apache-2.0",
"github.com/google/uuid": "BSD-3-Clause",
"github.com/gorilla/css": "BSD-3-Clause",
"github.com/gorilla/websocket": "BSD-2-Clause",
"github.com/grpc-ecosystem/grpc-gateway": "BSD-3-Clause",
"github.com/hashicorp/go-version": "MPL-2.0",
"github.com/henvic/httpretty": "MIT",
"github.com/huandu/xstrings": "MIT",
"github.com/in-toto/attestation": "Apache-2.0",
"github.com/in-toto/in-toto-golang": "Apache-2.0",
"github.com/itchyny/gojq": "MIT",
"github.com/itchyny/timefmt-go": "MIT",
"github.com/joho/godotenv": "MIT",
"github.com/kballard/go-shellquote": "MIT",
"github.com/klauspost/compress": "Apache-2.0",
"github.com/lucasb-eyer/go-colorful": "MIT",
"github.com/mattn/go-colorable": "MIT",
"github.com/mattn/go-isatty": "MIT",
"github.com/mattn/go-runewidth": "MIT",
"github.com/mgutz/ansi": "MIT",
"github.com/microcosm-cc/bluemonday": "BSD-3-Clause",
"github.com/microsoft/dev-tunnels": "MIT",
"github.com/mitchellh/copystructure": "MIT",
"github.com/mitchellh/go-homedir": "MIT",
"github.com/mitchellh/hashstructure": "MIT",
"github.com/mitchellh/reflectwalk": "MIT",
"github.com/muesli/ansi": "MIT",
"github.com/muesli/cancelreader": "MIT",
"github.com/muesli/reflow": "MIT",
"github.com/muesli/termenv": "MIT",
"github.com/muhammadmuzzammil1998/jsonc": "MIT",
"github.com/oklog/ulid": "Apache-2.0",
"github.com/opencontainers/go-digest": "Apache-2.0",
"github.com/opencontainers/image-spec": "Apache-2.0",
"github.com/opentracing/opentracing-go": "Apache-2.0",
"github.com/pkg/errors": "BSD-2-Clause",
"github.com/pmezard/go-difflib": "BSD-3-Clause",
"github.com/rivo/tview": "MIT",
"github.com/rivo/uniseg": "MIT",
"github.com/rodaine/table": "MIT",
"github.com/secure-systems-lab/go-securesystemslib": "MIT",
"github.com/shibumi/go-pathspec": "Apache-2.0",
"github.com/shopspring/decimal": "MIT",
"github.com/shurcooL/githubv4": "MIT",
"github.com/shurcooL/graphql": "MIT",
"github.com/sigstore/protobuf-specs": "Apache-2.0",
"github.com/sigstore/rekor-tiles": "Apache-2.0",
"github.com/sigstore/rekor": "Apache-2.0",
"github.com/sigstore/sigstore-go": "Apache-2.0",
"github.com/sigstore/sigstore": "Apache-2.0",
"github.com/sigstore/timestamp-authority": "Apache-2.0",
"github.com/sirupsen/logrus": "MIT",
"github.com/spf13/cast": "MIT",
"github.com/spf13/cobra": "Apache-2.0",
"github.com/spf13/pflag": "BSD-3-Clause",
"github.com/stretchr/objx": "MIT",
"github.com/stretchr/testify": "MIT",
"github.com/theupdateframework/go-tuf": "MIT",
"github.com/thlib/go-timezone-local": "Unlicense",
"github.com/transparency-dev/formats": "Apache-2.0",
"github.com/transparency-dev/merkle": "Apache-2.0",
"github.com/vbatts/tar-split": "BSD-3-Clause",
"github.com/vmihailenco/msgpack": "BSD-2-Clause",
"github.com/vmihailenco/tagparser": "BSD-2-Clause",
"github.com/xo/terminfo": "MIT",
"github.com/yuin/goldmark-emoji": "MIT",
"github.com/yuin/goldmark": "MIT",
"github.com/zalando/go-keyring": "MIT",
"dario.cat/mergo": "BSD-3-Clause",
}
# Rate-limit helpers (thread-safe)
_last_request: dict[str, float] = {}
_rate_lock = threading.Lock()
def _rate_limit(domain: str, interval: float = 0.15) -> None:
with _rate_lock:
now = time.time()
last = _last_request.get(domain, 0)
wait = interval - (now - last)
if wait > 0:
time.sleep(wait)
_last_request[domain] = time.time()
def _get_json(url: str, domain: str) -> dict | None:
_rate_limit(domain)
req = urllib.request.Request(
url, headers={"User-Agent": "sbom-license-resolver/1.0"}
)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
return json.loads(resp.read())
except (urllib.error.HTTPError, urllib.error.URLError, TimeoutError):
return None
# ---------------------------------------------------------------------------
# Registry lookups
# ---------------------------------------------------------------------------
def lookup_cargo(name: str, version: str) -> str | None:
"""Query crates.io for a crate's license."""
data = _get_json(f"https://crates.io/api/v1/crates/{name}/{version}", "crates.io")
if data and "version" in data:
return data["version"].get("license")
# fallback: try crate-level
data = _get_json(f"https://crates.io/api/v1/crates/{name}", "crates.io")
if data and "crate" in data:
versions = data.get("versions", [])
for v in versions:
if v.get("num") == version:
return v.get("license")
# last resort: latest version license
if versions:
return versions[0].get("license")
return None
def lookup_npm(name: str, version: str) -> str | None:
"""Query npm registry for a package's license."""
encoded = urllib.parse.quote(name, safe="")
data = _get_json(
f"https://registry.npmjs.org/{encoded}/{version}", "registry.npmjs.org"
)
if data:
lic = data.get("license")
if isinstance(lic, dict):
return lic.get("type")
if isinstance(lic, str):
return lic
return None
def lookup_pypi(name: str, version: str) -> str | None:
"""Query PyPI for a package's license."""
data = _get_json(f"https://pypi.org/pypi/{name}/{version}/json", "pypi.org")
if data and "info" in data:
lic = data["info"].get("license")
if lic and len(lic) < 100: # skip full license texts
return lic
# Try classifiers
for c in data["info"].get("classifiers", []):
if c.startswith("License :: OSI Approved :: "):
return c.split(" :: ")[-1]
return None
def resolve_go_name(full_name: str) -> str | None:
"""Resolve a Go module name to a known license via GO_KNOWN map."""
# Try exact match first
if full_name in GO_KNOWN:
return GO_KNOWN[full_name]
# Try stripping version suffix (e.g. github.com/foo/bar/v2 -> github.com/foo/bar)
parts = full_name.split("/")
for i in range(len(parts), 1, -1):
candidate = "/".join(parts[:i])
if candidate in GO_KNOWN:
return GO_KNOWN[candidate]
# Try just org/repo for github.com paths
if full_name.startswith("github.com/") and len(parts) >= 3:
base = "/".join(parts[:3])
if base in GO_KNOWN:
return GO_KNOWN[base]
return None
def resolve_component(comp: dict) -> str | None:
"""Try to resolve a license for a component."""
name = comp.get("name", "")
version = comp.get("version", "")
purl = comp.get("purl", "")
comp_type = comp.get("type", "")
# 1. Check hardcoded known licenses
if name in KNOWN_LICENSES:
return KNOWN_LICENSES[name]
# 2. Route by purl type
if purl.startswith("pkg:cargo/"):
return lookup_cargo(name, version)
if purl.startswith("pkg:npm/"):
return lookup_npm(name, version)
if purl.startswith("pkg:pypi/"):
return lookup_pypi(name, version)
if purl.startswith("pkg:golang/"):
return resolve_go_name(name)
if purl.startswith("pkg:deb/"):
# Check known map by base package name
base = name.split(":")[0] # strip arch qualifier
if base in KNOWN_LICENSES:
return KNOWN_LICENSES[base]
return None
if comp_type == "operating-system":
return KNOWN_LICENSES.get(name)
if not purl:
return KNOWN_LICENSES.get(name)
return None
def set_license(comp: dict, license_id: str) -> None:
"""Set the license on a component, replacing hash or empty."""
comp["licenses"] = [{"license": {"id": license_id}}]
def needs_fix(comp: dict) -> bool:
"""Check if component has missing or hash-based license."""
licenses = comp.get("licenses", [])
if not licenses:
return True
for entry in licenses:
lic = entry.get("license", {})
lid = lic.get("id", "")
lname = lic.get("name", "")
if lid.startswith("sha256:") or lname.startswith("sha256:"):
return True
return False
def _find_sbom_files() -> list[Path]:
"""Find SBOM JSON files in the default output directory."""
repo_root = Path(__file__).resolve().parent.parent.parent
output_dir = repo_root / "deploy" / "sbom" / "output"
return sorted(output_dir.glob("*.cdx.json"))
def _classify_registry(comp: dict) -> str:
"""Return the registry group for a component."""
purl = comp.get("purl", "")
name = comp.get("name", "")
comp_type = comp.get("type", "")
if name in KNOWN_LICENSES:
return "known"
if purl.startswith("pkg:cargo/"):
return "crates.io"
if purl.startswith("pkg:npm/"):
return "npm"
if purl.startswith("pkg:pypi/"):
return "pypi"
if purl.startswith("pkg:golang/"):
return "golang"
if purl.startswith("pkg:deb/"):
return "deb"
if comp_type == "operating-system" or not purl:
return "known"
return "other"
def _resolve_one(key: str, comp: dict) -> tuple[str, str | None]:
"""Resolve a single component, returning (key, license_id | None)."""
return key, resolve_component(comp)
# Concurrency: different registries can run in parallel; within a domain
# the rate limiter serialises requests via the shared lock.
_MAX_WORKERS = 12
def main() -> None:
files = [Path(p) for p in sys.argv[1:]] if len(sys.argv) > 1 else _find_sbom_files()
if not files:
print("No SBOM JSON files found.")
print("Run 'mise run sbom:generate' first, or pass file paths as arguments.")
sys.exit(1)
print(f"Loading {len(files)} SBOM file(s)...")
# Collect unique components needing fixes
to_resolve: dict[str, dict] = {} # key -> representative component
total_components = 0
for f in files:
with f.open() as fh:
sbom = json.load(fh)
components = sbom.get("components", [])
total_components += len(components)
for comp in components:
if needs_fix(comp):
key = f"{comp.get('name', '')}@{comp.get('version', '')}"
if key not in to_resolve:
to_resolve[key] = comp
total = len(to_resolve)
print(f" {total_components} total components, {total} need license resolution")
if total == 0:
print("All licenses already resolved.")
return
# Classify by registry for progress reporting
groups: dict[str, list[tuple[str, dict]]] = {}
for key, comp in to_resolve.items():
registry = _classify_registry(comp)
groups.setdefault(registry, []).append((key, comp))
print("\n Breakdown by registry:")
for registry in sorted(groups):
count = len(groups[registry])
marker = "(local)" if registry in {"known", "golang", "deb"} else "(API)"
print(f" {registry:<12} {count:>5} {marker}")
# Resolve -- local lookups first (instant), then API calls concurrently
resolved: dict[str, str] = {}
failed: list[str] = []
t0 = time.monotonic()
local_registries = {"known", "golang", "deb", "other"}
api_registries = {"crates.io", "npm", "pypi"}
# Phase 1: local (no network)
local_items = [
(key, comp) for reg in local_registries for key, comp in groups.get(reg, [])
]
for key, comp in local_items:
lic = resolve_component(comp)
if lic:
resolved[key] = lic
else:
failed.append(key)
if local_items:
print(
f"\n Local lookups: {len(resolved)} resolved, "
f"{len(failed)} unresolved ({time.monotonic() - t0:.1f}s)"
)
# Phase 2: API calls (concurrent)
api_items = [
(key, comp) for reg in api_registries for key, comp in groups.get(reg, [])
]
if api_items:
api_total = len(api_items)
api_resolved = 0
api_failed = 0
print(
f"\n Resolving {api_total} packages via registry APIs "
f"({_MAX_WORKERS} workers)..."
)
with ThreadPoolExecutor(max_workers=_MAX_WORKERS) as pool:
futures = {
pool.submit(_resolve_one, key, comp): key for key, comp in api_items
}
for done_count, future in enumerate(as_completed(futures), 1):
key, lic = future.result()
if lic:
resolved[key] = lic
api_resolved += 1
else:
failed.append(key)
api_failed += 1
if done_count % 50 == 0 or done_count == api_total:
elapsed = time.monotonic() - t0
sys.stdout.write(
f"\r [{done_count}/{api_total}] "
f"resolved={api_resolved} failed={api_failed} "
f"({elapsed:.1f}s)"
)
sys.stdout.flush()
print() # newline after progress
elapsed = time.monotonic() - t0
print(
f"\n Done: {len(resolved)}/{total} resolved, "
f"{len(failed)} unresolved ({elapsed:.1f}s)"
)
# Apply to all files
total_patched = 0
for f in files:
with f.open() as fh:
sbom = json.load(fh)
patched = 0
for comp in sbom.get("components", []):
if needs_fix(comp):
key = f"{comp.get('name', '')}@{comp.get('version', '')}"
if key in resolved:
set_license(comp, resolved[key])
patched += 1
with f.open("w") as fh:
json.dump(sbom, fh, indent=2)
fh.write("\n")
total_patched += patched
print(f" {f.name}: patched {patched} components")
print(f"\n Total patches applied: {total_patched}")
if failed:
print(f"\n --- Unresolved ({len(failed)}) ---")
for key in sorted(failed):
comp = to_resolve[key]
print(f" {key} purl={comp.get('purl', '(none)')}")
if __name__ == "__main__":
main()
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Convert CycloneDX SBOM JSON files to CSV.
Usage:
python sbom_to_csv.py # convert all *.cdx.json in deploy/sbom/output/
python sbom_to_csv.py file1.json ... # convert specific files
"""
from __future__ import annotations
import csv
import json
import sys
from pathlib import Path
def extract_licenses(component: dict) -> str:
"""Pull license IDs/names from CycloneDX license entries."""
licenses = component.get("licenses", [])
ids = []
for entry in licenses:
lic = entry.get("license", {})
ids.append(lic.get("id") or lic.get("name", ""))
return " | ".join(filter(None, ids))
def sbom_to_csv(json_path: Path) -> Path:
"""Read a CycloneDX JSON SBOM and write a CSV beside it."""
with json_path.open() as f:
sbom = json.load(f)
csv_path = json_path.with_suffix(".csv")
components = sbom.get("components", [])
with csv_path.open("w", newline="") as f:
writer = csv.writer(f)
writer.writerow(["name", "version", "type", "purl", "licenses", "bom-ref"])
for comp in components:
writer.writerow(
[
comp.get("name", ""),
comp.get("version", ""),
comp.get("type", ""),
comp.get("purl", ""),
extract_licenses(comp),
comp.get("bom-ref", ""),
]
)
return csv_path
def _find_sbom_files() -> list[Path]:
"""Find SBOM JSON files in the default output directory."""
repo_root = Path(__file__).resolve().parent.parent.parent
output_dir = repo_root / "deploy" / "sbom" / "output"
return sorted(output_dir.glob("*.cdx.json"))
def main() -> None:
files = [Path(p) for p in sys.argv[1:]] if len(sys.argv) > 1 else _find_sbom_files()
if not files:
print("No SBOM JSON files found.")
print("Run 'mise run sbom:generate' first, or pass file paths as arguments.")
sys.exit(1)
for path in files:
csv_path = sbom_to_csv(path)
with csv_path.open() as count_fh:
components_count = sum(1 for _ in count_fh) - 1 # minus header
print(f"{path.name} -> {csv_path.name} ({components_count} components)")
if __name__ == "__main__":
main()
+3 -1
View File
@@ -19,6 +19,8 @@ uv = "0.10.2"
protoc = "29.6"
helm = "4.1.1"
"ubi:mozilla/sccache" = { version = "0.14.0", matching = "sccache-v" }
"ubi:anchore/syft" = { version = "1.42.2", matching = "syft_" }
"ubi:EmbarkStudios/cargo-about" = "0.8.4"
[env]
_.path = ["{{config_root}}/scripts/bin"]
@@ -42,7 +44,7 @@ DOCKER_BUILDKIT = "1"
[vars]
# Python paths to include in formatting/linting
python_paths = "python/ tasks/scripts/*.py"
python_paths = "python/ tasks/scripts/*.py deploy/sbom/*.py"
[task_config]
includes = ["tasks/*.toml"]
+198 -146
View File
@@ -3,13 +3,16 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Generate THIRD-PARTY-NOTICES from Rust and Python dependency metadata.
"""Generate THIRD-PARTY-NOTICES with full license texts.
Uses cargo-about (Rust) and pip-licenses (Python) to collect third-party
dependency licenses and produce a single attribution file at the repo root.
Usage:
python scripts/generate_third_party_notices.py
uv run python scripts/generate_third_party_notices.py
mise run notices
Writes THIRD-PARTY-NOTICES to the repo root. Requires `cargo` on PATH
for Rust deps. Python deps are read from pyproject.toml.
Requires cargo-about (installed via mise) and pip-licenses (fetched via uv).
"""
from __future__ import annotations
@@ -17,8 +20,34 @@ from __future__ import annotations
import json
import subprocess
import sys
import time
from pathlib import Path
# Our own workspace crates and packages — excluded from notices.
WORKSPACE_CRATES = frozenset(
{
"navigator-bootstrap",
"navigator-cli",
"navigator-core",
"navigator-policy",
"navigator-providers",
"navigator-router",
"navigator-sandbox",
"navigator-server",
"navigator-tui",
"openshell-e2e",
}
)
OWN_PYTHON_PACKAGES = frozenset(
{
"openshell",
}
)
SEPARATOR = "=" * 80
THIN_SEP = "-" * 80
def find_repo_root() -> Path:
"""Walk up from CWD to find the directory containing .git."""
@@ -30,193 +59,216 @@ def find_repo_root() -> Path:
return Path.cwd()
# Workspace member crate names to exclude (these are ours, not third-party).
WORKSPACE_PREFIXES = ("navigator-",)
# ---------------------------------------------------------------------------
# Rust dependencies via cargo-about
# ---------------------------------------------------------------------------
def get_rust_deps_from_cargo_metadata() -> list[dict[str, str]] | None:
"""Try to extract third-party Rust deps via `cargo metadata`."""
def get_rust_notices() -> list[dict]:
"""Run cargo-about and return structured license groups.
Each entry: {id, crates: [{name, version, repository, description}], text}
"""
print(" Running cargo-about generate --format json ...")
try:
result = subprocess.run(
["cargo", "metadata", "--format-version", "1"],
["cargo-about", "generate", "--format", "json"],
capture_output=True,
text=True,
check=True,
)
except (FileNotFoundError, subprocess.CalledProcessError):
return None
meta = json.loads(result.stdout)
workspace_members = set(meta.get("workspace_members", []))
deps = []
for pkg in meta["packages"]:
if pkg["id"] in workspace_members:
continue
if any(pkg["name"].startswith(p) for p in WORKSPACE_PREFIXES):
continue
deps.append({
"name": pkg["name"],
"version": pkg["version"],
"license": pkg.get("license") or "Unknown",
"repository": pkg.get("repository") or "",
})
return sorted(deps, key=lambda d: d["name"].lower())
def get_rust_deps_from_lockfile(root: Path) -> list[dict[str, str]]:
"""Fallback: parse Cargo.lock for name+version (no license info)."""
lockfile = root / "Cargo.lock"
if not lockfile.exists():
except FileNotFoundError:
print(
" WARNING: cargo-about not found, skipping Rust notices", file=sys.stderr
)
return []
except subprocess.CalledProcessError as e:
print(f" WARNING: cargo-about failed: {e.stderr[:200]}", file=sys.stderr)
return []
deps = []
content = lockfile.read_text()
name = None
version = None
for line in content.splitlines():
if line.startswith("name = "):
name = line.split('"')[1]
elif line.startswith("version = ") and '"' in line:
version = line.split('"')[1]
elif line == "[[package]]" or line == "":
if name and version:
if not any(name.startswith(p) for p in WORKSPACE_PREFIXES):
deps.append({
"name": name,
"version": version,
"license": "See crates.io",
"repository": f"https://crates.io/crates/{name}",
})
name = None
version = None
data = json.loads(result.stdout)
groups: list[dict] = []
# Catch the last entry.
if name and version:
if not any(name.startswith(p) for p in WORKSPACE_PREFIXES):
deps.append({
"name": name,
"version": version,
"license": "See crates.io",
"repository": f"https://crates.io/crates/{name}",
})
for lic in data.get("licenses", []):
crates = []
for entry in lic.get("used_by", []):
crate = entry.get("crate", {})
name = crate.get("name", "")
if name in WORKSPACE_CRATES:
continue
crates.append(
{
"name": name,
"version": crate.get("version", ""),
"repository": crate.get("repository", ""),
"description": crate.get("description", ""),
}
)
return sorted(deps, key=lambda d: d["name"].lower())
if not crates:
continue
groups.append(
{
"id": lic.get("id", "Unknown"),
"crates": sorted(crates, key=lambda c: c["name"].lower()),
"text": (lic.get("text") or "").rstrip(),
}
)
return groups
def get_rust_deps(root: Path) -> list[dict[str, str]]:
"""Extract third-party Rust dependencies.
# ---------------------------------------------------------------------------
# Python dependencies via pip-licenses
# ---------------------------------------------------------------------------
Prefers `cargo metadata` for full license info. Falls back to parsing
Cargo.lock when cargo is not available.
def get_python_notices() -> list[dict]:
"""Run pip-licenses and return structured package notices.
Each entry: {name, version, license_id, text}
"""
deps = get_rust_deps_from_cargo_metadata()
if deps is not None:
return deps
print(" cargo not found, falling back to Cargo.lock parsing", file=sys.stderr)
return get_rust_deps_from_lockfile(root)
def get_python_deps(root: Path) -> list[dict[str, str]]:
"""Extract Python dependencies from pyproject.toml [project.dependencies]."""
pyproject = root / "pyproject.toml"
if not pyproject.exists():
print(" Running pip-licenses ...")
try:
result = subprocess.run(
[
"uv",
"run",
"--with",
"pip-licenses",
"pip-licenses",
"--format=json",
"--with-license-file",
"--no-license-path",
],
capture_output=True,
text=True,
check=True,
)
except FileNotFoundError:
print(" WARNING: uv not found, skipping Python notices", file=sys.stderr)
return []
except subprocess.CalledProcessError as e:
print(f" WARNING: pip-licenses failed: {e.stderr[:200]}", file=sys.stderr)
return []
# Simple parser: read the dependencies list without a TOML library.
content = pyproject.read_text()
deps = []
in_deps = False
for line in content.splitlines():
stripped = line.strip()
if stripped.startswith("dependencies = ["):
in_deps = True
packages: list[dict] = []
for pkg in json.loads(result.stdout):
name = pkg.get("Name", "")
if name.lower() in OWN_PYTHON_PACKAGES:
continue
# Skip pip/setuptools/wheel (installer tools, not shipped deps)
if name.lower() in {"pip", "wheel"}:
continue
if in_deps:
if stripped == "]":
break
# Parse "package>=version" style.
dep = stripped.strip('",').strip()
if dep:
# Split on first version specifier.
for sep in (">=", "==", "~=", "!=", "<", ">"):
if sep in dep:
name = dep[:dep.index(sep)].strip()
deps.append({
"name": name,
"version": dep[dep.index(sep):].strip(),
"license": "See PyPI",
"repository": f"https://pypi.org/project/{name}/",
})
break
else:
deps.append({
"name": dep,
"version": "",
"license": "See PyPI",
"repository": f"https://pypi.org/project/{dep}/",
})
return sorted(deps, key=lambda d: d["name"].lower())
packages.append(
{
"name": name,
"version": pkg.get("Version", ""),
"license_id": pkg.get("License", "Unknown"),
"text": (pkg.get("LicenseText") or "").rstrip(),
}
)
return sorted(packages, key=lambda p: p["name"].lower())
def format_notices(rust_deps: list[dict], python_deps: list[dict]) -> str:
"""Format the THIRD-PARTY-NOTICES file content."""
lines = [
"THIRD-PARTY SOFTWARE NOTICES",
# ---------------------------------------------------------------------------
# Output formatting
# ---------------------------------------------------------------------------
def format_notices(
rust_groups: list[dict],
python_packages: list[dict],
) -> str:
"""Format the complete THIRD-PARTY-NOTICES file."""
lines: list[str] = [
"THIRD-PARTY SOFTWARE NOTICES AND INFORMATION",
"",
"This file lists the third-party software packages used by OpenShell,",
"along with their respective licenses.",
"This product includes third-party software components. The following",
"notices and licenses are provided in compliance with the terms of the",
"respective licenses.",
"",
"To regenerate: uv run python scripts/generate_third_party_notices.py",
"To regenerate: mise run notices",
"",
]
if rust_deps:
lines.append("=" * 80)
lines.append("Rust Dependencies")
lines.append("=" * 80)
# --- Rust section ---
if rust_groups:
rust_crate_count = sum(len(g["crates"]) for g in rust_groups)
lines.append(SEPARATOR)
lines.append(f"Rust Dependencies ({rust_crate_count} packages)")
lines.append(SEPARATOR)
lines.append("")
for dep in rust_deps:
lines.append(f"Package: {dep['name']} {dep['version']}")
lines.append(f"License: {dep['license']}")
if dep["repository"]:
lines.append(f"Repository: {dep['repository']}")
for group in rust_groups:
lines.append(SEPARATOR)
lines.append(f"License: {group['id']}")
lines.append(THIN_SEP)
lines.append("")
lines.append("Used by:")
for crate in group["crates"]:
repo = f" ({crate['repository']})" if crate["repository"] else ""
lines.append(f" - {crate['name']} {crate['version']}{repo}")
lines.append("")
if python_deps:
lines.append("=" * 80)
lines.append("Python Dependencies")
lines.append("=" * 80)
if group["text"]:
lines.append(group["text"])
lines.append("")
# --- Python section ---
if python_packages:
lines.append(SEPARATOR)
lines.append(f"Python Dependencies ({len(python_packages)} packages)")
lines.append(SEPARATOR)
lines.append("")
for dep in python_deps:
version_str = f" {dep['version']}" if dep["version"] else ""
lines.append(f"Package: {dep['name']}{version_str}")
lines.append(f"License: {dep['license']}")
if dep["repository"]:
lines.append(f"Repository: {dep['repository']}")
for pkg in python_packages:
lines.append(SEPARATOR)
lines.append(f"{pkg['name']} {pkg['version']}")
lines.append(f"License: {pkg['license_id']}")
lines.append(THIN_SEP)
lines.append("")
if pkg["text"]:
lines.append(pkg["text"])
lines.append("")
return "\n".join(lines)
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main() -> int:
root = find_repo_root()
t0 = time.monotonic()
print("Generating third-party notices...")
print()
print("Collecting Rust dependencies...")
rust_deps = get_rust_deps(root)
print(f" Found {len(rust_deps)} Rust dependencies")
rust_groups = get_rust_notices()
rust_count = sum(len(g["crates"]) for g in rust_groups)
print(f" {rust_count} Rust packages across {len(rust_groups)} license groups")
print()
print("Collecting Python dependencies...")
python_deps = get_python_deps(root)
print(f" Found {len(python_deps)} Python dependencies")
python_packages = get_python_notices()
print(f" {len(python_packages)} Python packages")
print()
notices = format_notices(rust_deps, python_deps)
notices = format_notices(rust_groups, python_packages)
output = root / "THIRD-PARTY-NOTICES"
output.write_text(notices)
print(f"Wrote {output}")
elapsed = time.monotonic() - t0
line_count = notices.count("\n") + 1
print(f"Wrote {output.name} ({line_count} lines, {len(notices)} bytes)")
print(f"Done in {elapsed:.1f}s")
return 0
+11
View File
@@ -0,0 +1,11 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# Third-party license notice generation tasks
#
# Usage:
# mise run notices # regenerate THIRD-PARTY-NOTICES at repo root
[notices]
description = "Regenerate THIRD-PARTY-NOTICES with full license texts"
run = "uv run python scripts/generate_third_party_notices.py"
+101
View File
@@ -0,0 +1,101 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# SBOM generation, license resolution, and CSV export tasks
#
# Usage:
# mise run sbom # generate SBOMs, resolve licenses, export CSVs
# mise run sbom:check # advisory license check (for CI)
[sbom]
description = "Generate SBOMs, resolve licenses, and export CSVs to deploy/sbom/output/"
depends = ["sbom:csv"]
["sbom:generate"]
description = "Generate CycloneDX SBOMs with Syft"
hide = true
run = """
#!/usr/bin/env bash
set -euo pipefail
OUTPUT_DIR="deploy/sbom/output"
mkdir -p "$OUTPUT_DIR"
VERSION=$(uv run python tasks/scripts/release.py get-version --cargo)
echo "Generating SBOM for workspace (version ${VERSION})..."
syft dir:. \
--exclude './.github/**' \
--exclude './.venv/**' \
--exclude './.cache/**' \
--output "cyclonedx-json=$OUTPUT_DIR/openshell-source-${VERSION}.cdx.json" \
--source-name openshell \
--source-version "$VERSION"
echo ""
echo "SBOM written to $OUTPUT_DIR/"
ls -la "$OUTPUT_DIR"/*.cdx.json
"""
["sbom:resolve"]
description = "Resolve missing licenses in SBOM JSON files via public registries"
depends = ["sbom:generate"]
hide = true
run = "uv run python deploy/sbom/resolve_licenses.py"
["sbom:csv"]
description = "Convert SBOM JSON files to CSV"
depends = ["sbom:resolve"]
hide = true
run = "uv run python deploy/sbom/sbom_to_csv.py"
["sbom:check"]
description = "Check SBOMs for unresolved licenses (advisory, non-blocking)"
hide = true
run = """
#!/usr/bin/env bash
set -euo pipefail
OUTPUT_DIR="deploy/sbom/output"
if [ ! -d "$OUTPUT_DIR" ] || [ -z "$(ls -A "$OUTPUT_DIR"/*.cdx.json 2>/dev/null)" ]; then
echo "No SBOM files found in $OUTPUT_DIR/. Run 'mise run sbom' first."
exit 0
fi
echo "Checking for unresolved licenses..."
UNRESOLVED=0
for f in "$OUTPUT_DIR"/*.cdx.json; do
COUNT=$(uv run python -c "
import json, sys
with open('$f') as fh:
sbom = json.load(fh)
missing = 0
for c in sbom.get('components', []):
lics = c.get('licenses', [])
if not lics:
missing += 1
continue
for e in lics:
lid = e.get('license', {}).get('id', '')
lname = e.get('license', {}).get('name', '')
if lid.startswith('sha256:') or lname.startswith('sha256:'):
missing += 1
break
print(missing)
")
if [ "$COUNT" -gt 0 ]; then
echo " $(basename "$f"): $COUNT components with unresolved licenses"
UNRESOLVED=$((UNRESOLVED + COUNT))
fi
done
if [ "$UNRESOLVED" -gt 0 ]; then
echo ""
echo "WARNING: $UNRESOLVED total components with unresolved licenses."
echo "This is advisory -- not blocking the build."
else
echo "All licenses resolved."
fi
"""