mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-01 23:28:42 +08:00
feat(sbom): add SBOM generation, license resolution, and CSV export tooling (#239)
* feat(sbom): add SBOM generation, license resolution, and CSV export tooling Add mise-integrated SBOM pipeline for container images using Syft. Includes license resolution via crates.io/npm/PyPI APIs and CycloneDX JSON to CSV conversion. Adds agent skill for on-demand SBOM operations. Closes #237 * fix(sbom): chain task dependencies to run generate → resolve → csv sequentially * fix(sbom): add concurrent license resolution, progress logging, and exclude dev artifacts * feat(notices): add mise run notices to generate THIRD-PARTY-NOTICES with full license texts Use cargo-about for Rust crate licenses and pip-licenses for Python packages. Produces a single attribution file with per-package copyright notices and full license text for open-source compliance.
This commit is contained in:
@@ -0,0 +1,105 @@
|
||||
---
|
||||
name: sbom
|
||||
description: Generate and manage Software Bill of Materials (SBOMs) for the OpenShell project. Covers SBOM generation with Syft, license resolution via public registries, and CSV export for compliance review. Trigger keywords - SBOM, sbom, bill of materials, license audit, license resolution, generate sbom, sbom csv, dependency license, supply chain, license scan.
|
||||
---
|
||||
|
||||
# SBOM Generation and License Resolution
|
||||
|
||||
Generate CycloneDX SBOMs, resolve missing licenses, and export to CSV for compliance review.
|
||||
|
||||
## Overview
|
||||
|
||||
The OpenShell SBOM tooling produces CycloneDX JSON SBOMs using Syft, resolves missing or hash-based licenses by querying public registries (crates.io, npm, PyPI), and exports the results to CSV for stakeholder review.
|
||||
|
||||
SBOMs are **release artifacts only** -- they are generated on demand and not committed to the repository. Output lands in `deploy/sbom/output/` (gitignored).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `mise install` has been run (installs Syft and other tools)
|
||||
- The repository is checked out at the root
|
||||
|
||||
## Workflow 1: Full SBOM Generation (One Command)
|
||||
|
||||
```bash
|
||||
mise run sbom
|
||||
```
|
||||
|
||||
This single command chains three stages:
|
||||
|
||||
1. **Generate** (`sbom:generate`): Syft scans the workspace source tree and produces a CycloneDX JSON SBOM
|
||||
2. **Resolve** (`sbom:resolve`): Public registry APIs fill in missing or hash-based licenses in the JSON
|
||||
3. **CSV** (`sbom:csv`): JSON SBOMs are converted to CSV for review
|
||||
|
||||
Output directory: `deploy/sbom/output/`
|
||||
|
||||
After running, the user can find:
|
||||
- `deploy/sbom/output/*.cdx.json` -- full CycloneDX SBOMs
|
||||
- `deploy/sbom/output/*.csv` -- CSV exports ready for spreadsheet review
|
||||
|
||||
## Workflow 2: Individual Stages
|
||||
|
||||
Run stages independently when debugging or iterating:
|
||||
|
||||
```bash
|
||||
mise run sbom:generate # Generate JSON SBOMs only (requires Syft)
|
||||
mise run sbom:resolve # Resolve licenses in existing JSONs (queries APIs)
|
||||
mise run sbom:csv # Convert existing JSONs to CSV
|
||||
```
|
||||
|
||||
## Workflow 3: License Check (CI Advisory)
|
||||
|
||||
```bash
|
||||
mise run sbom:check
|
||||
```
|
||||
|
||||
Reports unresolved licenses without failing. Intended for PR CI as a non-blocking advisory check. Requires that SBOMs have already been generated (`mise run sbom:generate`).
|
||||
|
||||
## Workflow 4: Processing External SBOMs
|
||||
|
||||
The Python scripts accept explicit file paths, so they can process SBOMs from any source (e.g., NVIDIA nSpect pipeline output):
|
||||
|
||||
```bash
|
||||
uv run python deploy/sbom/resolve_licenses.py /path/to/external-sbom.json
|
||||
uv run python deploy/sbom/sbom_to_csv.py /path/to/external-sbom.json
|
||||
```
|
||||
|
||||
## License Resolution Details
|
||||
|
||||
The resolver queries these public registries:
|
||||
|
||||
| Registry | Package URL prefix | Method |
|
||||
|----------|-------------------|--------|
|
||||
| crates.io | `pkg:cargo/*` | REST API |
|
||||
| npm | `pkg:npm/*` | Registry API |
|
||||
| PyPI | `pkg:pypi/*` | JSON API |
|
||||
| Go modules | `pkg:golang/*` | Known license map (no API) |
|
||||
| Debian/Ubuntu | `pkg:deb/*` | Known license map |
|
||||
|
||||
Components from private registries (e.g., `@openclaw/*` npm packages) are not resolved and will appear in the "unresolved" report.
|
||||
|
||||
## Output Files
|
||||
|
||||
| Pattern | Description |
|
||||
|---------|-------------|
|
||||
| `deploy/sbom/output/openshell-source-{version}.cdx.json` | CycloneDX JSON SBOM |
|
||||
| `deploy/sbom/output/openshell-source-{version}.csv` | CSV export (name, version, type, purl, licenses, bom-ref) |
|
||||
|
||||
## Key Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `deploy/sbom/resolve_licenses.py` | License resolution script |
|
||||
| `deploy/sbom/sbom_to_csv.py` | JSON-to-CSV converter |
|
||||
| `tasks/sbom.toml` | Mise task definitions |
|
||||
| `mise.toml` | Syft tool definition (under `[tools]`) |
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Task | Command |
|
||||
|------|---------|
|
||||
| Full pipeline | `mise run sbom` |
|
||||
| Generate only | `mise run sbom:generate` |
|
||||
| Resolve licenses | `mise run sbom:resolve` |
|
||||
| Export CSV | `mise run sbom:csv` |
|
||||
| CI license check | `mise run sbom:check` |
|
||||
| Process external SBOM | `uv run python deploy/sbom/resolve_licenses.py <file>` |
|
||||
@@ -184,6 +184,9 @@ _build/
|
||||
# Docker build artifacts (image tarballs, packaged helm charts)
|
||||
deploy/docker/.build/
|
||||
|
||||
# SBOM generated output (JSON, CSV) — release artifacts, not committed
|
||||
deploy/sbom/output/
|
||||
|
||||
# Local mise settings
|
||||
mise.local.toml
|
||||
|
||||
|
||||
+15969
-2262
File diff suppressed because it is too large
Load Diff
+27
@@ -0,0 +1,27 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# cargo-about configuration for generating third-party license notices.
|
||||
# See https://embarkstudios.github.io/cargo-about/
|
||||
|
||||
# Accepted licenses (SPDX expressions). All permissive/weak-copyleft licenses
|
||||
# found in this workspace's dependency tree.
|
||||
accepted = [
|
||||
"0BSD",
|
||||
"Apache-2.0",
|
||||
"Apache-2.0 WITH LLVM-exception",
|
||||
"BSD-1-Clause",
|
||||
"BSD-2-Clause",
|
||||
"BSD-3-Clause",
|
||||
"BSL-1.0",
|
||||
"CC0-1.0",
|
||||
"CDLA-Permissive-2.0",
|
||||
"ISC",
|
||||
"LGPL-2.1-or-later",
|
||||
"MIT",
|
||||
"MIT-0",
|
||||
"OpenSSL",
|
||||
"Unicode-3.0",
|
||||
"Unlicense",
|
||||
"Zlib",
|
||||
]
|
||||
@@ -0,0 +1,538 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
"""Resolve missing and hash-based licenses in CycloneDX SBOM JSON files.
|
||||
|
||||
Queries public registries:
|
||||
- crates.io (pkg:cargo/*)
|
||||
- npm (pkg:npm/*)
|
||||
- PyPI (pkg:pypi/*)
|
||||
- Known maps (pkg:golang/*, pkg:deb/*, operating-system, application)
|
||||
|
||||
Updates the JSON files in-place, then reports what was resolved.
|
||||
|
||||
Usage:
|
||||
python resolve_licenses.py # resolve all *.cdx.json in deploy/sbom/output/
|
||||
python resolve_licenses.py file1.json ... # resolve specific files
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Known licenses for packages that registries won't resolve
|
||||
# ---------------------------------------------------------------------------
|
||||
KNOWN_LICENSES: dict[str, str] = {
|
||||
# libxcrypt -- all variants
|
||||
"libcrypt-dev": "LGPL-2.1-or-later",
|
||||
"libcrypt1": "LGPL-2.1-or-later",
|
||||
# python3-defaults -- all variants
|
||||
"python3": "PSF-2.0",
|
||||
"python3-minimal": "PSF-2.0",
|
||||
"libpython3-stdlib": "PSF-2.0",
|
||||
"python3-venv": "PSF-2.0",
|
||||
# Go standard library / well-known
|
||||
"golang.org/x/crypto": "BSD-3-Clause",
|
||||
"golang.org/x/mod": "BSD-3-Clause",
|
||||
"golang.org/x/net": "BSD-3-Clause",
|
||||
"golang.org/x/sync": "BSD-3-Clause",
|
||||
"golang.org/x/sys": "BSD-3-Clause",
|
||||
"golang.org/x/term": "BSD-3-Clause",
|
||||
"golang.org/x/text": "BSD-3-Clause",
|
||||
"google.golang.org/grpc": "Apache-2.0",
|
||||
"google.golang.org/protobuf": "BSD-3-Clause",
|
||||
"google.golang.org/genproto/googleapis/api": "Apache-2.0",
|
||||
"google.golang.org/genproto/googleapis/rpc": "Apache-2.0",
|
||||
"gopkg.in/yaml.v3": "MIT",
|
||||
"go.yaml.in/yaml/v3": "MIT",
|
||||
"go.opentelemetry.io/otel": "Apache-2.0",
|
||||
"go.opentelemetry.io/otel/metric": "Apache-2.0",
|
||||
"go.opentelemetry.io/otel/trace": "Apache-2.0",
|
||||
"go.opentelemetry.io/auto/sdk": "Apache-2.0",
|
||||
"go.mongodb.org/mongo-driver": "Apache-2.0",
|
||||
# Debian / Ubuntu system packages
|
||||
"debian": "GPL-2.0-only",
|
||||
"ubuntu": "GPL-2.0-only",
|
||||
# Application entries without purl
|
||||
"Simple Launcher": "Proprietary",
|
||||
"python": "PSF-2.0",
|
||||
}
|
||||
|
||||
# Well-known Go module licenses (GitHub-based)
|
||||
GO_KNOWN: dict[str, str] = {
|
||||
"github.com/AlecAivazis/survey": "MIT",
|
||||
"github.com/MakeNowJust/heredoc": "MIT",
|
||||
"github.com/Masterminds/goutils": "Apache-2.0",
|
||||
"github.com/Masterminds/semver": "MIT",
|
||||
"github.com/Masterminds/sprig": "MIT",
|
||||
"github.com/alecthomas/chroma": "MIT",
|
||||
"github.com/asaskevich/govalidator": "MIT",
|
||||
"github.com/atotto/clipboard": "BSD-3-Clause",
|
||||
"github.com/aymanbagabas/go-osc52": "MIT",
|
||||
"github.com/aymerick/douceur": "MIT",
|
||||
"github.com/blang/semver": "MIT",
|
||||
"github.com/briandowns/spinner": "Apache-2.0",
|
||||
"github.com/catppuccin/go": "MIT",
|
||||
"github.com/cenkalti/backoff": "MIT",
|
||||
"github.com/charmbracelet/bubbles": "MIT",
|
||||
"github.com/charmbracelet/bubbletea": "MIT",
|
||||
"github.com/charmbracelet/colorprofile": "MIT",
|
||||
"github.com/charmbracelet/glamour": "MIT",
|
||||
"github.com/charmbracelet/huh": "MIT",
|
||||
"github.com/charmbracelet/lipgloss": "MIT",
|
||||
"github.com/charmbracelet/x": "MIT",
|
||||
"github.com/cli/browser": "BSD-2-Clause",
|
||||
"github.com/cli/cli": "MIT",
|
||||
"github.com/cli/go-gh": "MIT",
|
||||
"github.com/cli/oauth": "MIT",
|
||||
"github.com/cli/safeexec": "BSD-2-Clause",
|
||||
"github.com/cli/shurcooL-graphql": "MIT",
|
||||
"github.com/containerd/stargz-snapshotter": "Apache-2.0",
|
||||
"github.com/cyberphone/json-canonicalization": "Apache-2.0",
|
||||
"github.com/davecgh/go-spew": "ISC",
|
||||
"github.com/digitorus/pkcs7": "MIT",
|
||||
"github.com/digitorus/timestamp": "MIT",
|
||||
"github.com/distribution/reference": "Apache-2.0",
|
||||
"github.com/dlclark/regexp2": "MIT",
|
||||
"github.com/docker/cli": "Apache-2.0",
|
||||
"github.com/docker/distribution": "Apache-2.0",
|
||||
"github.com/docker/docker-credential-helpers": "MIT",
|
||||
"github.com/dustin/go-humanize": "MIT",
|
||||
"github.com/fatih/color": "MIT",
|
||||
"github.com/gabriel-vasile/mimetype": "MIT",
|
||||
"github.com/gdamore/encoding": "Apache-2.0",
|
||||
"github.com/gdamore/tcell": "Apache-2.0",
|
||||
"github.com/go-logr/logr": "Apache-2.0",
|
||||
"github.com/go-logr/stdr": "Apache-2.0",
|
||||
"github.com/go-openapi/analysis": "Apache-2.0",
|
||||
"github.com/go-openapi/errors": "Apache-2.0",
|
||||
"github.com/go-openapi/jsonpointer": "Apache-2.0",
|
||||
"github.com/go-openapi/jsonreference": "Apache-2.0",
|
||||
"github.com/go-openapi/loads": "Apache-2.0",
|
||||
"github.com/go-openapi/runtime": "Apache-2.0",
|
||||
"github.com/go-openapi/spec": "Apache-2.0",
|
||||
"github.com/go-openapi/strfmt": "Apache-2.0",
|
||||
"github.com/go-openapi/swag": "Apache-2.0",
|
||||
"github.com/go-openapi/validate": "Apache-2.0",
|
||||
"github.com/go-viper/mapstructure": "MIT",
|
||||
"github.com/godbus/dbus": "BSD-2-Clause",
|
||||
"github.com/golang/snappy": "BSD-3-Clause",
|
||||
"github.com/google/certificate-transparency-go": "Apache-2.0",
|
||||
"github.com/google/go-containerregistry": "Apache-2.0",
|
||||
"github.com/google/shlex": "Apache-2.0",
|
||||
"github.com/google/uuid": "BSD-3-Clause",
|
||||
"github.com/gorilla/css": "BSD-3-Clause",
|
||||
"github.com/gorilla/websocket": "BSD-2-Clause",
|
||||
"github.com/grpc-ecosystem/grpc-gateway": "BSD-3-Clause",
|
||||
"github.com/hashicorp/go-version": "MPL-2.0",
|
||||
"github.com/henvic/httpretty": "MIT",
|
||||
"github.com/huandu/xstrings": "MIT",
|
||||
"github.com/in-toto/attestation": "Apache-2.0",
|
||||
"github.com/in-toto/in-toto-golang": "Apache-2.0",
|
||||
"github.com/itchyny/gojq": "MIT",
|
||||
"github.com/itchyny/timefmt-go": "MIT",
|
||||
"github.com/joho/godotenv": "MIT",
|
||||
"github.com/kballard/go-shellquote": "MIT",
|
||||
"github.com/klauspost/compress": "Apache-2.0",
|
||||
"github.com/lucasb-eyer/go-colorful": "MIT",
|
||||
"github.com/mattn/go-colorable": "MIT",
|
||||
"github.com/mattn/go-isatty": "MIT",
|
||||
"github.com/mattn/go-runewidth": "MIT",
|
||||
"github.com/mgutz/ansi": "MIT",
|
||||
"github.com/microcosm-cc/bluemonday": "BSD-3-Clause",
|
||||
"github.com/microsoft/dev-tunnels": "MIT",
|
||||
"github.com/mitchellh/copystructure": "MIT",
|
||||
"github.com/mitchellh/go-homedir": "MIT",
|
||||
"github.com/mitchellh/hashstructure": "MIT",
|
||||
"github.com/mitchellh/reflectwalk": "MIT",
|
||||
"github.com/muesli/ansi": "MIT",
|
||||
"github.com/muesli/cancelreader": "MIT",
|
||||
"github.com/muesli/reflow": "MIT",
|
||||
"github.com/muesli/termenv": "MIT",
|
||||
"github.com/muhammadmuzzammil1998/jsonc": "MIT",
|
||||
"github.com/oklog/ulid": "Apache-2.0",
|
||||
"github.com/opencontainers/go-digest": "Apache-2.0",
|
||||
"github.com/opencontainers/image-spec": "Apache-2.0",
|
||||
"github.com/opentracing/opentracing-go": "Apache-2.0",
|
||||
"github.com/pkg/errors": "BSD-2-Clause",
|
||||
"github.com/pmezard/go-difflib": "BSD-3-Clause",
|
||||
"github.com/rivo/tview": "MIT",
|
||||
"github.com/rivo/uniseg": "MIT",
|
||||
"github.com/rodaine/table": "MIT",
|
||||
"github.com/secure-systems-lab/go-securesystemslib": "MIT",
|
||||
"github.com/shibumi/go-pathspec": "Apache-2.0",
|
||||
"github.com/shopspring/decimal": "MIT",
|
||||
"github.com/shurcooL/githubv4": "MIT",
|
||||
"github.com/shurcooL/graphql": "MIT",
|
||||
"github.com/sigstore/protobuf-specs": "Apache-2.0",
|
||||
"github.com/sigstore/rekor-tiles": "Apache-2.0",
|
||||
"github.com/sigstore/rekor": "Apache-2.0",
|
||||
"github.com/sigstore/sigstore-go": "Apache-2.0",
|
||||
"github.com/sigstore/sigstore": "Apache-2.0",
|
||||
"github.com/sigstore/timestamp-authority": "Apache-2.0",
|
||||
"github.com/sirupsen/logrus": "MIT",
|
||||
"github.com/spf13/cast": "MIT",
|
||||
"github.com/spf13/cobra": "Apache-2.0",
|
||||
"github.com/spf13/pflag": "BSD-3-Clause",
|
||||
"github.com/stretchr/objx": "MIT",
|
||||
"github.com/stretchr/testify": "MIT",
|
||||
"github.com/theupdateframework/go-tuf": "MIT",
|
||||
"github.com/thlib/go-timezone-local": "Unlicense",
|
||||
"github.com/transparency-dev/formats": "Apache-2.0",
|
||||
"github.com/transparency-dev/merkle": "Apache-2.0",
|
||||
"github.com/vbatts/tar-split": "BSD-3-Clause",
|
||||
"github.com/vmihailenco/msgpack": "BSD-2-Clause",
|
||||
"github.com/vmihailenco/tagparser": "BSD-2-Clause",
|
||||
"github.com/xo/terminfo": "MIT",
|
||||
"github.com/yuin/goldmark-emoji": "MIT",
|
||||
"github.com/yuin/goldmark": "MIT",
|
||||
"github.com/zalando/go-keyring": "MIT",
|
||||
"dario.cat/mergo": "BSD-3-Clause",
|
||||
}
|
||||
|
||||
# Rate-limit helpers (thread-safe)
|
||||
_last_request: dict[str, float] = {}
|
||||
_rate_lock = threading.Lock()
|
||||
|
||||
|
||||
def _rate_limit(domain: str, interval: float = 0.15) -> None:
|
||||
with _rate_lock:
|
||||
now = time.time()
|
||||
last = _last_request.get(domain, 0)
|
||||
wait = interval - (now - last)
|
||||
if wait > 0:
|
||||
time.sleep(wait)
|
||||
_last_request[domain] = time.time()
|
||||
|
||||
|
||||
def _get_json(url: str, domain: str) -> dict | None:
|
||||
_rate_limit(domain)
|
||||
req = urllib.request.Request(
|
||||
url, headers={"User-Agent": "sbom-license-resolver/1.0"}
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=10) as resp:
|
||||
return json.loads(resp.read())
|
||||
except (urllib.error.HTTPError, urllib.error.URLError, TimeoutError):
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Registry lookups
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def lookup_cargo(name: str, version: str) -> str | None:
|
||||
"""Query crates.io for a crate's license."""
|
||||
data = _get_json(f"https://crates.io/api/v1/crates/{name}/{version}", "crates.io")
|
||||
if data and "version" in data:
|
||||
return data["version"].get("license")
|
||||
# fallback: try crate-level
|
||||
data = _get_json(f"https://crates.io/api/v1/crates/{name}", "crates.io")
|
||||
if data and "crate" in data:
|
||||
versions = data.get("versions", [])
|
||||
for v in versions:
|
||||
if v.get("num") == version:
|
||||
return v.get("license")
|
||||
# last resort: latest version license
|
||||
if versions:
|
||||
return versions[0].get("license")
|
||||
return None
|
||||
|
||||
|
||||
def lookup_npm(name: str, version: str) -> str | None:
|
||||
"""Query npm registry for a package's license."""
|
||||
encoded = urllib.parse.quote(name, safe="")
|
||||
data = _get_json(
|
||||
f"https://registry.npmjs.org/{encoded}/{version}", "registry.npmjs.org"
|
||||
)
|
||||
if data:
|
||||
lic = data.get("license")
|
||||
if isinstance(lic, dict):
|
||||
return lic.get("type")
|
||||
if isinstance(lic, str):
|
||||
return lic
|
||||
return None
|
||||
|
||||
|
||||
def lookup_pypi(name: str, version: str) -> str | None:
|
||||
"""Query PyPI for a package's license."""
|
||||
data = _get_json(f"https://pypi.org/pypi/{name}/{version}/json", "pypi.org")
|
||||
if data and "info" in data:
|
||||
lic = data["info"].get("license")
|
||||
if lic and len(lic) < 100: # skip full license texts
|
||||
return lic
|
||||
# Try classifiers
|
||||
for c in data["info"].get("classifiers", []):
|
||||
if c.startswith("License :: OSI Approved :: "):
|
||||
return c.split(" :: ")[-1]
|
||||
return None
|
||||
|
||||
|
||||
def resolve_go_name(full_name: str) -> str | None:
|
||||
"""Resolve a Go module name to a known license via GO_KNOWN map."""
|
||||
# Try exact match first
|
||||
if full_name in GO_KNOWN:
|
||||
return GO_KNOWN[full_name]
|
||||
# Try stripping version suffix (e.g. github.com/foo/bar/v2 -> github.com/foo/bar)
|
||||
parts = full_name.split("/")
|
||||
for i in range(len(parts), 1, -1):
|
||||
candidate = "/".join(parts[:i])
|
||||
if candidate in GO_KNOWN:
|
||||
return GO_KNOWN[candidate]
|
||||
# Try just org/repo for github.com paths
|
||||
if full_name.startswith("github.com/") and len(parts) >= 3:
|
||||
base = "/".join(parts[:3])
|
||||
if base in GO_KNOWN:
|
||||
return GO_KNOWN[base]
|
||||
return None
|
||||
|
||||
|
||||
def resolve_component(comp: dict) -> str | None:
|
||||
"""Try to resolve a license for a component."""
|
||||
name = comp.get("name", "")
|
||||
version = comp.get("version", "")
|
||||
purl = comp.get("purl", "")
|
||||
comp_type = comp.get("type", "")
|
||||
|
||||
# 1. Check hardcoded known licenses
|
||||
if name in KNOWN_LICENSES:
|
||||
return KNOWN_LICENSES[name]
|
||||
|
||||
# 2. Route by purl type
|
||||
if purl.startswith("pkg:cargo/"):
|
||||
return lookup_cargo(name, version)
|
||||
if purl.startswith("pkg:npm/"):
|
||||
return lookup_npm(name, version)
|
||||
if purl.startswith("pkg:pypi/"):
|
||||
return lookup_pypi(name, version)
|
||||
if purl.startswith("pkg:golang/"):
|
||||
return resolve_go_name(name)
|
||||
if purl.startswith("pkg:deb/"):
|
||||
# Check known map by base package name
|
||||
base = name.split(":")[0] # strip arch qualifier
|
||||
if base in KNOWN_LICENSES:
|
||||
return KNOWN_LICENSES[base]
|
||||
return None
|
||||
if comp_type == "operating-system":
|
||||
return KNOWN_LICENSES.get(name)
|
||||
if not purl:
|
||||
return KNOWN_LICENSES.get(name)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def set_license(comp: dict, license_id: str) -> None:
|
||||
"""Set the license on a component, replacing hash or empty."""
|
||||
comp["licenses"] = [{"license": {"id": license_id}}]
|
||||
|
||||
|
||||
def needs_fix(comp: dict) -> bool:
|
||||
"""Check if component has missing or hash-based license."""
|
||||
licenses = comp.get("licenses", [])
|
||||
if not licenses:
|
||||
return True
|
||||
for entry in licenses:
|
||||
lic = entry.get("license", {})
|
||||
lid = lic.get("id", "")
|
||||
lname = lic.get("name", "")
|
||||
if lid.startswith("sha256:") or lname.startswith("sha256:"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _find_sbom_files() -> list[Path]:
|
||||
"""Find SBOM JSON files in the default output directory."""
|
||||
repo_root = Path(__file__).resolve().parent.parent.parent
|
||||
output_dir = repo_root / "deploy" / "sbom" / "output"
|
||||
return sorted(output_dir.glob("*.cdx.json"))
|
||||
|
||||
|
||||
def _classify_registry(comp: dict) -> str:
|
||||
"""Return the registry group for a component."""
|
||||
purl = comp.get("purl", "")
|
||||
name = comp.get("name", "")
|
||||
comp_type = comp.get("type", "")
|
||||
|
||||
if name in KNOWN_LICENSES:
|
||||
return "known"
|
||||
if purl.startswith("pkg:cargo/"):
|
||||
return "crates.io"
|
||||
if purl.startswith("pkg:npm/"):
|
||||
return "npm"
|
||||
if purl.startswith("pkg:pypi/"):
|
||||
return "pypi"
|
||||
if purl.startswith("pkg:golang/"):
|
||||
return "golang"
|
||||
if purl.startswith("pkg:deb/"):
|
||||
return "deb"
|
||||
if comp_type == "operating-system" or not purl:
|
||||
return "known"
|
||||
return "other"
|
||||
|
||||
|
||||
def _resolve_one(key: str, comp: dict) -> tuple[str, str | None]:
|
||||
"""Resolve a single component, returning (key, license_id | None)."""
|
||||
return key, resolve_component(comp)
|
||||
|
||||
|
||||
# Concurrency: different registries can run in parallel; within a domain
|
||||
# the rate limiter serialises requests via the shared lock.
|
||||
_MAX_WORKERS = 12
|
||||
|
||||
|
||||
def main() -> None:
|
||||
files = [Path(p) for p in sys.argv[1:]] if len(sys.argv) > 1 else _find_sbom_files()
|
||||
|
||||
if not files:
|
||||
print("No SBOM JSON files found.")
|
||||
print("Run 'mise run sbom:generate' first, or pass file paths as arguments.")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"Loading {len(files)} SBOM file(s)...")
|
||||
|
||||
# Collect unique components needing fixes
|
||||
to_resolve: dict[str, dict] = {} # key -> representative component
|
||||
total_components = 0
|
||||
for f in files:
|
||||
with f.open() as fh:
|
||||
sbom = json.load(fh)
|
||||
components = sbom.get("components", [])
|
||||
total_components += len(components)
|
||||
for comp in components:
|
||||
if needs_fix(comp):
|
||||
key = f"{comp.get('name', '')}@{comp.get('version', '')}"
|
||||
if key not in to_resolve:
|
||||
to_resolve[key] = comp
|
||||
|
||||
total = len(to_resolve)
|
||||
print(f" {total_components} total components, {total} need license resolution")
|
||||
|
||||
if total == 0:
|
||||
print("All licenses already resolved.")
|
||||
return
|
||||
|
||||
# Classify by registry for progress reporting
|
||||
groups: dict[str, list[tuple[str, dict]]] = {}
|
||||
for key, comp in to_resolve.items():
|
||||
registry = _classify_registry(comp)
|
||||
groups.setdefault(registry, []).append((key, comp))
|
||||
|
||||
print("\n Breakdown by registry:")
|
||||
for registry in sorted(groups):
|
||||
count = len(groups[registry])
|
||||
marker = "(local)" if registry in {"known", "golang", "deb"} else "(API)"
|
||||
print(f" {registry:<12} {count:>5} {marker}")
|
||||
|
||||
# Resolve -- local lookups first (instant), then API calls concurrently
|
||||
resolved: dict[str, str] = {}
|
||||
failed: list[str] = []
|
||||
t0 = time.monotonic()
|
||||
|
||||
local_registries = {"known", "golang", "deb", "other"}
|
||||
api_registries = {"crates.io", "npm", "pypi"}
|
||||
|
||||
# Phase 1: local (no network)
|
||||
local_items = [
|
||||
(key, comp) for reg in local_registries for key, comp in groups.get(reg, [])
|
||||
]
|
||||
for key, comp in local_items:
|
||||
lic = resolve_component(comp)
|
||||
if lic:
|
||||
resolved[key] = lic
|
||||
else:
|
||||
failed.append(key)
|
||||
|
||||
if local_items:
|
||||
print(
|
||||
f"\n Local lookups: {len(resolved)} resolved, "
|
||||
f"{len(failed)} unresolved ({time.monotonic() - t0:.1f}s)"
|
||||
)
|
||||
|
||||
# Phase 2: API calls (concurrent)
|
||||
api_items = [
|
||||
(key, comp) for reg in api_registries for key, comp in groups.get(reg, [])
|
||||
]
|
||||
|
||||
if api_items:
|
||||
api_total = len(api_items)
|
||||
api_resolved = 0
|
||||
api_failed = 0
|
||||
print(
|
||||
f"\n Resolving {api_total} packages via registry APIs "
|
||||
f"({_MAX_WORKERS} workers)..."
|
||||
)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=_MAX_WORKERS) as pool:
|
||||
futures = {
|
||||
pool.submit(_resolve_one, key, comp): key for key, comp in api_items
|
||||
}
|
||||
for done_count, future in enumerate(as_completed(futures), 1):
|
||||
key, lic = future.result()
|
||||
if lic:
|
||||
resolved[key] = lic
|
||||
api_resolved += 1
|
||||
else:
|
||||
failed.append(key)
|
||||
api_failed += 1
|
||||
|
||||
if done_count % 50 == 0 or done_count == api_total:
|
||||
elapsed = time.monotonic() - t0
|
||||
sys.stdout.write(
|
||||
f"\r [{done_count}/{api_total}] "
|
||||
f"resolved={api_resolved} failed={api_failed} "
|
||||
f"({elapsed:.1f}s)"
|
||||
)
|
||||
sys.stdout.flush()
|
||||
|
||||
print() # newline after progress
|
||||
|
||||
elapsed = time.monotonic() - t0
|
||||
print(
|
||||
f"\n Done: {len(resolved)}/{total} resolved, "
|
||||
f"{len(failed)} unresolved ({elapsed:.1f}s)"
|
||||
)
|
||||
|
||||
# Apply to all files
|
||||
total_patched = 0
|
||||
for f in files:
|
||||
with f.open() as fh:
|
||||
sbom = json.load(fh)
|
||||
|
||||
patched = 0
|
||||
for comp in sbom.get("components", []):
|
||||
if needs_fix(comp):
|
||||
key = f"{comp.get('name', '')}@{comp.get('version', '')}"
|
||||
if key in resolved:
|
||||
set_license(comp, resolved[key])
|
||||
patched += 1
|
||||
|
||||
with f.open("w") as fh:
|
||||
json.dump(sbom, fh, indent=2)
|
||||
fh.write("\n")
|
||||
|
||||
total_patched += patched
|
||||
print(f" {f.name}: patched {patched} components")
|
||||
|
||||
print(f"\n Total patches applied: {total_patched}")
|
||||
|
||||
if failed:
|
||||
print(f"\n --- Unresolved ({len(failed)}) ---")
|
||||
for key in sorted(failed):
|
||||
comp = to_resolve[key]
|
||||
print(f" {key} purl={comp.get('purl', '(none)')}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
"""Convert CycloneDX SBOM JSON files to CSV.
|
||||
|
||||
Usage:
|
||||
python sbom_to_csv.py # convert all *.cdx.json in deploy/sbom/output/
|
||||
python sbom_to_csv.py file1.json ... # convert specific files
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def extract_licenses(component: dict) -> str:
|
||||
"""Pull license IDs/names from CycloneDX license entries."""
|
||||
licenses = component.get("licenses", [])
|
||||
ids = []
|
||||
for entry in licenses:
|
||||
lic = entry.get("license", {})
|
||||
ids.append(lic.get("id") or lic.get("name", ""))
|
||||
return " | ".join(filter(None, ids))
|
||||
|
||||
|
||||
def sbom_to_csv(json_path: Path) -> Path:
|
||||
"""Read a CycloneDX JSON SBOM and write a CSV beside it."""
|
||||
with json_path.open() as f:
|
||||
sbom = json.load(f)
|
||||
|
||||
csv_path = json_path.with_suffix(".csv")
|
||||
components = sbom.get("components", [])
|
||||
|
||||
with csv_path.open("w", newline="") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow(["name", "version", "type", "purl", "licenses", "bom-ref"])
|
||||
|
||||
for comp in components:
|
||||
writer.writerow(
|
||||
[
|
||||
comp.get("name", ""),
|
||||
comp.get("version", ""),
|
||||
comp.get("type", ""),
|
||||
comp.get("purl", ""),
|
||||
extract_licenses(comp),
|
||||
comp.get("bom-ref", ""),
|
||||
]
|
||||
)
|
||||
|
||||
return csv_path
|
||||
|
||||
|
||||
def _find_sbom_files() -> list[Path]:
|
||||
"""Find SBOM JSON files in the default output directory."""
|
||||
repo_root = Path(__file__).resolve().parent.parent.parent
|
||||
output_dir = repo_root / "deploy" / "sbom" / "output"
|
||||
return sorted(output_dir.glob("*.cdx.json"))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
files = [Path(p) for p in sys.argv[1:]] if len(sys.argv) > 1 else _find_sbom_files()
|
||||
|
||||
if not files:
|
||||
print("No SBOM JSON files found.")
|
||||
print("Run 'mise run sbom:generate' first, or pass file paths as arguments.")
|
||||
sys.exit(1)
|
||||
|
||||
for path in files:
|
||||
csv_path = sbom_to_csv(path)
|
||||
with csv_path.open() as count_fh:
|
||||
components_count = sum(1 for _ in count_fh) - 1 # minus header
|
||||
print(f"{path.name} -> {csv_path.name} ({components_count} components)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -19,6 +19,8 @@ uv = "0.10.2"
|
||||
protoc = "29.6"
|
||||
helm = "4.1.1"
|
||||
"ubi:mozilla/sccache" = { version = "0.14.0", matching = "sccache-v" }
|
||||
"ubi:anchore/syft" = { version = "1.42.2", matching = "syft_" }
|
||||
"ubi:EmbarkStudios/cargo-about" = "0.8.4"
|
||||
|
||||
[env]
|
||||
_.path = ["{{config_root}}/scripts/bin"]
|
||||
@@ -42,7 +44,7 @@ DOCKER_BUILDKIT = "1"
|
||||
|
||||
[vars]
|
||||
# Python paths to include in formatting/linting
|
||||
python_paths = "python/ tasks/scripts/*.py"
|
||||
python_paths = "python/ tasks/scripts/*.py deploy/sbom/*.py"
|
||||
|
||||
[task_config]
|
||||
includes = ["tasks/*.toml"]
|
||||
|
||||
@@ -3,13 +3,16 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
"""Generate THIRD-PARTY-NOTICES from Rust and Python dependency metadata.
|
||||
"""Generate THIRD-PARTY-NOTICES with full license texts.
|
||||
|
||||
Uses cargo-about (Rust) and pip-licenses (Python) to collect third-party
|
||||
dependency licenses and produce a single attribution file at the repo root.
|
||||
|
||||
Usage:
|
||||
python scripts/generate_third_party_notices.py
|
||||
uv run python scripts/generate_third_party_notices.py
|
||||
mise run notices
|
||||
|
||||
Writes THIRD-PARTY-NOTICES to the repo root. Requires `cargo` on PATH
|
||||
for Rust deps. Python deps are read from pyproject.toml.
|
||||
Requires cargo-about (installed via mise) and pip-licenses (fetched via uv).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -17,8 +20,34 @@ from __future__ import annotations
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
# Our own workspace crates and packages — excluded from notices.
|
||||
WORKSPACE_CRATES = frozenset(
|
||||
{
|
||||
"navigator-bootstrap",
|
||||
"navigator-cli",
|
||||
"navigator-core",
|
||||
"navigator-policy",
|
||||
"navigator-providers",
|
||||
"navigator-router",
|
||||
"navigator-sandbox",
|
||||
"navigator-server",
|
||||
"navigator-tui",
|
||||
"openshell-e2e",
|
||||
}
|
||||
)
|
||||
|
||||
OWN_PYTHON_PACKAGES = frozenset(
|
||||
{
|
||||
"openshell",
|
||||
}
|
||||
)
|
||||
|
||||
SEPARATOR = "=" * 80
|
||||
THIN_SEP = "-" * 80
|
||||
|
||||
|
||||
def find_repo_root() -> Path:
|
||||
"""Walk up from CWD to find the directory containing .git."""
|
||||
@@ -30,193 +59,216 @@ def find_repo_root() -> Path:
|
||||
return Path.cwd()
|
||||
|
||||
|
||||
# Workspace member crate names to exclude (these are ours, not third-party).
|
||||
WORKSPACE_PREFIXES = ("navigator-",)
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rust dependencies via cargo-about
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def get_rust_deps_from_cargo_metadata() -> list[dict[str, str]] | None:
|
||||
"""Try to extract third-party Rust deps via `cargo metadata`."""
|
||||
def get_rust_notices() -> list[dict]:
|
||||
"""Run cargo-about and return structured license groups.
|
||||
|
||||
Each entry: {id, crates: [{name, version, repository, description}], text}
|
||||
"""
|
||||
print(" Running cargo-about generate --format json ...")
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["cargo", "metadata", "--format-version", "1"],
|
||||
["cargo-about", "generate", "--format", "json"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
except (FileNotFoundError, subprocess.CalledProcessError):
|
||||
return None
|
||||
|
||||
meta = json.loads(result.stdout)
|
||||
workspace_members = set(meta.get("workspace_members", []))
|
||||
|
||||
deps = []
|
||||
for pkg in meta["packages"]:
|
||||
if pkg["id"] in workspace_members:
|
||||
continue
|
||||
if any(pkg["name"].startswith(p) for p in WORKSPACE_PREFIXES):
|
||||
continue
|
||||
deps.append({
|
||||
"name": pkg["name"],
|
||||
"version": pkg["version"],
|
||||
"license": pkg.get("license") or "Unknown",
|
||||
"repository": pkg.get("repository") or "",
|
||||
})
|
||||
|
||||
return sorted(deps, key=lambda d: d["name"].lower())
|
||||
|
||||
|
||||
def get_rust_deps_from_lockfile(root: Path) -> list[dict[str, str]]:
|
||||
"""Fallback: parse Cargo.lock for name+version (no license info)."""
|
||||
lockfile = root / "Cargo.lock"
|
||||
if not lockfile.exists():
|
||||
except FileNotFoundError:
|
||||
print(
|
||||
" WARNING: cargo-about not found, skipping Rust notices", file=sys.stderr
|
||||
)
|
||||
return []
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f" WARNING: cargo-about failed: {e.stderr[:200]}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
deps = []
|
||||
content = lockfile.read_text()
|
||||
name = None
|
||||
version = None
|
||||
for line in content.splitlines():
|
||||
if line.startswith("name = "):
|
||||
name = line.split('"')[1]
|
||||
elif line.startswith("version = ") and '"' in line:
|
||||
version = line.split('"')[1]
|
||||
elif line == "[[package]]" or line == "":
|
||||
if name and version:
|
||||
if not any(name.startswith(p) for p in WORKSPACE_PREFIXES):
|
||||
deps.append({
|
||||
"name": name,
|
||||
"version": version,
|
||||
"license": "See crates.io",
|
||||
"repository": f"https://crates.io/crates/{name}",
|
||||
})
|
||||
name = None
|
||||
version = None
|
||||
data = json.loads(result.stdout)
|
||||
groups: list[dict] = []
|
||||
|
||||
# Catch the last entry.
|
||||
if name and version:
|
||||
if not any(name.startswith(p) for p in WORKSPACE_PREFIXES):
|
||||
deps.append({
|
||||
"name": name,
|
||||
"version": version,
|
||||
"license": "See crates.io",
|
||||
"repository": f"https://crates.io/crates/{name}",
|
||||
})
|
||||
for lic in data.get("licenses", []):
|
||||
crates = []
|
||||
for entry in lic.get("used_by", []):
|
||||
crate = entry.get("crate", {})
|
||||
name = crate.get("name", "")
|
||||
if name in WORKSPACE_CRATES:
|
||||
continue
|
||||
crates.append(
|
||||
{
|
||||
"name": name,
|
||||
"version": crate.get("version", ""),
|
||||
"repository": crate.get("repository", ""),
|
||||
"description": crate.get("description", ""),
|
||||
}
|
||||
)
|
||||
|
||||
return sorted(deps, key=lambda d: d["name"].lower())
|
||||
if not crates:
|
||||
continue
|
||||
|
||||
groups.append(
|
||||
{
|
||||
"id": lic.get("id", "Unknown"),
|
||||
"crates": sorted(crates, key=lambda c: c["name"].lower()),
|
||||
"text": (lic.get("text") or "").rstrip(),
|
||||
}
|
||||
)
|
||||
|
||||
return groups
|
||||
|
||||
|
||||
def get_rust_deps(root: Path) -> list[dict[str, str]]:
|
||||
"""Extract third-party Rust dependencies.
|
||||
# ---------------------------------------------------------------------------
|
||||
# Python dependencies via pip-licenses
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Prefers `cargo metadata` for full license info. Falls back to parsing
|
||||
Cargo.lock when cargo is not available.
|
||||
|
||||
def get_python_notices() -> list[dict]:
|
||||
"""Run pip-licenses and return structured package notices.
|
||||
|
||||
Each entry: {name, version, license_id, text}
|
||||
"""
|
||||
deps = get_rust_deps_from_cargo_metadata()
|
||||
if deps is not None:
|
||||
return deps
|
||||
|
||||
print(" cargo not found, falling back to Cargo.lock parsing", file=sys.stderr)
|
||||
return get_rust_deps_from_lockfile(root)
|
||||
|
||||
|
||||
def get_python_deps(root: Path) -> list[dict[str, str]]:
|
||||
"""Extract Python dependencies from pyproject.toml [project.dependencies]."""
|
||||
pyproject = root / "pyproject.toml"
|
||||
if not pyproject.exists():
|
||||
print(" Running pip-licenses ...")
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[
|
||||
"uv",
|
||||
"run",
|
||||
"--with",
|
||||
"pip-licenses",
|
||||
"pip-licenses",
|
||||
"--format=json",
|
||||
"--with-license-file",
|
||||
"--no-license-path",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
except FileNotFoundError:
|
||||
print(" WARNING: uv not found, skipping Python notices", file=sys.stderr)
|
||||
return []
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f" WARNING: pip-licenses failed: {e.stderr[:200]}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
# Simple parser: read the dependencies list without a TOML library.
|
||||
content = pyproject.read_text()
|
||||
deps = []
|
||||
in_deps = False
|
||||
for line in content.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("dependencies = ["):
|
||||
in_deps = True
|
||||
packages: list[dict] = []
|
||||
for pkg in json.loads(result.stdout):
|
||||
name = pkg.get("Name", "")
|
||||
if name.lower() in OWN_PYTHON_PACKAGES:
|
||||
continue
|
||||
# Skip pip/setuptools/wheel (installer tools, not shipped deps)
|
||||
if name.lower() in {"pip", "wheel"}:
|
||||
continue
|
||||
if in_deps:
|
||||
if stripped == "]":
|
||||
break
|
||||
# Parse "package>=version" style.
|
||||
dep = stripped.strip('",').strip()
|
||||
if dep:
|
||||
# Split on first version specifier.
|
||||
for sep in (">=", "==", "~=", "!=", "<", ">"):
|
||||
if sep in dep:
|
||||
name = dep[:dep.index(sep)].strip()
|
||||
deps.append({
|
||||
"name": name,
|
||||
"version": dep[dep.index(sep):].strip(),
|
||||
"license": "See PyPI",
|
||||
"repository": f"https://pypi.org/project/{name}/",
|
||||
})
|
||||
break
|
||||
else:
|
||||
deps.append({
|
||||
"name": dep,
|
||||
"version": "",
|
||||
"license": "See PyPI",
|
||||
"repository": f"https://pypi.org/project/{dep}/",
|
||||
})
|
||||
|
||||
return sorted(deps, key=lambda d: d["name"].lower())
|
||||
packages.append(
|
||||
{
|
||||
"name": name,
|
||||
"version": pkg.get("Version", ""),
|
||||
"license_id": pkg.get("License", "Unknown"),
|
||||
"text": (pkg.get("LicenseText") or "").rstrip(),
|
||||
}
|
||||
)
|
||||
|
||||
return sorted(packages, key=lambda p: p["name"].lower())
|
||||
|
||||
|
||||
def format_notices(rust_deps: list[dict], python_deps: list[dict]) -> str:
|
||||
"""Format the THIRD-PARTY-NOTICES file content."""
|
||||
lines = [
|
||||
"THIRD-PARTY SOFTWARE NOTICES",
|
||||
# ---------------------------------------------------------------------------
|
||||
# Output formatting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def format_notices(
|
||||
rust_groups: list[dict],
|
||||
python_packages: list[dict],
|
||||
) -> str:
|
||||
"""Format the complete THIRD-PARTY-NOTICES file."""
|
||||
lines: list[str] = [
|
||||
"THIRD-PARTY SOFTWARE NOTICES AND INFORMATION",
|
||||
"",
|
||||
"This file lists the third-party software packages used by OpenShell,",
|
||||
"along with their respective licenses.",
|
||||
"This product includes third-party software components. The following",
|
||||
"notices and licenses are provided in compliance with the terms of the",
|
||||
"respective licenses.",
|
||||
"",
|
||||
"To regenerate: uv run python scripts/generate_third_party_notices.py",
|
||||
"To regenerate: mise run notices",
|
||||
"",
|
||||
]
|
||||
|
||||
if rust_deps:
|
||||
lines.append("=" * 80)
|
||||
lines.append("Rust Dependencies")
|
||||
lines.append("=" * 80)
|
||||
# --- Rust section ---
|
||||
if rust_groups:
|
||||
rust_crate_count = sum(len(g["crates"]) for g in rust_groups)
|
||||
lines.append(SEPARATOR)
|
||||
lines.append(f"Rust Dependencies ({rust_crate_count} packages)")
|
||||
lines.append(SEPARATOR)
|
||||
lines.append("")
|
||||
for dep in rust_deps:
|
||||
lines.append(f"Package: {dep['name']} {dep['version']}")
|
||||
lines.append(f"License: {dep['license']}")
|
||||
if dep["repository"]:
|
||||
lines.append(f"Repository: {dep['repository']}")
|
||||
|
||||
for group in rust_groups:
|
||||
lines.append(SEPARATOR)
|
||||
lines.append(f"License: {group['id']}")
|
||||
lines.append(THIN_SEP)
|
||||
lines.append("")
|
||||
lines.append("Used by:")
|
||||
for crate in group["crates"]:
|
||||
repo = f" ({crate['repository']})" if crate["repository"] else ""
|
||||
lines.append(f" - {crate['name']} {crate['version']}{repo}")
|
||||
lines.append("")
|
||||
|
||||
if python_deps:
|
||||
lines.append("=" * 80)
|
||||
lines.append("Python Dependencies")
|
||||
lines.append("=" * 80)
|
||||
if group["text"]:
|
||||
lines.append(group["text"])
|
||||
lines.append("")
|
||||
|
||||
# --- Python section ---
|
||||
if python_packages:
|
||||
lines.append(SEPARATOR)
|
||||
lines.append(f"Python Dependencies ({len(python_packages)} packages)")
|
||||
lines.append(SEPARATOR)
|
||||
lines.append("")
|
||||
for dep in python_deps:
|
||||
version_str = f" {dep['version']}" if dep["version"] else ""
|
||||
lines.append(f"Package: {dep['name']}{version_str}")
|
||||
lines.append(f"License: {dep['license']}")
|
||||
if dep["repository"]:
|
||||
lines.append(f"Repository: {dep['repository']}")
|
||||
|
||||
for pkg in python_packages:
|
||||
lines.append(SEPARATOR)
|
||||
lines.append(f"{pkg['name']} {pkg['version']}")
|
||||
lines.append(f"License: {pkg['license_id']}")
|
||||
lines.append(THIN_SEP)
|
||||
lines.append("")
|
||||
if pkg["text"]:
|
||||
lines.append(pkg["text"])
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main() -> int:
|
||||
root = find_repo_root()
|
||||
t0 = time.monotonic()
|
||||
|
||||
print("Generating third-party notices...")
|
||||
print()
|
||||
|
||||
print("Collecting Rust dependencies...")
|
||||
rust_deps = get_rust_deps(root)
|
||||
print(f" Found {len(rust_deps)} Rust dependencies")
|
||||
rust_groups = get_rust_notices()
|
||||
rust_count = sum(len(g["crates"]) for g in rust_groups)
|
||||
print(f" {rust_count} Rust packages across {len(rust_groups)} license groups")
|
||||
print()
|
||||
|
||||
print("Collecting Python dependencies...")
|
||||
python_deps = get_python_deps(root)
|
||||
print(f" Found {len(python_deps)} Python dependencies")
|
||||
python_packages = get_python_notices()
|
||||
print(f" {len(python_packages)} Python packages")
|
||||
print()
|
||||
|
||||
notices = format_notices(rust_deps, python_deps)
|
||||
notices = format_notices(rust_groups, python_packages)
|
||||
output = root / "THIRD-PARTY-NOTICES"
|
||||
output.write_text(notices)
|
||||
print(f"Wrote {output}")
|
||||
|
||||
elapsed = time.monotonic() - t0
|
||||
line_count = notices.count("\n") + 1
|
||||
print(f"Wrote {output.name} ({line_count} lines, {len(notices)} bytes)")
|
||||
print(f"Done in {elapsed:.1f}s")
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# Third-party license notice generation tasks
|
||||
#
|
||||
# Usage:
|
||||
# mise run notices # regenerate THIRD-PARTY-NOTICES at repo root
|
||||
|
||||
[notices]
|
||||
description = "Regenerate THIRD-PARTY-NOTICES with full license texts"
|
||||
run = "uv run python scripts/generate_third_party_notices.py"
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# SBOM generation, license resolution, and CSV export tasks
|
||||
#
|
||||
# Usage:
|
||||
# mise run sbom # generate SBOMs, resolve licenses, export CSVs
|
||||
# mise run sbom:check # advisory license check (for CI)
|
||||
|
||||
[sbom]
|
||||
description = "Generate SBOMs, resolve licenses, and export CSVs to deploy/sbom/output/"
|
||||
depends = ["sbom:csv"]
|
||||
|
||||
["sbom:generate"]
|
||||
description = "Generate CycloneDX SBOMs with Syft"
|
||||
hide = true
|
||||
run = """
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
OUTPUT_DIR="deploy/sbom/output"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
VERSION=$(uv run python tasks/scripts/release.py get-version --cargo)
|
||||
|
||||
echo "Generating SBOM for workspace (version ${VERSION})..."
|
||||
|
||||
syft dir:. \
|
||||
--exclude './.github/**' \
|
||||
--exclude './.venv/**' \
|
||||
--exclude './.cache/**' \
|
||||
--output "cyclonedx-json=$OUTPUT_DIR/openshell-source-${VERSION}.cdx.json" \
|
||||
--source-name openshell \
|
||||
--source-version "$VERSION"
|
||||
|
||||
echo ""
|
||||
echo "SBOM written to $OUTPUT_DIR/"
|
||||
ls -la "$OUTPUT_DIR"/*.cdx.json
|
||||
"""
|
||||
|
||||
["sbom:resolve"]
|
||||
description = "Resolve missing licenses in SBOM JSON files via public registries"
|
||||
depends = ["sbom:generate"]
|
||||
hide = true
|
||||
run = "uv run python deploy/sbom/resolve_licenses.py"
|
||||
|
||||
["sbom:csv"]
|
||||
description = "Convert SBOM JSON files to CSV"
|
||||
depends = ["sbom:resolve"]
|
||||
hide = true
|
||||
run = "uv run python deploy/sbom/sbom_to_csv.py"
|
||||
|
||||
["sbom:check"]
|
||||
description = "Check SBOMs for unresolved licenses (advisory, non-blocking)"
|
||||
hide = true
|
||||
run = """
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
OUTPUT_DIR="deploy/sbom/output"
|
||||
|
||||
if [ ! -d "$OUTPUT_DIR" ] || [ -z "$(ls -A "$OUTPUT_DIR"/*.cdx.json 2>/dev/null)" ]; then
|
||||
echo "No SBOM files found in $OUTPUT_DIR/. Run 'mise run sbom' first."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Checking for unresolved licenses..."
|
||||
UNRESOLVED=0
|
||||
for f in "$OUTPUT_DIR"/*.cdx.json; do
|
||||
COUNT=$(uv run python -c "
|
||||
import json, sys
|
||||
with open('$f') as fh:
|
||||
sbom = json.load(fh)
|
||||
missing = 0
|
||||
for c in sbom.get('components', []):
|
||||
lics = c.get('licenses', [])
|
||||
if not lics:
|
||||
missing += 1
|
||||
continue
|
||||
for e in lics:
|
||||
lid = e.get('license', {}).get('id', '')
|
||||
lname = e.get('license', {}).get('name', '')
|
||||
if lid.startswith('sha256:') or lname.startswith('sha256:'):
|
||||
missing += 1
|
||||
break
|
||||
print(missing)
|
||||
")
|
||||
if [ "$COUNT" -gt 0 ]; then
|
||||
echo " $(basename "$f"): $COUNT components with unresolved licenses"
|
||||
UNRESOLVED=$((UNRESOLVED + COUNT))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$UNRESOLVED" -gt 0 ]; then
|
||||
echo ""
|
||||
echo "WARNING: $UNRESOLVED total components with unresolved licenses."
|
||||
echo "This is advisory -- not blocking the build."
|
||||
else
|
||||
echo "All licenses resolved."
|
||||
fi
|
||||
"""
|
||||
Reference in New Issue
Block a user