mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-31 07:44:57 +08:00
feat(sbom): add SBOM generation, license resolution, and CSV export tooling
Add mise-integrated SBOM pipeline for container images using Syft. Includes license resolution via crates.io/npm/PyPI APIs and CycloneDX JSON to CSV conversion. Adds agent skill for on-demand SBOM operations. Closes #237
This commit is contained in:
@@ -0,0 +1,105 @@
|
||||
---
|
||||
name: sbom
|
||||
description: Generate and manage Software Bill of Materials (SBOMs) for the OpenShell project. Covers SBOM generation with Syft, license resolution via public registries, and CSV export for compliance review. Trigger keywords - SBOM, sbom, bill of materials, license audit, license resolution, generate sbom, sbom csv, dependency license, supply chain, license scan.
|
||||
---
|
||||
|
||||
# SBOM Generation and License Resolution
|
||||
|
||||
Generate CycloneDX SBOMs, resolve missing licenses, and export to CSV for compliance review.
|
||||
|
||||
## Overview
|
||||
|
||||
The OpenShell SBOM tooling produces CycloneDX JSON SBOMs using Syft, resolves missing or hash-based licenses by querying public registries (crates.io, npm, PyPI), and exports the results to CSV for stakeholder review.
|
||||
|
||||
SBOMs are **release artifacts only** -- they are generated on demand and not committed to the repository. Output lands in `deploy/sbom/output/` (gitignored).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `mise install` has been run (installs Syft and other tools)
|
||||
- The repository is checked out at the root
|
||||
|
||||
## Workflow 1: Full SBOM Generation (One Command)
|
||||
|
||||
```bash
|
||||
mise run sbom
|
||||
```
|
||||
|
||||
This single command chains three stages:
|
||||
|
||||
1. **Generate** (`sbom:generate`): Syft scans the workspace source tree and produces a CycloneDX JSON SBOM
|
||||
2. **Resolve** (`sbom:resolve`): Public registry APIs fill in missing or hash-based licenses in the JSON
|
||||
3. **CSV** (`sbom:csv`): JSON SBOMs are converted to CSV for review
|
||||
|
||||
Output directory: `deploy/sbom/output/`
|
||||
|
||||
After running, the user can find:
|
||||
- `deploy/sbom/output/*.cdx.json` -- full CycloneDX SBOMs
|
||||
- `deploy/sbom/output/*.csv` -- CSV exports ready for spreadsheet review
|
||||
|
||||
## Workflow 2: Individual Stages
|
||||
|
||||
Run stages independently when debugging or iterating:
|
||||
|
||||
```bash
|
||||
mise run sbom:generate # Generate JSON SBOMs only (requires Syft)
|
||||
mise run sbom:resolve # Resolve licenses in existing JSONs (queries APIs)
|
||||
mise run sbom:csv # Convert existing JSONs to CSV
|
||||
```
|
||||
|
||||
## Workflow 3: License Check (CI Advisory)
|
||||
|
||||
```bash
|
||||
mise run sbom:check
|
||||
```
|
||||
|
||||
Reports unresolved licenses without failing. Intended for PR CI as a non-blocking advisory check. Requires that SBOMs have already been generated (`mise run sbom:generate`).
|
||||
|
||||
## Workflow 4: Processing External SBOMs
|
||||
|
||||
The Python scripts accept explicit file paths, so they can process SBOMs from any source (e.g., NVIDIA nSpect pipeline output):
|
||||
|
||||
```bash
|
||||
uv run python deploy/sbom/resolve_licenses.py /path/to/external-sbom.json
|
||||
uv run python deploy/sbom/sbom_to_csv.py /path/to/external-sbom.json
|
||||
```
|
||||
|
||||
## License Resolution Details
|
||||
|
||||
The resolver queries these public registries:
|
||||
|
||||
| Registry | Package URL prefix | Method |
|
||||
|----------|-------------------|--------|
|
||||
| crates.io | `pkg:cargo/*` | REST API |
|
||||
| npm | `pkg:npm/*` | Registry API |
|
||||
| PyPI | `pkg:pypi/*` | JSON API |
|
||||
| Go modules | `pkg:golang/*` | Known license map (no API) |
|
||||
| Debian/Ubuntu | `pkg:deb/*` | Known license map |
|
||||
|
||||
Components from private registries (e.g., `@openclaw/*` npm packages) are not resolved and will appear in the "unresolved" report.
|
||||
|
||||
## Output Files
|
||||
|
||||
| Pattern | Description |
|
||||
|---------|-------------|
|
||||
| `deploy/sbom/output/openshell-source-{version}.cdx.json` | CycloneDX JSON SBOM |
|
||||
| `deploy/sbom/output/openshell-source-{version}.csv` | CSV export (name, version, type, purl, licenses, bom-ref) |
|
||||
|
||||
## Key Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `deploy/sbom/resolve_licenses.py` | License resolution script |
|
||||
| `deploy/sbom/sbom_to_csv.py` | JSON-to-CSV converter |
|
||||
| `tasks/sbom.toml` | Mise task definitions |
|
||||
| `mise.toml` | Syft tool definition (under `[tools]`) |
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Task | Command |
|
||||
|------|---------|
|
||||
| Full pipeline | `mise run sbom` |
|
||||
| Generate only | `mise run sbom:generate` |
|
||||
| Resolve licenses | `mise run sbom:resolve` |
|
||||
| Export CSV | `mise run sbom:csv` |
|
||||
| CI license check | `mise run sbom:check` |
|
||||
| Process external SBOM | `uv run python deploy/sbom/resolve_licenses.py <file>` |
|
||||
@@ -184,6 +184,9 @@ _build/
|
||||
# Docker build artifacts (image tarballs, packaged helm charts)
|
||||
deploy/docker/.build/
|
||||
|
||||
# SBOM generated output (JSON, CSV) — release artifacts, not committed
|
||||
deploy/sbom/output/
|
||||
|
||||
# Local mise settings
|
||||
mise.local.toml
|
||||
|
||||
|
||||
@@ -0,0 +1,433 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
"""Resolve missing and hash-based licenses in CycloneDX SBOM JSON files.
|
||||
|
||||
Queries public registries:
|
||||
- crates.io (pkg:cargo/*)
|
||||
- npm (pkg:npm/*)
|
||||
- PyPI (pkg:pypi/*)
|
||||
- Known maps (pkg:golang/*, pkg:deb/*, operating-system, application)
|
||||
|
||||
Updates the JSON files in-place, then reports what was resolved.
|
||||
|
||||
Usage:
|
||||
python resolve_licenses.py # resolve all *.cdx.json in deploy/sbom/output/
|
||||
python resolve_licenses.py file1.json ... # resolve specific files
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Known licenses for packages that registries won't resolve
|
||||
# ---------------------------------------------------------------------------
|
||||
KNOWN_LICENSES: dict[str, str] = {
|
||||
# libxcrypt -- all variants
|
||||
"libcrypt-dev": "LGPL-2.1-or-later",
|
||||
"libcrypt1": "LGPL-2.1-or-later",
|
||||
# python3-defaults -- all variants
|
||||
"python3": "PSF-2.0",
|
||||
"python3-minimal": "PSF-2.0",
|
||||
"libpython3-stdlib": "PSF-2.0",
|
||||
"python3-venv": "PSF-2.0",
|
||||
# Go standard library / well-known
|
||||
"golang.org/x/crypto": "BSD-3-Clause",
|
||||
"golang.org/x/mod": "BSD-3-Clause",
|
||||
"golang.org/x/net": "BSD-3-Clause",
|
||||
"golang.org/x/sync": "BSD-3-Clause",
|
||||
"golang.org/x/sys": "BSD-3-Clause",
|
||||
"golang.org/x/term": "BSD-3-Clause",
|
||||
"golang.org/x/text": "BSD-3-Clause",
|
||||
"google.golang.org/grpc": "Apache-2.0",
|
||||
"google.golang.org/protobuf": "BSD-3-Clause",
|
||||
"google.golang.org/genproto/googleapis/api": "Apache-2.0",
|
||||
"google.golang.org/genproto/googleapis/rpc": "Apache-2.0",
|
||||
"gopkg.in/yaml.v3": "MIT",
|
||||
"go.yaml.in/yaml/v3": "MIT",
|
||||
"go.opentelemetry.io/otel": "Apache-2.0",
|
||||
"go.opentelemetry.io/otel/metric": "Apache-2.0",
|
||||
"go.opentelemetry.io/otel/trace": "Apache-2.0",
|
||||
"go.opentelemetry.io/auto/sdk": "Apache-2.0",
|
||||
"go.mongodb.org/mongo-driver": "Apache-2.0",
|
||||
# Debian / Ubuntu system packages
|
||||
"debian": "GPL-2.0-only",
|
||||
"ubuntu": "GPL-2.0-only",
|
||||
# Application entries without purl
|
||||
"Simple Launcher": "Proprietary",
|
||||
"python": "PSF-2.0",
|
||||
}
|
||||
|
||||
# Well-known Go module licenses (GitHub-based)
|
||||
GO_KNOWN: dict[str, str] = {
|
||||
"github.com/AlecAivazis/survey": "MIT",
|
||||
"github.com/MakeNowJust/heredoc": "MIT",
|
||||
"github.com/Masterminds/goutils": "Apache-2.0",
|
||||
"github.com/Masterminds/semver": "MIT",
|
||||
"github.com/Masterminds/sprig": "MIT",
|
||||
"github.com/alecthomas/chroma": "MIT",
|
||||
"github.com/asaskevich/govalidator": "MIT",
|
||||
"github.com/atotto/clipboard": "BSD-3-Clause",
|
||||
"github.com/aymanbagabas/go-osc52": "MIT",
|
||||
"github.com/aymerick/douceur": "MIT",
|
||||
"github.com/blang/semver": "MIT",
|
||||
"github.com/briandowns/spinner": "Apache-2.0",
|
||||
"github.com/catppuccin/go": "MIT",
|
||||
"github.com/cenkalti/backoff": "MIT",
|
||||
"github.com/charmbracelet/bubbles": "MIT",
|
||||
"github.com/charmbracelet/bubbletea": "MIT",
|
||||
"github.com/charmbracelet/colorprofile": "MIT",
|
||||
"github.com/charmbracelet/glamour": "MIT",
|
||||
"github.com/charmbracelet/huh": "MIT",
|
||||
"github.com/charmbracelet/lipgloss": "MIT",
|
||||
"github.com/charmbracelet/x": "MIT",
|
||||
"github.com/cli/browser": "BSD-2-Clause",
|
||||
"github.com/cli/cli": "MIT",
|
||||
"github.com/cli/go-gh": "MIT",
|
||||
"github.com/cli/oauth": "MIT",
|
||||
"github.com/cli/safeexec": "BSD-2-Clause",
|
||||
"github.com/cli/shurcooL-graphql": "MIT",
|
||||
"github.com/containerd/stargz-snapshotter": "Apache-2.0",
|
||||
"github.com/cyberphone/json-canonicalization": "Apache-2.0",
|
||||
"github.com/davecgh/go-spew": "ISC",
|
||||
"github.com/digitorus/pkcs7": "MIT",
|
||||
"github.com/digitorus/timestamp": "MIT",
|
||||
"github.com/distribution/reference": "Apache-2.0",
|
||||
"github.com/dlclark/regexp2": "MIT",
|
||||
"github.com/docker/cli": "Apache-2.0",
|
||||
"github.com/docker/distribution": "Apache-2.0",
|
||||
"github.com/docker/docker-credential-helpers": "MIT",
|
||||
"github.com/dustin/go-humanize": "MIT",
|
||||
"github.com/fatih/color": "MIT",
|
||||
"github.com/gabriel-vasile/mimetype": "MIT",
|
||||
"github.com/gdamore/encoding": "Apache-2.0",
|
||||
"github.com/gdamore/tcell": "Apache-2.0",
|
||||
"github.com/go-logr/logr": "Apache-2.0",
|
||||
"github.com/go-logr/stdr": "Apache-2.0",
|
||||
"github.com/go-openapi/analysis": "Apache-2.0",
|
||||
"github.com/go-openapi/errors": "Apache-2.0",
|
||||
"github.com/go-openapi/jsonpointer": "Apache-2.0",
|
||||
"github.com/go-openapi/jsonreference": "Apache-2.0",
|
||||
"github.com/go-openapi/loads": "Apache-2.0",
|
||||
"github.com/go-openapi/runtime": "Apache-2.0",
|
||||
"github.com/go-openapi/spec": "Apache-2.0",
|
||||
"github.com/go-openapi/strfmt": "Apache-2.0",
|
||||
"github.com/go-openapi/swag": "Apache-2.0",
|
||||
"github.com/go-openapi/validate": "Apache-2.0",
|
||||
"github.com/go-viper/mapstructure": "MIT",
|
||||
"github.com/godbus/dbus": "BSD-2-Clause",
|
||||
"github.com/golang/snappy": "BSD-3-Clause",
|
||||
"github.com/google/certificate-transparency-go": "Apache-2.0",
|
||||
"github.com/google/go-containerregistry": "Apache-2.0",
|
||||
"github.com/google/shlex": "Apache-2.0",
|
||||
"github.com/google/uuid": "BSD-3-Clause",
|
||||
"github.com/gorilla/css": "BSD-3-Clause",
|
||||
"github.com/gorilla/websocket": "BSD-2-Clause",
|
||||
"github.com/grpc-ecosystem/grpc-gateway": "BSD-3-Clause",
|
||||
"github.com/hashicorp/go-version": "MPL-2.0",
|
||||
"github.com/henvic/httpretty": "MIT",
|
||||
"github.com/huandu/xstrings": "MIT",
|
||||
"github.com/in-toto/attestation": "Apache-2.0",
|
||||
"github.com/in-toto/in-toto-golang": "Apache-2.0",
|
||||
"github.com/itchyny/gojq": "MIT",
|
||||
"github.com/itchyny/timefmt-go": "MIT",
|
||||
"github.com/joho/godotenv": "MIT",
|
||||
"github.com/kballard/go-shellquote": "MIT",
|
||||
"github.com/klauspost/compress": "Apache-2.0",
|
||||
"github.com/lucasb-eyer/go-colorful": "MIT",
|
||||
"github.com/mattn/go-colorable": "MIT",
|
||||
"github.com/mattn/go-isatty": "MIT",
|
||||
"github.com/mattn/go-runewidth": "MIT",
|
||||
"github.com/mgutz/ansi": "MIT",
|
||||
"github.com/microcosm-cc/bluemonday": "BSD-3-Clause",
|
||||
"github.com/microsoft/dev-tunnels": "MIT",
|
||||
"github.com/mitchellh/copystructure": "MIT",
|
||||
"github.com/mitchellh/go-homedir": "MIT",
|
||||
"github.com/mitchellh/hashstructure": "MIT",
|
||||
"github.com/mitchellh/reflectwalk": "MIT",
|
||||
"github.com/muesli/ansi": "MIT",
|
||||
"github.com/muesli/cancelreader": "MIT",
|
||||
"github.com/muesli/reflow": "MIT",
|
||||
"github.com/muesli/termenv": "MIT",
|
||||
"github.com/muhammadmuzzammil1998/jsonc": "MIT",
|
||||
"github.com/oklog/ulid": "Apache-2.0",
|
||||
"github.com/opencontainers/go-digest": "Apache-2.0",
|
||||
"github.com/opencontainers/image-spec": "Apache-2.0",
|
||||
"github.com/opentracing/opentracing-go": "Apache-2.0",
|
||||
"github.com/pkg/errors": "BSD-2-Clause",
|
||||
"github.com/pmezard/go-difflib": "BSD-3-Clause",
|
||||
"github.com/rivo/tview": "MIT",
|
||||
"github.com/rivo/uniseg": "MIT",
|
||||
"github.com/rodaine/table": "MIT",
|
||||
"github.com/secure-systems-lab/go-securesystemslib": "MIT",
|
||||
"github.com/shibumi/go-pathspec": "Apache-2.0",
|
||||
"github.com/shopspring/decimal": "MIT",
|
||||
"github.com/shurcooL/githubv4": "MIT",
|
||||
"github.com/shurcooL/graphql": "MIT",
|
||||
"github.com/sigstore/protobuf-specs": "Apache-2.0",
|
||||
"github.com/sigstore/rekor-tiles": "Apache-2.0",
|
||||
"github.com/sigstore/rekor": "Apache-2.0",
|
||||
"github.com/sigstore/sigstore-go": "Apache-2.0",
|
||||
"github.com/sigstore/sigstore": "Apache-2.0",
|
||||
"github.com/sigstore/timestamp-authority": "Apache-2.0",
|
||||
"github.com/sirupsen/logrus": "MIT",
|
||||
"github.com/spf13/cast": "MIT",
|
||||
"github.com/spf13/cobra": "Apache-2.0",
|
||||
"github.com/spf13/pflag": "BSD-3-Clause",
|
||||
"github.com/stretchr/objx": "MIT",
|
||||
"github.com/stretchr/testify": "MIT",
|
||||
"github.com/theupdateframework/go-tuf": "MIT",
|
||||
"github.com/thlib/go-timezone-local": "Unlicense",
|
||||
"github.com/transparency-dev/formats": "Apache-2.0",
|
||||
"github.com/transparency-dev/merkle": "Apache-2.0",
|
||||
"github.com/vbatts/tar-split": "BSD-3-Clause",
|
||||
"github.com/vmihailenco/msgpack": "BSD-2-Clause",
|
||||
"github.com/vmihailenco/tagparser": "BSD-2-Clause",
|
||||
"github.com/xo/terminfo": "MIT",
|
||||
"github.com/yuin/goldmark-emoji": "MIT",
|
||||
"github.com/yuin/goldmark": "MIT",
|
||||
"github.com/zalando/go-keyring": "MIT",
|
||||
"dario.cat/mergo": "BSD-3-Clause",
|
||||
}
|
||||
|
||||
# Rate-limit helpers
|
||||
_last_request: dict[str, float] = {}
|
||||
|
||||
|
||||
def _rate_limit(domain: str, interval: float = 0.15) -> None:
|
||||
now = time.time()
|
||||
last = _last_request.get(domain, 0)
|
||||
wait = interval - (now - last)
|
||||
if wait > 0:
|
||||
time.sleep(wait)
|
||||
_last_request[domain] = time.time()
|
||||
|
||||
|
||||
def _get_json(url: str, domain: str) -> dict | None:
|
||||
_rate_limit(domain)
|
||||
req = urllib.request.Request(
|
||||
url, headers={"User-Agent": "sbom-license-resolver/1.0"}
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=10) as resp:
|
||||
return json.loads(resp.read())
|
||||
except (urllib.error.HTTPError, urllib.error.URLError, TimeoutError):
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Registry lookups
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def lookup_cargo(name: str, version: str) -> str | None:
|
||||
"""Query crates.io for a crate's license."""
|
||||
data = _get_json(f"https://crates.io/api/v1/crates/{name}/{version}", "crates.io")
|
||||
if data and "version" in data:
|
||||
return data["version"].get("license")
|
||||
# fallback: try crate-level
|
||||
data = _get_json(f"https://crates.io/api/v1/crates/{name}", "crates.io")
|
||||
if data and "crate" in data:
|
||||
versions = data.get("versions", [])
|
||||
for v in versions:
|
||||
if v.get("num") == version:
|
||||
return v.get("license")
|
||||
# last resort: latest version license
|
||||
if versions:
|
||||
return versions[0].get("license")
|
||||
return None
|
||||
|
||||
|
||||
def lookup_npm(name: str, version: str) -> str | None:
|
||||
"""Query npm registry for a package's license."""
|
||||
encoded = urllib.parse.quote(name, safe="")
|
||||
data = _get_json(
|
||||
f"https://registry.npmjs.org/{encoded}/{version}", "registry.npmjs.org"
|
||||
)
|
||||
if data:
|
||||
lic = data.get("license")
|
||||
if isinstance(lic, dict):
|
||||
return lic.get("type")
|
||||
if isinstance(lic, str):
|
||||
return lic
|
||||
return None
|
||||
|
||||
|
||||
def lookup_pypi(name: str, version: str) -> str | None:
|
||||
"""Query PyPI for a package's license."""
|
||||
data = _get_json(f"https://pypi.org/pypi/{name}/{version}/json", "pypi.org")
|
||||
if data and "info" in data:
|
||||
lic = data["info"].get("license")
|
||||
if lic and len(lic) < 100: # skip full license texts
|
||||
return lic
|
||||
# Try classifiers
|
||||
for c in data["info"].get("classifiers", []):
|
||||
if c.startswith("License :: OSI Approved :: "):
|
||||
return c.split(" :: ")[-1]
|
||||
return None
|
||||
|
||||
|
||||
def resolve_go_name(full_name: str) -> str | None:
|
||||
"""Resolve a Go module name to a known license via GO_KNOWN map."""
|
||||
# Try exact match first
|
||||
if full_name in GO_KNOWN:
|
||||
return GO_KNOWN[full_name]
|
||||
# Try stripping version suffix (e.g. github.com/foo/bar/v2 -> github.com/foo/bar)
|
||||
parts = full_name.split("/")
|
||||
for i in range(len(parts), 1, -1):
|
||||
candidate = "/".join(parts[:i])
|
||||
if candidate in GO_KNOWN:
|
||||
return GO_KNOWN[candidate]
|
||||
# Try just org/repo for github.com paths
|
||||
if full_name.startswith("github.com/") and len(parts) >= 3:
|
||||
base = "/".join(parts[:3])
|
||||
if base in GO_KNOWN:
|
||||
return GO_KNOWN[base]
|
||||
return None
|
||||
|
||||
|
||||
def resolve_component(comp: dict) -> str | None:
|
||||
"""Try to resolve a license for a component."""
|
||||
name = comp.get("name", "")
|
||||
version = comp.get("version", "")
|
||||
purl = comp.get("purl", "")
|
||||
comp_type = comp.get("type", "")
|
||||
|
||||
# 1. Check hardcoded known licenses
|
||||
if name in KNOWN_LICENSES:
|
||||
return KNOWN_LICENSES[name]
|
||||
|
||||
# 2. Route by purl type
|
||||
if purl.startswith("pkg:cargo/"):
|
||||
return lookup_cargo(name, version)
|
||||
if purl.startswith("pkg:npm/"):
|
||||
return lookup_npm(name, version)
|
||||
if purl.startswith("pkg:pypi/"):
|
||||
return lookup_pypi(name, version)
|
||||
if purl.startswith("pkg:golang/"):
|
||||
return resolve_go_name(name)
|
||||
if purl.startswith("pkg:deb/"):
|
||||
# Check known map by base package name
|
||||
base = name.split(":")[0] # strip arch qualifier
|
||||
if base in KNOWN_LICENSES:
|
||||
return KNOWN_LICENSES[base]
|
||||
return None
|
||||
if comp_type == "operating-system":
|
||||
return KNOWN_LICENSES.get(name)
|
||||
if not purl:
|
||||
return KNOWN_LICENSES.get(name)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def set_license(comp: dict, license_id: str) -> None:
|
||||
"""Set the license on a component, replacing hash or empty."""
|
||||
comp["licenses"] = [{"license": {"id": license_id}}]
|
||||
|
||||
|
||||
def needs_fix(comp: dict) -> bool:
|
||||
"""Check if component has missing or hash-based license."""
|
||||
licenses = comp.get("licenses", [])
|
||||
if not licenses:
|
||||
return True
|
||||
for entry in licenses:
|
||||
lic = entry.get("license", {})
|
||||
lid = lic.get("id", "")
|
||||
lname = lic.get("name", "")
|
||||
if lid.startswith("sha256:") or lname.startswith("sha256:"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _find_sbom_files() -> list[Path]:
|
||||
"""Find SBOM JSON files in the default output directory."""
|
||||
repo_root = Path(__file__).resolve().parent.parent.parent
|
||||
output_dir = repo_root / "deploy" / "sbom" / "output"
|
||||
return sorted(output_dir.glob("*.cdx.json"))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
files = [Path(p) for p in sys.argv[1:]] if len(sys.argv) > 1 else _find_sbom_files()
|
||||
|
||||
if not files:
|
||||
print("No SBOM JSON files found.")
|
||||
print("Run 'mise run sbom:generate' first, or pass file paths as arguments.")
|
||||
sys.exit(1)
|
||||
|
||||
# Collect unique components needing fixes
|
||||
to_resolve: dict[str, dict] = {} # key -> representative component
|
||||
for f in files:
|
||||
with f.open() as fh:
|
||||
sbom = json.load(fh)
|
||||
for comp in sbom.get("components", []):
|
||||
if needs_fix(comp):
|
||||
key = f"{comp.get('name', '')}@{comp.get('version', '')}"
|
||||
if key not in to_resolve:
|
||||
to_resolve[key] = comp
|
||||
|
||||
total = len(to_resolve)
|
||||
print(f"Found {total} unique components needing license resolution.")
|
||||
print()
|
||||
|
||||
if total == 0:
|
||||
print("All licenses already resolved.")
|
||||
return
|
||||
|
||||
# Resolve licenses
|
||||
resolved: dict[str, str] = {} # key -> license_id
|
||||
failed: list[str] = []
|
||||
|
||||
for i, (key, comp) in enumerate(sorted(to_resolve.items()), 1):
|
||||
sys.stdout.write(f"\r [{i}/{total}] Resolving {key[:60]:<60}")
|
||||
sys.stdout.flush()
|
||||
|
||||
license_id = resolve_component(comp)
|
||||
if license_id:
|
||||
resolved[key] = license_id
|
||||
else:
|
||||
failed.append(key)
|
||||
|
||||
print(f"\n\nResolved: {len(resolved)}/{total}")
|
||||
print(f"Unresolved: {len(failed)}/{total}")
|
||||
|
||||
# Apply to all files
|
||||
total_patched = 0
|
||||
for f in files:
|
||||
with f.open() as fh:
|
||||
sbom = json.load(fh)
|
||||
|
||||
patched = 0
|
||||
for comp in sbom.get("components", []):
|
||||
if needs_fix(comp):
|
||||
key = f"{comp.get('name', '')}@{comp.get('version', '')}"
|
||||
if key in resolved:
|
||||
set_license(comp, resolved[key])
|
||||
patched += 1
|
||||
|
||||
with f.open("w") as fh:
|
||||
json.dump(sbom, fh, indent=2)
|
||||
fh.write("\n")
|
||||
|
||||
total_patched += patched
|
||||
print(f" {f.name}: patched {patched} components")
|
||||
|
||||
print(f"\nTotal patches applied: {total_patched}")
|
||||
|
||||
if failed:
|
||||
print(f"\n--- Unresolved ({len(failed)}) ---")
|
||||
for key in sorted(failed):
|
||||
comp = to_resolve[key]
|
||||
print(f" {key} purl={comp.get('purl', '(none)')}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
"""Convert CycloneDX SBOM JSON files to CSV.
|
||||
|
||||
Usage:
|
||||
python sbom_to_csv.py # convert all *.cdx.json in deploy/sbom/output/
|
||||
python sbom_to_csv.py file1.json ... # convert specific files
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def extract_licenses(component: dict) -> str:
|
||||
"""Pull license IDs/names from CycloneDX license entries."""
|
||||
licenses = component.get("licenses", [])
|
||||
ids = []
|
||||
for entry in licenses:
|
||||
lic = entry.get("license", {})
|
||||
ids.append(lic.get("id") or lic.get("name", ""))
|
||||
return " | ".join(filter(None, ids))
|
||||
|
||||
|
||||
def sbom_to_csv(json_path: Path) -> Path:
|
||||
"""Read a CycloneDX JSON SBOM and write a CSV beside it."""
|
||||
with json_path.open() as f:
|
||||
sbom = json.load(f)
|
||||
|
||||
csv_path = json_path.with_suffix(".csv")
|
||||
components = sbom.get("components", [])
|
||||
|
||||
with csv_path.open("w", newline="") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow(["name", "version", "type", "purl", "licenses", "bom-ref"])
|
||||
|
||||
for comp in components:
|
||||
writer.writerow(
|
||||
[
|
||||
comp.get("name", ""),
|
||||
comp.get("version", ""),
|
||||
comp.get("type", ""),
|
||||
comp.get("purl", ""),
|
||||
extract_licenses(comp),
|
||||
comp.get("bom-ref", ""),
|
||||
]
|
||||
)
|
||||
|
||||
return csv_path
|
||||
|
||||
|
||||
def _find_sbom_files() -> list[Path]:
|
||||
"""Find SBOM JSON files in the default output directory."""
|
||||
repo_root = Path(__file__).resolve().parent.parent.parent
|
||||
output_dir = repo_root / "deploy" / "sbom" / "output"
|
||||
return sorted(output_dir.glob("*.cdx.json"))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
files = [Path(p) for p in sys.argv[1:]] if len(sys.argv) > 1 else _find_sbom_files()
|
||||
|
||||
if not files:
|
||||
print("No SBOM JSON files found.")
|
||||
print("Run 'mise run sbom:generate' first, or pass file paths as arguments.")
|
||||
sys.exit(1)
|
||||
|
||||
for path in files:
|
||||
csv_path = sbom_to_csv(path)
|
||||
with csv_path.open() as count_fh:
|
||||
components_count = sum(1 for _ in count_fh) - 1 # minus header
|
||||
print(f"{path.name} -> {csv_path.name} ({components_count} components)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -19,6 +19,7 @@ uv = "0.10.2"
|
||||
protoc = "29.6"
|
||||
helm = "4.1.1"
|
||||
"ubi:mozilla/sccache" = { version = "0.14.0", matching = "sccache-v" }
|
||||
"ubi:anchore/syft" = { version = "1.42.2", matching = "syft_" }
|
||||
|
||||
[env]
|
||||
_.path = ["{{config_root}}/scripts/bin"]
|
||||
@@ -42,7 +43,7 @@ DOCKER_BUILDKIT = "1"
|
||||
|
||||
[vars]
|
||||
# Python paths to include in formatting/linting
|
||||
python_paths = "python/ tasks/scripts/*.py"
|
||||
python_paths = "python/ tasks/scripts/*.py deploy/sbom/*.py"
|
||||
|
||||
[task_config]
|
||||
includes = ["tasks/*.toml"]
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# SBOM generation, license resolution, and CSV export tasks
|
||||
#
|
||||
# Usage:
|
||||
# mise run sbom # generate SBOMs, resolve licenses, export CSVs
|
||||
# mise run sbom:check # advisory license check (for CI)
|
||||
|
||||
[sbom]
|
||||
description = "Generate SBOMs, resolve licenses, and export CSVs to deploy/sbom/output/"
|
||||
depends = ["sbom:generate", "sbom:resolve", "sbom:csv"]
|
||||
|
||||
["sbom:generate"]
|
||||
description = "Generate CycloneDX SBOMs with Syft"
|
||||
hide = true
|
||||
run = """
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
OUTPUT_DIR="deploy/sbom/output"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
VERSION=$(uv run python tasks/scripts/release.py get-version --cargo)
|
||||
|
||||
echo "Generating SBOM for workspace (version ${VERSION})..."
|
||||
|
||||
syft dir:. \
|
||||
--output "cyclonedx-json=$OUTPUT_DIR/openshell-source-${VERSION}.cdx.json" \
|
||||
--source-name openshell \
|
||||
--source-version "$VERSION"
|
||||
|
||||
echo ""
|
||||
echo "SBOM written to $OUTPUT_DIR/"
|
||||
ls -la "$OUTPUT_DIR"/*.cdx.json
|
||||
"""
|
||||
|
||||
["sbom:resolve"]
|
||||
description = "Resolve missing licenses in SBOM JSON files via public registries"
|
||||
hide = true
|
||||
run = "uv run python deploy/sbom/resolve_licenses.py"
|
||||
|
||||
["sbom:csv"]
|
||||
description = "Convert SBOM JSON files to CSV"
|
||||
hide = true
|
||||
run = "uv run python deploy/sbom/sbom_to_csv.py"
|
||||
|
||||
["sbom:check"]
|
||||
description = "Check SBOMs for unresolved licenses (advisory, non-blocking)"
|
||||
hide = true
|
||||
run = """
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
OUTPUT_DIR="deploy/sbom/output"
|
||||
|
||||
if [ ! -d "$OUTPUT_DIR" ] || [ -z "$(ls -A "$OUTPUT_DIR"/*.cdx.json 2>/dev/null)" ]; then
|
||||
echo "No SBOM files found in $OUTPUT_DIR/. Run 'mise run sbom' first."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Checking for unresolved licenses..."
|
||||
UNRESOLVED=0
|
||||
for f in "$OUTPUT_DIR"/*.cdx.json; do
|
||||
COUNT=$(uv run python -c "
|
||||
import json, sys
|
||||
with open('$f') as fh:
|
||||
sbom = json.load(fh)
|
||||
missing = 0
|
||||
for c in sbom.get('components', []):
|
||||
lics = c.get('licenses', [])
|
||||
if not lics:
|
||||
missing += 1
|
||||
continue
|
||||
for e in lics:
|
||||
lid = e.get('license', {}).get('id', '')
|
||||
lname = e.get('license', {}).get('name', '')
|
||||
if lid.startswith('sha256:') or lname.startswith('sha256:'):
|
||||
missing += 1
|
||||
break
|
||||
print(missing)
|
||||
")
|
||||
if [ "$COUNT" -gt 0 ]; then
|
||||
echo " $(basename "$f"): $COUNT components with unresolved licenses"
|
||||
UNRESOLVED=$((UNRESOLVED + COUNT))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$UNRESOLVED" -gt 0 ]; then
|
||||
echo ""
|
||||
echo "WARNING: $UNRESOLVED total components with unresolved licenses."
|
||||
echo "This is advisory -- not blocking the build."
|
||||
else
|
||||
echo "All licenses resolved."
|
||||
fi
|
||||
"""
|
||||
Reference in New Issue
Block a user