mirror of
https://github.com/ankitects/anki.git
synced 2026-10-22 06:49:59 +08:00
Fernando Lins
587d6ec796
## Linked issue Follow-up to #5498, which was closed in favor of a smaller and more maintainable implementation. ## Summary / motivation Run CI-based SonarCloud analysis for internal and fork pull requests while keeping the Quality Gate informational. The analysis matrix is: | Event | Static analysis | Coverage in CI | Coverage in SonarCloud | | --- | --- | --- | --- | | Internal pull request | Yes | Yes | Yes | | Push to `main` or `release/**` | Yes | Yes | Yes | | Fork pull request | Yes | Yes | No | Fork pull requests continue to run the full test coverage and coverage-regression checks in the unprivileged CI workflow. Their contributor-produced coverage reports are not passed to the privileged SonarCloud workflow. The fork scan reports static bugs, vulnerabilities, security hotspots, and code smells. It: - validates the PR state, repositories, branches, and tested SHA against the GitHub API; - checks out the exact revision tested by CI; - replaces `sonar-project.properties` with the trusted default-branch version; - provides `SONAR_TOKEN` only to the scanner step; - never builds, installs, or executes contributor code; - keeps scanner-side Clippy and SCA disabled; - clears coverage report paths and excludes coverage calculation for forks. Automatic Analysis remains disabled because it cannot be combined with CI-based analysis and does not provide the required Rust and coverage support. This also configures Vitest to produce repository-relative LCOV paths such as `SF:ts/lib/...`, allowing SonarCloud to resolve TypeScript files when reports are downloaded into a separate checkout. ## How to test ### Checklist - [x] I ran `just check` or an equivalent relevant check locally. - [ ] I added or updated tests when the change is non-trivial or behavior changed. The remaining behavior is specific to GitHub's `workflow_run` environment and can only be exercised end-to-end after the workflow is available on the default branch. ### Details The following checks passed locally: - `just test-ts --coverage` - `just test-ts --coverage --html` - `just fmt` - `just check` - YAML and JSON parsing - `git diff --check` Both TypeScript coverage runs passed all 67 tests. The generated LCOV report was also checked to confirm that source paths are repository-relative and begin with `SF:ts/`. After this lands, end-to-end validation should cover: 1. A push to `main`, confirming that Python, TypeScript, and Rust coverage reports are imported. 2. An internal PR, confirming static analysis and coverage. 3. A fork PR, confirming static analysis while the coverage download is skipped and no coverage condition is evaluated. ## Before / after behavior Before: - fork PR analysis fails at the protected checkout; - the `actions/checkout` version comments are outdated; - TypeScript LCOV paths are relative to `ts/` and may not resolve in the SonarCloud checkout. After: - internal PRs and trusted branch pushes receive static analysis and coverage; - fork PRs receive static analysis without importing contributor-produced coverage; - fork tests and coverage-regression checks continue to run in regular CI; - TypeScript LCOV paths resolve from the repository root; - technical scanner failures remain visible without making the Sonar Quality Gate a required merge check. ## Risk / compatibility / migration The fork analysis runs in a privileged `workflow_run` and necessarily asks the SonarScanner to parse untrusted source while it has access to a project-scoped token. The workflow limits this exposure by using GitHub-hosted runners, read-only permissions, trusted scanner configuration, an exact tested SHA, API-validated PR metadata, and no builds, dependency installation, local Actions, caches, or fork-produced coverage artifacts. This is intentionally a smaller trust boundary than the data-only snapshot and custom coverage parser proposed in #5498. It prevents direct execution of contributor code but does not eliminate potential vulnerabilities in GitHub Actions or Sonar analyzers. Automatic Analysis must remain disabled, `SONAR_TOKEN` should remain restricted to this project, and the SonarCloud check must remain non-required.
92 lines
4.0 KiB
Plaintext
92 lines
4.0 KiB
Plaintext
---
|
|
title: "Testing and Coverage"
|
|
---
|
|
|
|
CI runs Rust, Python, and TypeScript tests plus lint/type checks. Coverage is
|
|
orchestrated in `justfile` with direct CLI calls, using ninja only to prepare
|
|
generated build artifacts that the test commands need.
|
|
|
|
## Quick reference
|
|
|
|
```sh
|
|
just test # run all tests (no coverage)
|
|
just test --coverage # run all tests + enforce coverage thresholds
|
|
just test --coverage --html # same + generate HTML reports under out/coverage/
|
|
|
|
just test-rust # Rust only
|
|
just test-rust --coverage
|
|
just test-rust --coverage --html
|
|
|
|
just test-py # Python (pylib + qt) only
|
|
just test-py --coverage
|
|
just test-py --coverage --html
|
|
|
|
just test-ts # TypeScript/Svelte Vitest only
|
|
just test-ts --coverage
|
|
just test-ts --coverage --html
|
|
```
|
|
|
|
HTML reports are written under `out/coverage/` (gitignored).
|
|
|
|
## Coverage tools and thresholds
|
|
|
|
| Stack | Test runner | Coverage tool | Minimum |
|
|
| ------------------- | ------------------------------------ | ---------------- | ------: |
|
|
| Rust workspace | `cargo nextest` via `cargo-llvm-cov` | `cargo-llvm-cov` | 60% |
|
|
| Python `pylib/anki` | `pytest pylib/tests` | `coverage.py` | 63% |
|
|
| Python `qt/aqt` | `pytest qt/tests` | `coverage.py` | 20% |
|
|
| TypeScript/Svelte | `vitest run` | Vitest V8 | 5% |
|
|
|
|
Linux pull requests run `just test --coverage` in CI. macOS and Windows
|
|
jobs run `just test` (no coverage enforcement) for now.
|
|
|
|
## Notes
|
|
|
|
- **Rust** — `cargo-llvm-cov` is installed on demand into `out/bin/` to avoid
|
|
polluting the global cargo install. Coverage runs rebuild the workspace with
|
|
instrumentation, so they are slower than plain `just test-rust`.
|
|
**Windows ARM64 (`aarch64-pc-windows-msvc`) is not supported**: the Rust
|
|
compiler produces malformed `.profraw` files on that target
|
|
([rust-lang/rust#150123](https://github.com/rust-lang/rust/issues/150123),
|
|
[cargo-llvm-cov#436](https://github.com/taiki-e/cargo-llvm-cov/issues/436)).
|
|
`just test-rust --coverage` exits with a clear message on ARM64 Windows;
|
|
use `just test-rust` (no coverage) or rely on CI (Linux) for enforcement.
|
|
- **Python** — coverage is split across two suites (`pylib` and `qt`) because
|
|
they have different `PYTHONPATH` setups and test folders.
|
|
- **TypeScript** — coverage is measured only over code reachable through
|
|
Vitest's module graph. Svelte component rendering behavior is not covered.
|
|
|
|
## SonarCloud
|
|
|
|
SonarCloud runs automatically after successful CI pushes and pull requests. It
|
|
runs static analysis across Python, TypeScript, and Rust. For internal pull
|
|
requests and trusted branch pushes, it also aggregates the coverage reports
|
|
generated by the three stacks.
|
|
|
|
- **Dashboard**: https://sonarcloud.io/project/overview?id=ankitects_anki
|
|
- **Internal coverage**: uses the same LCOV/XML reports produced by `just test --coverage`
|
|
- **Fork coverage**: remains enforced in CI but is not imported into SonarCloud
|
|
- **Analysis results**: SonarCloud reports quality gate status on each analyzed PR
|
|
|
|
The quality gate is configured as non-blocking — a failing gate does not prevent
|
|
merging, but the results are visible on the PR.
|
|
|
|
See [SonarCloud analysis](/developers/sonarcloud) for the analysis matrix and the fork trust
|
|
boundary.
|
|
|
|
## Gaps and future improvements
|
|
|
|
- Raise thresholds gradually as the test suite grows and CI timings stabilise.
|
|
- Exclude generated files from coverage denominators where appropriate.
|
|
- Publish `out/coverage/` as a CI artifact so reviewers can browse HTML
|
|
reports directly from a PR.
|
|
- Consider diff/changed-file coverage once baselines are stable — it is a
|
|
better enforcement mechanism for incremental improvement than whole-repo
|
|
thresholds.
|
|
- Add component or browser tests for Svelte UI surfaces if Svelte coverage
|
|
is intended to cover rendered component behaviour.
|
|
|
|
## See also
|
|
|
|
- [Writing Unit Tests for Anki](/developers/unit-testing)
|