Clarify end-to-end test expectations (#1952)

## Description

Clarify that mocks are appropriate for focused interface and wiring
coverage, but tests claiming backend or runtime behavior must execute
the real implementation. Document `torch.compile` as a concrete example:
call the real compiler, and wrap/delegate to it when call tracing is
needed.

This follows the testing guidance raised in #1550.

## Validation

- `git diff --check`
- `pre-commit run --files CONTRIBUTING.md`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Clarified test design guidance to encourage validating real behavior
end-to-end.
* Added advice to avoid substituting core runtime behavior with fakes
when tests should cover actual execution.
* Noted that wrapping or delegation is still appropriate when tests need
to observe calls or counts.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: realAsma <akuriparambi@nvidia.com>
This commit is contained in:
realAsma
2026-07-09 11:17:44 -07:00
committed by GitHub
parent e911c3b7fd
commit 9c40c34674
+5
View File
@@ -169,6 +169,11 @@ nox -s "unit-3.12(torch_211, tf_latest)"
in. Checked-in tests should document expected behavior, protect against regressions, or flag backward-incompatible
behavior changes. Remove redundant lower-level tests when a higher-level test already covers the same behavior,
keeping CI/CD fast and lean.
- **Exercise the behavior a test claims to validate.** Mocks are useful for focused interface and wiring coverage, but
replacing the implementation under test does not validate its real behavior. Include an end-to-end test that runs
the actual implementation whenever the test claims backend or runtime behavior. For example, a test of
`torch.compile` execution must invoke the real `torch.compile`; if the call also needs to be counted or traced, wrap
and delegate to the original function instead of replacing it with a fake.
- **Keep `tests/unit` offline — no HuggingFace Hub access.** Unit tests must be hermetic so they never flake on
network/timeout issues. Do not call `from_pretrained("<org>/<model>")`, `load_dataset("<hub-id>")`,
`snapshot_download(...)`, etc. with Hub IDs. Instead build dummy models, tokenizers, configs, and datasets locally —