mirror of https://github.com/NVIDIA/Model-Optimizer.git synced 2026-10-30 07:34:44 +08:00

Files

T

Chenjie Luo 7c8557158d Add job cancellation support to the debugger command relay (#1262 )

## Summary

- Add a `cancel` subcommand to the client that terminates the currently
running command on the server
- Server now runs commands in the background with PID tracking, enabling
cancellation mid-execution
- Client-side timeouts automatically cancel the running command on the
server (previously the server process was left running)
- Hardened against race conditions through 4 rounds of adversarial
review (15 fixes total)

### Key changes

**server.sh:**
- Commands run in background with PID tracked in `$RELAY_DIR/running`
(atomic tmp+mv write)
- Cancel detection loop checks for `$RELAY_DIR/cancel` file with cmd_id
verification
- SIGTERM with 5s grace period, then SIGKILL escalation for stuck
processes
- `.exit` file written before `running` marker removed (ordering
guarantee)
- `set -e`-safe: `wait` uses `|| exit_code=$?` pattern; cleanup trap
fully guarded
- Stale cancel files cleared at command start; mismatched/empty signals
rejected
- Command file read into memory and removed before execution (eliminates
TOCTOU with client timeout)

**client.sh:**
- New `cancel` subcommand: writes target cmd_id to cancel file, waits
for server acknowledgment (30s timeout)
- `run` timeout now sends targeted cancel signal (verifies cmd_id match
to avoid killing wrong command)
- `run` timeout cleans up orphaned result files
- `status` shows currently running command
- `flush` rejects if a command is currently running (prevents state
corruption)
- Exit code validated as numeric before use

### Protocol additions

```
.relay/
├── running    # server writes cmd_id:pid while executing (atomic)
├── cancel     # client writes target cmd_id to request cancellation
```

## Test plan

- [ ] Start server in Docker, handshake from host
- [ ] Run a command (`client.sh run "sleep 30"`), cancel it (`client.sh
cancel`), verify exit code 130
- [ ] Run a command with short timeout (`--timeout 5 run "sleep 30"`),
verify auto-cancel
- [ ] Run a command that exits non-zero, verify server stays alive
- [ ] Run `status` during execution, verify it shows the running command
- [ ] Attempt `flush` during execution, verify it is rejected
- [ ] Cancel when nothing is running, verify clean message

🤖 Generated with [Claude Code](https://claude.com/claude-code)

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: ✅
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: N/A (bash scripts for dev
tooling, tested manually)
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A (internal tooling)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added a comprehensive debug skill guide and protocol reference with
quick CLI examples and a new “Cancelling Commands” section.

* **New Features**
* Client-side `cancel` command to terminate the currently running remote
command.
  * Status now reports active command (or `(idle)`).

* **Improvements**
* Stronger startup/validation guidance, safer shutdown/cleanup,
deterministic cancel exit semantics (130), and auto-cancel on
client-side timeout.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>

2026-04-15 16:49:42 +00:00

2.0 KiB

Raw Permalink Blame History

Remote Command Relay

This directory contains a file-based command relay for executing commands inside a remote Docker container from the host machine (where Claude Code runs).

How to Use (for Claude Code)

Setup (one-time per session)

The user must start the server inside Docker first:

# Inside Docker container (auto-detects repo root from script location):
bash /path/to/modelopt/tools/debugger/server.sh

Then Claude Code performs the handshake:

bash tools/debugger/client.sh handshake

Running Commands

# Run any command in the Docker container (workdir = auto-detected repo root):
bash tools/debugger/client.sh run "<command>"

# For long-running tasks, increase timeout:
bash tools/debugger/client.sh --timeout 1800 run "<command>"

Key Paths Inside Docker

Path	Description
Repo root (auto-detected)	ModelOpt source, used as workdir
`/hf-local`	HuggingFace model cache

Examples

# Run PTQ test
bash tools/debugger/client.sh run "bash llm_ptq/scripts/huggingface_example.sh"

# Run pytest
bash tools/debugger/client.sh run "python -m pytest tests/gpu -k test_quantize"

# Check GPU
bash tools/debugger/client.sh run "nvidia-smi"

# Use HF models from local cache
bash tools/debugger/client.sh run "python script.py --model /hf-local/Qwen/Qwen3-8B"

Cancelling Commands

# Cancel the currently running command
bash tools/debugger/client.sh cancel

# Client-side timeout also auto-cancels the running command

Important Notes

The server must be started by the user manually inside Docker before the handshake.
Default command timeout is 600 seconds (10 minutes). Use --timeout for longer tasks.
Commands execute sequentially — one at a time.
A running command can be cancelled; cancelled commands exit with code 130.
All commands run with the auto-detected repo root as the working directory.
The .relay/ directory is ephemeral and git-ignored.

2.0 KiB Raw Permalink Blame History