Files
Model-Optimizer/tools/debugger/client.sh
T
Chenjie Luo 7c8557158d Add job cancellation support to the debugger command relay (#1262)
## Summary

- Add a `cancel` subcommand to the client that terminates the currently
running command on the server
- Server now runs commands in the background with PID tracking, enabling
cancellation mid-execution
- Client-side timeouts automatically cancel the running command on the
server (previously the server process was left running)
- Hardened against race conditions through 4 rounds of adversarial
review (15 fixes total)

### Key changes

**server.sh:**
- Commands run in background with PID tracked in `$RELAY_DIR/running`
(atomic tmp+mv write)
- Cancel detection loop checks for `$RELAY_DIR/cancel` file with cmd_id
verification
- SIGTERM with 5s grace period, then SIGKILL escalation for stuck
processes
- `.exit` file written before `running` marker removed (ordering
guarantee)
- `set -e`-safe: `wait` uses `|| exit_code=$?` pattern; cleanup trap
fully guarded
- Stale cancel files cleared at command start; mismatched/empty signals
rejected
- Command file read into memory and removed before execution (eliminates
TOCTOU with client timeout)

**client.sh:**
- New `cancel` subcommand: writes target cmd_id to cancel file, waits
for server acknowledgment (30s timeout)
- `run` timeout now sends targeted cancel signal (verifies cmd_id match
to avoid killing wrong command)
- `run` timeout cleans up orphaned result files
- `status` shows currently running command
- `flush` rejects if a command is currently running (prevents state
corruption)
- Exit code validated as numeric before use

### Protocol additions

```
.relay/
├── running    # server writes cmd_id:pid while executing (atomic)
├── cancel     # client writes target cmd_id to request cancellation
```

## Test plan

- [ ] Start server in Docker, handshake from host
- [ ] Run a command (`client.sh run "sleep 30"`), cancel it (`client.sh
cancel`), verify exit code 130
- [ ] Run a command with short timeout (`--timeout 5 run "sleep 30"`),
verify auto-cancel
- [ ] Run a command that exits non-zero, verify server stays alive
- [ ] Run `status` during execution, verify it shows the running command
- [ ] Attempt `flush` during execution, verify it is rejected
- [ ] Cancel when nothing is running, verify clean message

🤖 Generated with [Claude Code](https://claude.com/claude-code)

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: ✅
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: N/A (bash scripts for dev
tooling, tested manually)
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A (internal tooling)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added a comprehensive debug skill guide and protocol reference with
quick CLI examples and a new “Cancelling Commands” section.

* **New Features**
* Client-side `cancel` command to terminate the currently running remote
command.
  * Status now reports active command (or `(idle)`).

* **Improvements**
* Stronger startup/validation guidance, safer shutdown/cleanup,
deterministic cancel exit semantics (130), and auto-cancel on
client-side timeout.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
2026-04-15 16:49:42 +00:00

260 lines
9.3 KiB
Bash
Executable File

#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# File-based command relay client.
# Run this from the host / Claude Code side. It sends commands to the server
# running inside Docker by writing files to the shared relay directory.
#
# Usage:
# bash client.sh handshake - Connect to server
# bash client.sh run <command...> - Run a command and print output
# bash client.sh cancel - Cancel the running command
# bash client.sh status - Check server status
#
# Options:
# --relay-dir <path> Path to relay directory (default: <script_dir>/.relay)
# --timeout <secs> Timeout waiting for result (default: 600)
set -euo pipefail
RELAY_DIR=""
TIMEOUT=600
POLL_INTERVAL=1
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Parse global options before subcommand
while [[ $# -gt 0 ]]; do
case "$1" in
--relay-dir) RELAY_DIR="$2"; shift 2 ;;
--timeout) TIMEOUT="$2"; shift 2 ;;
*) break ;;
esac
done
if [[ -z "$RELAY_DIR" ]]; then
RELAY_DIR="$SCRIPT_DIR/.relay"
fi
CMD_DIR="$RELAY_DIR/cmd"
RESULT_DIR="$RELAY_DIR/result"
SUBCOMMAND="${1:-}"
shift || true
# Helper: wait for a specific command to finish (running marker gone or cmd_id changed)
wait_for_cancel_completion() {
local target_id="$1" wait_timeout="$2" elapsed=0 current_info current_id
while [[ $elapsed -lt $wait_timeout ]]; do
if [[ ! -f "$RELAY_DIR/running" ]]; then
return 0
fi
current_info=$(cat "$RELAY_DIR/running" 2>/dev/null) || true
current_id="${current_info%%:*}"
[[ "$current_id" != "$target_id" ]] && return 0
sleep "$POLL_INTERVAL"
elapsed=$((elapsed + POLL_INTERVAL))
done
return 1
}
case "$SUBCOMMAND" in
handshake)
# Check server is ready
if [[ ! -f "$RELAY_DIR/server.ready" ]]; then
echo "ERROR: Server not ready. Start server.sh in Docker first."
exit 1
fi
SERVER_INFO=$(cat "$RELAY_DIR/server.ready")
echo "Server found: $SERVER_INFO"
# Send client handshake
echo "$(hostname):$$:$(date -Iseconds)" > "$RELAY_DIR/client.ready"
# Wait for server acknowledgment
elapsed=0
while [[ ! -f "$RELAY_DIR/handshake.done" ]]; do
sleep "$POLL_INTERVAL"
elapsed=$((elapsed + POLL_INTERVAL))
if [[ $elapsed -ge 120 ]]; then
echo "ERROR: Handshake timed out after 120s."
exit 1
fi
done
echo "Handshake complete."
;;
run)
# Verify handshake was done
if [[ ! -f "$RELAY_DIR/handshake.done" ]]; then
echo "ERROR: Not connected. Run 'client.sh handshake' first."
exit 1
fi
# Generate a unique command ID (timestamp + PID to avoid collisions)
cmd_id="$(date +%s%N)_$$"
echo "[client] Running: $*"
# Write the command file atomically (tmp + mv)
echo "$*" > "$CMD_DIR/$cmd_id.sh.tmp"
mv "$CMD_DIR/$cmd_id.sh.tmp" "$CMD_DIR/$cmd_id.sh"
# Wait for result
elapsed=0
while [[ ! -f "$RESULT_DIR/$cmd_id.exit" ]]; do
# Check if server is still alive
if [[ ! -f "$RELAY_DIR/server.ready" ]]; then
echo "ERROR: Server appears to have stopped."
rm -f "$CMD_DIR/$cmd_id.sh"
exit 1
fi
sleep "$POLL_INTERVAL"
elapsed=$((elapsed + POLL_INTERVAL))
if [[ $elapsed -ge $TIMEOUT ]]; then
# Result might have arrived during the last sleep
[[ -f "$RESULT_DIR/$cmd_id.exit" ]] && break
echo "ERROR: Command timed out after ${TIMEOUT}s."
# Cancel the running command only if it is OUR command
if [[ -f "$RELAY_DIR/running" ]]; then
running_info=$(cat "$RELAY_DIR/running" 2>/dev/null) || true
if [[ -n "$running_info" && "$running_info" == *:* ]]; then
running_id="${running_info%%:*}"
if [[ "$running_id" == "$cmd_id" ]]; then
echo "Sending cancel signal..."
echo "$cmd_id" > "$RELAY_DIR/cancel.tmp"
mv "$RELAY_DIR/cancel.tmp" "$RELAY_DIR/cancel"
wait_for_cancel_completion "$cmd_id" 10 || true
fi
fi
fi
# Clean up command and any orphaned result files
rm -f "$CMD_DIR/$cmd_id.sh"
rm -f "$RESULT_DIR/$cmd_id.exit" "$RESULT_DIR/$cmd_id.log"
exit 1
fi
done
# Read and display results
exit_code=$(cat "$RESULT_DIR/$cmd_id.exit")
if ! [[ "$exit_code" =~ ^[0-9]+$ ]]; then
echo "WARNING: Invalid exit code '$exit_code', defaulting to 1."
exit_code=1
fi
if [[ -f "$RESULT_DIR/$cmd_id.log" ]]; then
cat "$RESULT_DIR/$cmd_id.log"
fi
# Clean up result files
rm -f "$RESULT_DIR/$cmd_id.exit" "$RESULT_DIR/$cmd_id.log"
exit "$exit_code"
;;
status)
if [[ -f "$RELAY_DIR/server.ready" ]]; then
echo "Server: $(cat "$RELAY_DIR/server.ready")"
else
echo "Server: not running"
fi
if [[ -f "$RELAY_DIR/handshake.done" ]]; then
echo "Handshake: complete"
elif [[ -f "$RELAY_DIR/client.ready" ]]; then
echo "Handshake: pending"
else
echo "Handshake: not started"
fi
if [[ -f "$RELAY_DIR/running" ]]; then
running_info=$(cat "$RELAY_DIR/running" 2>/dev/null) || running_info="(disappeared)"
echo "Running: $running_info"
else
echo "Running: (idle)"
fi
if [[ -d "$CMD_DIR" ]]; then
pending=$(find "$CMD_DIR" -maxdepth 1 -type f -name '*.sh' 2>/dev/null | wc -l)
else
pending=0
fi
echo "Pending commands: $pending"
;;
flush)
# Block flush if a command is actually running (server alive + running marker)
# Allow flush if server is dead (stale running marker from crash)
if [[ -f "$RELAY_DIR/running" ]] && [[ -f "$RELAY_DIR/server.ready" ]]; then
echo "ERROR: A command is currently running. Cancel it first or wait for it to finish."
exit 1
fi
if [[ -d "$RELAY_DIR" ]]; then
# Clear handshake and command/result files, but keep server.ready
rm -f "$RELAY_DIR/client.ready" "$RELAY_DIR/handshake.done"
rm -rf "$CMD_DIR" "$RESULT_DIR"
mkdir -p "$CMD_DIR" "$RESULT_DIR"
echo "Relay state cleared (server.ready preserved): $RELAY_DIR"
else
echo "Relay directory does not exist: $RELAY_DIR"
fi
;;
cancel)
# Check if there's a running command
if [[ -f "$RELAY_DIR/running" ]]; then
running_info=$(cat "$RELAY_DIR/running" 2>/dev/null) || true
if [[ -z "$running_info" || "$running_info" != *:* ]]; then
echo "WARNING: Running marker is corrupt or empty. Cannot identify command to cancel."
exit 1
fi
running_id="${running_info%%:*}"
echo "Cancelling running command: $running_id"
# Write cancel signal atomically with cmd_id so server can verify the target
echo "$running_id" > "$RELAY_DIR/cancel.tmp"
mv "$RELAY_DIR/cancel.tmp" "$RELAY_DIR/cancel"
# Wait for the server to process the cancellation
if wait_for_cancel_completion "$running_id" 30; then
echo "Command cancelled."
else
echo "WARNING: Cancel signal sent but command still running after 30s."
exit 1
fi
else
echo "No command is currently running."
fi
# Report pending commands
if [[ -d "$CMD_DIR" ]]; then
pending=$(find "$CMD_DIR" -maxdepth 1 -type f -name '*.sh' 2>/dev/null | wc -l)
if [[ "$pending" -gt 0 ]]; then
echo "$pending pending command(s) in queue. Use 'flush' to clear them."
fi
fi
;;
*)
echo "Usage: $0 [--relay-dir <path>] [--timeout <secs>] <subcommand>"
echo ""
echo "Subcommands:"
echo " handshake Connect to the server"
echo " run <cmd> Execute a command on the server"
echo " cancel Cancel the currently running command"
echo " status Check connection status"
echo " flush Clear the relay directory"
exit 1
;;
esac