rustfs/docs/testing/ftps-upload-memory.md

FTPS upload memory regression

The old FtpsDriver::put copied the entire input into a Vec before calling PutObject. Wrapping that completed allocation in StreamingBlob did not make input consumption streaming. It also sent large files through the 5 GiB single-PUT path.

Automated regression

cargo test -p rustfs-protocols --no-default-features --features ftps --lib

ftps_upload_streams_before_eof_and_preserves_part_bytes uses a reader that refuses to supply the next part until the previous part reaches the backend. Restoring the driver from commit 9c30cc88513c5e1b5443b6ee86bf4bc7965e79e9 while retaining the new test harness makes this test fail with read ahead of uploaded part. The multipart implementation passes. Tests also check exact bytes, empty files, part boundaries, part numbering, errors, permissions, cancellation cleanup, and buffer capacity.

Manual memory probe

The ignored test ftps_large_upload_memory_probe creates concurrent synthetic readers. Each reader yields between reads so all uploads are active together. It calls the real FTPS storage driver with a scripted storage backend that returns successful S3 responses and discards the payload. It neither stores files nor opens network connections. No large fixture is required.

Build the test executable first; measuring Cargo would include compiler memory:

export CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0
cargo test -p rustfs-protocols --no-default-features --features ftps --lib --no-run

Use the executable path printed by that command as test_binary, then on macOS:

/usr/bin/time -l "$test_binary" --ignored --exact \
  ftps::driver::upload::tests::ftps_large_upload_memory_probe --nocapture

RUSTFS_FTPS_TEST_BYTES=10737418240 RUSTFS_FTPS_TEST_CONCURRENCY=10 \
  /usr/bin/time -l "$test_binary" --ignored --exact \
  ftps::driver::upload::tests::ftps_large_upload_memory_probe --nocapture

On Linux use /usr/bin/time -v (its maximum RSS is reported in KiB). The default probe uses 64 MiB per upload and 10 uploads. These environment variables are test-only, not server configuration.

Observed on macOS arm64 with Rust 1.98.1, debug information disabled, one run per case (maximum resident set size reported in bytes):

Driver	Bytes per upload	Concurrent uploads	Maximum RSS
Original driver at 9c30cc8, same probe	67,108,864	10	710,033,408 (677.1 MiB)
Bounded multipart driver	67,108,864	10	179,781,632 (171.5 MiB)
Bounded multipart driver	10,737,418,240	10	187,465,728 (178.8 MiB)

For the original-driver comparison, only ftps/driver.rs was replaced with its base-commit version in the test checkout; the identical probe and dummy backend were retained, the executable rebuilt, and the 64 MiB case rerun. The original driver was not subjected to the 100 GiB aggregate case.

These measurements isolate protocol-driver buffering, not total server RSS, persisted-data correctness, TLS behavior, or storage throughput. The large case supplies 100 GiB of generated bytes without retaining them. A real server also uses memory for TLS, storage, caches, and allocator overhead. Memory still scales with active upload count. See FTPS operations for limits and cleanup behavior.