Object storage
Large file uploads to S3-compatible storage: design for interruption
Large objects expose weaknesses that tiny S3 tests never encounter. A successful 1 MB upload does not prove that a 200 GB archive can survive a WAN interruption, stay within quota or complete without exhausting client resources. Test interruption and restart behavior before trusting a large-file workflow with a production backup window or customer upload.
Use multipart for very large objects
Multipart upload lets the client retry a failed piece without retransmitting the entire object. It can also send parts concurrently to use available bandwidth more effectively.
Choose part size with provider limits and client memory in mind. Extremely small parts create excessive request overhead; extremely large parts reduce the benefit of granular retries.
Bound concurrency
Parallel uploads can saturate CPU, memory, source disk or network before they reach the storage service. Start conservatively and raise concurrency while monitoring end-to-end throughput and errors.
A backup server sharing bandwidth with production traffic may need a rate limit even when the object endpoint itself can accept more.
Leave quota headroom
In-progress parts can consume backend capacity. Starting a large upload with only a few megabytes below the hard quota is likely to fail or leave cleanup work.
Alert customers before capacity reaches the hard stop and make package upgrades idempotent so a quota change does not alter credentials or bucket identity.
Verify the completed bytes
ETag semantics can differ for multipart objects, so do not blindly treat every ETag as a simple MD5 checksum. Use application checksums or download verification appropriate to the client.
For backup archives, a restore or archive integrity test is stronger evidence than an upload-complete message.
Clean abandoned uploads
Interrupted clients can leave multipart uploads that were never completed. Inventory and abort old abandoned sessions based on conservative age rules.
Cleanup should identify exact upload IDs and bucket ownership so a maintenance job cannot abort another active customer transfer.