Object storage
S3 multipart upload: parts, retries, parallelism and cleanup
S3 multipart upload divides a large object into numbered parts that can be transferred independently and then completed as one object. It can improve throughput and makes individual retries smaller, but unfinished uploads also need deliberate cleanup.
An S3 multipart upload has a lifecycle
The client creates a multipart upload, uploads numbered parts, records the returned part identifiers and finally sends a completion request. Until completion, those parts are not the final object the application expects to read.
If the client crashes, the upload can remain incomplete. A production workflow should be able to list and abort abandoned multipart uploads so unfinished parts do not consume storage indefinitely.
Parallel parts can improve transfer speed
Sending several parts concurrently can use available bandwidth more effectively, especially over higher-latency links. Too much parallelism can exhaust memory, file descriptors or network capacity on the client.
Tune concurrency with representative object sizes and networks. A setting that helps a datacentre server can overwhelm a small office connection.
Part retries are smaller than restarting the whole object
When one part fails, a well-behaved client retransmits only that part, so a 100 GB object does not restart from byte zero. This is one reason backup and transfer tools often use multipart operations for large archives.
Retry logic should still have limits and visible errors. Infinite retries can hide a broken credential, endpoint or quota condition for hours.
Capacity planning must include in-progress parts
Provider accounting may count uploaded multipart parts before completion. A client near its hard capacity limit can therefore fail while assembling a large object even if the final object would barely fit.
Leave operational headroom and monitor usage reported by the storage platform. Treat quota failures as a capacity event and resolve the capacity problem before retrying writes.
Test completion and abandoned-upload cleanup
A small SDK upload can use one PUT and never exercise multipart APIs. Force a multipart test object, verify the completed object, start a second upload, abort it and confirm its unfinished parts are removed.
Those tests exercise the operations large transfers depend on, including failure cleanup.