Backups
How to verify S3 backups by restoring them, not just listing them
A green backup job proves that the software completed its configured work. A restore test proves whether the stored objects, catalogues, credentials and application procedure can actually produce usable data when the source is unavailable.
Choose a recovery point without using the source
Pretend the production server is gone. Use the backup catalogue or object manifest to identify a recovery point from storage-side information with no dependency on files that exist only on the source.
If operators cannot find the correct backup without the failed system, the disaster runbook is incomplete.
Restore to an isolated destination
Download into a clean directory, disposable VM or test database and leave production untouched during validation. This allows destructive validation without risking the live workload.
Use a fresh credential when practical to prove the documented credential-recovery process.
Validate content, not just transfer size
Compare checksums for ordinary files, run archive integrity tests and start restored applications or databases far enough to verify meaningful state. The right validation depends on what the backup is supposed to protect.
A directory with the expected number of bytes can still contain a corrupt archive or an inconsistent database.
Record time and bottlenecks
Measure how long it takes to identify, retrieve and make the data usable. Storage throughput is only one component; catalog lookup, decryption and application rebuild can dominate recovery time.
Use the result to decide whether the recovery objective is realistic and whether a smaller local recovery cache or additional copy is justified.
Turn failures into runbook changes
Missing keys, undocumented endpoints, wrong regions and insufficient permissions are valuable findings during a drill because they can be corrected before a real incident.
Keep restore evidence with the backup design and rerun the drill after material changes to client software, retention or storage provider configuration. Record the restored file ownership, permissions and application or database version when those details affect whether the recovered data can actually be used.