Migrating 100K+ Files from Salesforce to AWS S3 Without Losing One

Business Problem

The company's Salesforce org had become its accidental document archive. Years of attachments, product files, and case documentation had pushed file storage far past the included allocation, and Salesforce file storage is one of the most expensive places on the internet to keep a PDF. The finance team saw a growing line item; the platform team saw an org slowing under the weight. The files needed to move to AWS S3 — but they were live business records, referenced by users daily, and nobody was willing to accept "some files may not survive the move."

Technical Challenge

Moving 100K+ files out of a governed SaaS platform is nothing like copying a folder:

• Governor limits everywhere. Salesforce caps callout counts, callout sizes, heap usage, and execution time per transaction. A naive loop dies within minutes; the pipeline had to be batch-architected from the start.
• Failure is guaranteed at this volume. Network blips, oversized files, and API hiccups are statistical certainties across 100K+ transfers. The design question was never "will transfers fail?" but "how do we know exactly which ones, and how do they recover?"
• Zero-loss verification. A file that silently failed to arrive in S3 but was cleaned up in Salesforce would be unrecoverable. Every deletion had to be gated on verified arrival.

Solution Architecture

We built the migration as a resumable, self-auditing pipeline rather than a one-shot script:

• Batch Apex pipeline sized against callout and heap limits, streaming file bodies from Salesforce to S3 through authenticated REST callouts, with chunk sizing tuned to stay comfortably inside platform ceilings at maximum throughput.
• Named Credentials + AWS signature authentication, keeping secrets out of code and rotating cleanly — no access keys buried in Apex.
• Per-file error ledger: every transfer writes a status record — success, failure reason, HTTP response, retry count. Failed files are automatically re-queued with backoff; permanently failed files surface in a dashboard a human can act on. Nothing fails silently.
• Verify-then-release flow: Salesforce-side cleanup runs only after S3 arrival is confirmed, making data loss structurally impossible rather than statistically unlikely.
• Configurable via Custom Metadata — bucket targets, batch sizes, and retry policies adjustable without deployments, so operations could tune the run in flight.
• Apex best-practice hardening across the org's existing triggers and classes as part of the engagement, ensuring the new load didn't destabilize existing automation.

Business Impact

Over 100,000 files moved to AWS S3 with zero loss — every file either verified in S3 or accounted for on the error ledger with a reason and a retry path. Salesforce file storage dropped by roughly 90%, converting an escalating platform cost into commodity object storage priced in cents. The pipeline remains in place as ongoing archival infrastructure: new files age out to S3 automatically, so the storage problem is solved permanently, not deferred.