At CryoCloud, we build a cloud platform for processing and managing scientific imaging data. When CryoCloud first launched, all tenant data went into a single shared S3 bucket. Tenant separation was purely logical, using prefixes such as resources/{id}/. As the platform matured, we introduced per-tenant buckets for new tenants: proper isolation, tenant-specific lifecycle rules, simpler access policies. But every tenant created before that change stayed on the shared bucket, quietly accumulating data: petabytes of it by the time we looked seriously at moving it, terabytes per tenant. It was the kind of tech debt that’s easy to defer because “it still works.”

Eventually the split layout started getting in the way, and we decided to migrate. We needed zero downtime, rollback capability, and concrete proof that not a single file had been lost or mixed with another tenant’s data.

The plan

The migration broke down into eight steps:

1. Discovery - Collect a list of all legacy tenants and a complete inventory of their resources and data.
2. Bucket creation - Create new per-tenant buckets with configuration identical to what the application creates for new tenants.
3. Manifest generation - List every object per tenant in the legacy bucket and generate CSV manifests for S3 Batch Operations.
4. Retrieve archived data - CryoCloud lets users archive data in S3 Glacier Deep Archive, reducing storage costs by 23×. Restore those objects before moving them, then re-archive them after the migration.
5. Bulk copy - Submit one S3 Batch Operations S3PutObjectCopy job per tenant, running in parallel.
6. Delta sync - aws s3 sync to catch anything that changed between manifest generation and the copy.
7. Database migration - Atomic UPDATE statements switching bucket column references from the legacy bucket to the new per-tenant bucket. Rollback SQL generated for every tenant.
8. Cleanup - Once everything is validated, tag legacy data for deletion with a 30-day retention window and let a lifecycle rule handle the rest.

We validated every step on a staging environment with a much smaller dataset before touching production. That staging run surfaced almost every issue we’d later hit in production.

Discovery and bucket creation

The easy part: a few SQL queries produced a JSON manifest of all tenants with their resource counts, which we fed to a bucket creation script.

The bucket creation script had to exactly match what the application creates for new tenants. We verified this first by diffing the migration script’s bucket configuration against the application code, and later by comparing the configuration of every new bucket: lifecycle rules, versioning, tags, CORS, and other options. The only difference: the migration added a migrated_from_legacy: true tag for tracking.

All buckets created, 0 failures. On to the hard part.

Manifest generation and bulk copy

We chose S3 Batch Operations over aws s3 sync because it handles the scale and gives clear progress tracking. With millions of objects, Batch Operations runs the copy on AWS infrastructure, so there is no need to keep a machine running for days. You provide a CSV manifest listing every object, and AWS handles the rest. The manifest had to be exact: each job should copy only one tenant’s files and leave every other tenant’s data untouched.

Manifest generation was straightforward: list every object under each tenant’s prefixes and write a CSV with bucket,key per line. Upload the CSV to S3, submit the batch job.

S3 Batch Operations can also generate the manifest for you: you give it a source bucket and filters (prefix, size, storage class), and it builds the object list itself. This has long been available through the CLI and API, and arrived in the S3 console in September 2025. We needed per-tenant filtering that went beyond simple prefix matching, so we generated our own manifests, but for simpler migrations it eliminates this entire step and sidesteps the URL-encoding gotcha below.

The batch jobs ran in parallel, one per tenant. Most completed with 100% success. A few had failures, which brings us to the interesting part.

The gotchas

1. S3 Batch Operations CSV manifests must be URL-encoded

This was the most surprising discovery. We had a couple of tenants whose batch copy didn’t fully succeed. The failure rates were in the single digits, and every one reported NoSuchKey (404). The objects definitely existed. No operations had run between manifest generation and job execution.

After checking lifecycle rules and verifying the “missing” objects still existed via head-object, we combed through the job completion reports and found the pattern: every single failed object had a + character in its key. Every successful object did not.

It turns out S3 Batch Operations requires CSV manifest contents to be URL-encoded. Our manifest generation wrote raw keys such as X+0Y+0, but Batch Operations URL-decoded them on read, turning + into a space: X 0Y 0. The object doesn’t exist at the decoded path, hence NoSuchKey.

This affected tens of thousands of objects across production. Staging hadn’t caught it for the most mundane reason possible: no staging key happened to contain a +. The fix would have been to URL-encode keys when writing the CSV (X%2B0Y%2B0), but we discovered this after the fact. The workaround: aws s3 sync doesn’t have this issue, so our delta sync step caught all of these.

Lesson: When generating CSV manifests for S3 Batch Operations, URL-encode your keys. This is easy to miss because the list_objects_v2 API returns raw keys, and it’s natural to write them directly to CSV. Any key containing +, %, spaces, or other URL-special characters will be silently mangled otherwise. Always request a completion report when creating jobs. Reports are opt-in, and without the list of failed keys we would have been diagnosing blind.

2. The 5 GB single-part copy limit

S3 Batch Operations uses single-part copy, which has a 5 GB size limit. Any object larger than 5 GB fails with InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120.

We had a couple dozen objects across production hitting this. Again, the delta sync step handled them because aws s3 sync uses multipart upload automatically.

Lesson: S3 Batch Copy is not a complete solution for large objects. If you have many files over 5 GB, you can invoke a custom Lambda function via Batch Operations to perform multipart copies. It’s more complex to set up but worth it at scale. For a small number of oversized objects, a simple aws s3 sync follow-up is sufficient.

3. GlacierJobTier casing: BULK not Bulk

When submitting a batch restore job for Glacier Deep Archive objects, the GlacierJobTier parameter must be uppercase: BULK, not Bulk. The regular S3 restore_object API accepts title case, but Batch Operations doesn’t. You get an InvalidRequest error with no further explanation.

This cost us a failed job and a resubmission on staging. Small thing, poorly documented.

4. The IAM permission chain for Batch Operations

S3 Batch Operations requires a surprisingly deep permission chain:

• The user submitting the job needs: s3:CreateJob, s3:DescribeJob, s3:ListJobs, s3:UpdateJobStatus, s3:PutJobTagging, s3:GetJobTagging, and iam:PassRole for the batch operations role.
• The batch operations role needs: s3:GetObject and s3:ListBucket on the source bucket, s3:PutObject on the destination bucket, s3:GetObject on the bucket holding the manifest, s3:PutObject on the bucket receiving the completion report, and s3:RestoreObject on the source bucket when restoring objects.

We hit AccessDenied three separate times on staging, each time for a different missing permission. The errors only say “Access Denied,” not which permission is missing.

Lesson: Nobody said least-privilege permissions are easy.

The Glacier problem

Data in Glacier Deep Archive can’t be copied directly. You have to restore it first, wait for the restore to complete, then copy. This added a pause of up to 48 hours in the middle of the migration.

The sequence:

1. Generate glacier manifests - list all Glacier/Deep Archive objects per tenant
2. Submit batch restore - S3 Batch Operations S3InitiateRestoreObject with the BULK tier (cheapest, 12–48 hours for Deep Archive)
3. Wait - poll restore status via head-object, checking the Restore header
4. Batch copy - same as the bulk copy step, but using the glacier manifests
5. Sync >5 GB - catch any glacier objects that were too large for batch copy
6. Re-archive - return the copied data to Deep Archive

Restored copies are temporary. They stay around for however many days you set in the restore request (we used 7), and if you miss that window, you’re back to restoring and waiting out another full cycle. We submitted the copy jobs immediately after confirming restoration was complete.

What it cost

Two copies of everything for the length of the migration. That one we saw coming.

The rest of the bill was less intuitive. Most of the legacy data had aged into colder storage classes through lifecycle rules, which is fine until the day you need to read all of it back. We paid per gigabyte to pull it out of storage we had deliberately arranged never to touch, then paid again on the way back in: a copy lands in the destination bucket’s default storage class, not the one the source object was sitting in, so every re-archived object is a fresh transition, billed per object.

Requests were where it got interesting. Batch Operations bills per job and per object operation. Every copy is also a PUT on the destination. Manifest generation LISTs the entire source, and each delta sync LISTs it again. Individually they’re fractions of a cent. Times millions of objects, they outran the per-gigabyte charges.

One thing we never paid for: cross-region traffic. Every bucket involved, source and destination, sits in the same region, so none of this copying moved data between regions. If yours does, price that line first, because it can dwarf everything above it.

The one to watch is minimum billable durations. Deep Archive bills every object for at least 180 days, so deleting a legacy copy sooner than that saves you nothing: you still owe the remainder of the term. Worth doing that arithmetic before you settle on a retention window.

Database migration

The actual switchover was anticlimactic by design. For each tenant, a single transaction updates the bucket column across all tables referencing the storage bucket. Every UPDATE filters on both the tenant UUID and the old bucket name, making it idempotent.

Each switchover was bracketed by delta syncs: one immediately before the database update to carry over anything recently written to the legacy bucket, and one right after as a final sweep. Writes made after the update land directly in the new bucket, so nothing falls through the gap.

We rolled this out incrementally:

1. Smallest tenant first - verify in the app
2. A medium-sized tenant - verify in the app
3. All remaining tenants at once

Each step generated both forward and rollback SQL. The rollback is a mirror of the forward migration: swap the bucket names back. Since the data still exists in both locations, rollback would be safe and instant.

The result

In production, every legacy tenant moved, active and archived data alike, with zero downtime. The whole process took a few days, dominated by the Glacier restore wait. Every batch copy failure was caught by the delta sync step, and we never needed a single rollback.

Takeaways

• Run a small test batch first. S3 Batch Operations has sharp edges: URL-encoded manifests, the 5 GB copy limit, and case-sensitive parameters. None of them are well documented. Find them on a hundred objects, not on your whole dataset.
• Model the cost before you run it. The migration bill is a different shape from the storage bill: per-gigabyte retrievals, per-object re-transitions, per-job and per-object Batch Operations charges, and a request for every copy and every LIST. Work those out against your real object count and total size before you submit anything. At millions of objects the request lines dominate, and minimum billable durations can mean deleting the old copy early saves you nothing.
• Plan for a delta sync step. No matter how good your bulk copy is, you need a follow-up sync to catch edge cases and anything that changed mid-migration.
• Staging is only as good as its data. The one bug that got through did so because no staging key happened to contain a +. Make sure your staging data is weird in the same ways production data is.
• Generate rollback artifacts. Every destructive step produced rollback SQL or had a 30-day retention window. We never needed them, but knowing they existed made it much easier to proceed confidently.
• Start Glacier restores early. Bulk restores can take up to 48 hours, and that wait dominates the timeline. Restores are non-destructive, so kick them off first and let everything else run in parallel.
• Tag before you delete. Tag source objects as migrated instead of deleting them, then scan the legacy bucket for anything still untagged. An empty result is your proof that nothing was missed. Only then let a lifecycle rule clean up the tagged objects after a retention window.

– Ilja