Seamless Object Storage Migration from Amazon S3 to Google Cloud Storage

Overview

When moving workloads from AWS to GCP, one of the most critical steps is migrating object storage. Many organizations use Amazon S3 (Simple Storage Service) as their data lake or primary object storage, but when shifting to Google Cloud, the natural destination is Google Cloud Storage (GCS).

While you can use tools like gsutil for smaller migrations, Google Cloud’s Storage Transfer Service (STS) is a fully managed option designed for large-scale, reliable, and repeatable transfers. In this article, we’ll cover what Storage Transfer Service is, why it’s ideal for Amazon S3 → GCS migrations, and how to set it up step by step.

Storage Transfer Service

Storage Transfer Service is a managed data transfer tool in GCP that:

• Moves data from Amazon S3, Azure Blob, or on-premises storage into GCS.
• Supports one-time transfers or recurring syncs.
• Handles object-level integrity checks, retries, and parallelization automatically.
• Integrates directly with Amazon S3 using your AWS access keys (no need for local tools).

This makes it ideal for migrations where scalability, automation, and reliability are key.

Benefits of Using Storage Transfer Service

• Scalable – Handles terabytes to petabytes without manual scripting.
• Automated – Can be scheduled for recurring syncs (daily, hourly, etc.).
• Resilient – Automatic retries and integrity validation.
• Secure – Supports encryption, IAM-controlled access, and AWS keys with least-privilege.
• Cost-Effective – Reduces operational overhead compared to custom pipelines.

Prerequisites

Before starting the migration, ensure you have:

1. Google Cloud Project with billing enabled.
2. GCS Bucket created as the destination.
3. Amazon S3 Bucket with the data you want to migrate.
4. AWS IAM Access Key & Secret Key with permissions:
   • s3:GetObject
   • s3:ListBucket

Step-by-Step Migration with Storage Transfer Service

Step 1: Open Storage Transfer Service

• In the GCP Console, navigate to Storage → Transfer Service.
• Click Create transfer job.

Step 2: Configure Source (Amazon S3)

• Select the Amazon S3 bucket as the source.
• Provide:
  • Bucket name (e.g., my-source-bucket).
  • AWS Access Key ID & Secret Access Key.
• Grant permissions for listing and reading objects from S3.

Step 3: Configure Destination (GCS)

• Select the GCS bucket you created earlier.
• Example: gs://my-gcs-bucket.

Step 4: Choose Transfer Options

• Object overwrite options:
  • Overwrite when different (default).
  • Delete objects in the destination that are not in the source (for true mirroring).
• File filters: Migrate only specific prefixes or exclude certain files.
• Scheduling:
  • Run once (for one-time migration).
  • Recurring schedule (hourly/daily/weekly sync).

Step 5: Run & Monitor Transfer

• Review settings and click Create.
• Monitor transfer progress in the Transfer jobs dashboard.
• Logs and metrics are available for auditing in Cloud Monitoring.

Example: One-Time Migration

Here’s an example setup for moving an entire bucket from Amazon S3 to GCS once:

• Source: s3://customer-data-bucket
• Destination: gs://customer-data-gcp
• Schedule: One-time transfer
• Overwrite policy: Overwrite only if the object differs

Once executed, Storage Transfer Service will handle chunking, retries, and parallelization in the background.

Example: Ongoing Sync Migration

If your application is actively writing to Amazon S3 during migration, you may prefer a two-phase cutover:

1. Initial Bulk Migration – Run a one-time job to move historical data.
2. Incremental Sync – Configure a recurring transfer (e.g., every 6 hours) until applications are switched to GCS.

This reduces downtime and ensures no data is lost.

Best Practices

• AWS IAM Separation: Create a dedicated AWS IAM user for STS with read-only S3 permissions.
• Testing First: Start with a test job on a subset of data before migrating everything.
• Storage Classes: Choose the right GCS class (Standard, Nearline, Coldline, Archive) based on access frequency.
• Logging: Enable Cloud Audit Logs for full visibility into transfer activity.
• Cutover Plan: Ensure applications are updated to point to GCS once data is migrated.

Conclusion

Migrating data from Amazon S3 to GCS doesn’t have to be complex. With Google Cloud Storage Transfer Service, you get a secure, scalable, and automated migration path without writing custom scripts.

Drop a query if you have any questions regarding Amazon S3 or GCS and we will get back to you quickly.

About CloudThat