A Guide to Cross-Account Replication of Amazon S3 Objects

Overview

Cross-Account Replication in Amazon Simple Storage Service (S3) is a feature that allows users to replicate objects across different AWS accounts. This capability is valuable for various scenarios, including compliance, disaster recovery, and data aggregation across multiple accounts.

Introduction

Amazon S3 (Simple Storage Service) is a highly scalable and durable object storage service that Amazon Web Services (AWS) provides. Replicating Amazon S3 objects across AWS accounts is a common scenario for organizations that must share data securely between different environments. In this blog, we will explore the steps and best practices for replicating Amazon S3 objects cross-account in AWS.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Prerequisites

Before diving into the replication process, ensure you have the following prerequisites:

Two AWS accounts (Source and Destination).
Appropriate AWS IAM roles with the necessary permissions for cross-account access.
Amazon S3 buckets in both accounts.

Step-by-Step Guide

Step 1: Set Up Cross-Account AWS IAM Roles:

In the Source AWS account, create an AWS IAM role that grants permissions for reading Amazon S3 objects. In the Destination AWS account, create an AWS IAM role that grants permissions for writing Amazon S3 objects. Attach trust policies to allow cross-account access.

Trust relationship looks like the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Principal": {

"Service": "s3.amazonaws.com"

"Action": "sts:AssumeRole"

}

]

}

Permissions for AWS IAM Role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SourceBucketPermissions",
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectRetention",
        "s3:GetObjectVersionTagging",
        "s3:GetObjectVersionAcl",
        "s3:ListBucket",
        "s3:GetObjectVersionForReplication",
        "s3:GetObjectLegalHold",
        "s3:GetReplicationConfiguration"
      ],
      "Resource": [
        "arn:aws:s3:::SourceBucketName/*",
        "arn:aws:s3:::SourceBucketName"
      ]
    },
    {
      "Sid": "DestinationBucketPermissions",
      "Effect": "Allow",
      "Action": [
        "s3:ReplicateObject",
        "s3:ObjectOwnerOverrideToBucketOwner",
        "s3:GetObjectVersionTagging",
        "s3:ReplicateTags",
        "s3:ReplicateDelete"
      ],
      "Resource": [
        "arn:aws:s3:::DestBucketName/*"
      ]
    }
  ]
}

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "SourceBucketPermissions",

"Effect": "Allow",

"Action": [

"s3:GetObjectRetention",

"s3:GetObjectVersionTagging",

"s3:GetObjectVersionAcl",

"s3:ListBucket",

"s3:GetObjectVersionForReplication",

"s3:GetObjectLegalHold",

"s3:GetReplicationConfiguration"

"Resource": [

"arn:aws:s3:::SourceBucketName/*",

"arn:aws:s3:::SourceBucketName"

]

{

"Sid": "DestinationBucketPermissions",

"Effect": "Allow",

"Action": [

"s3:ReplicateObject",

"s3:ObjectOwnerOverrideToBucketOwner",

"s3:GetObjectVersionTagging",

"s3:ReplicateTags",

"s3:ReplicateDelete"

"Resource": [

"arn:aws:s3:::DestBucketName/*"

]

}

]

}

Step 2: Enable Versioning on Amazon S3 Buckets:

Enable versioning on both the source and destination Amazon S3 buckets. Versioning helps track changes and maintain a history of object modifications.

Step 3: Configure Cross-Account Replication:

Source Bucket Configuration:

Navigate to the Amazon S3 console in the Source account.
Select the source bucket and go to the “Management” tab.
Enable versioning if not already enabled.
Configure a replication rule specifying the destination bucket, AWS IAM role, and other replication settings.

2. Destination Bucket Configuration:

Navigate to the Amazon S3 console in the Destination account.
Configuring bucket policy, an example of bucket policy is given.

{
  "Version": "2012-10-17",
  "Id": "PolicyForDestinationBucket",
  "Statement": [
    {
      "Sid": "ReplicationPermissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"
      },
      "Action": [
        "s3:ReplicateDelete",
        "s3:ReplicateObject",
        "s3:ObjectOwnerOverrideToBucketOwner",
        "s3:GetBucketVersioning",
        "s3:PutBucketVersioning"
      ],
      "Resource": [
        "arn:aws:s3:::DestBucketName/*",
        "arn:aws:s3:::DestBucketName"
      ]
    }
  ]
}

{

"Version": "2012-10-17",

"Id": "PolicyForDestinationBucket",

"Statement": [

{

"Sid": "ReplicationPermissions",

"Effect": "Allow",

"Principal": {

"AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"

"Action": [

"s3:ReplicateDelete",

"s3:ReplicateObject",

"s3:ObjectOwnerOverrideToBucketOwner",

"s3:GetBucketVersioning",

"s3:PutBucketVersioning"

"Resource": [

"arn:aws:s3:::DestBucketName/*",

"arn:aws:s3:::DestBucketName"

]

}

]

}

Select the destination bucket and go to the “Management” tab.
Enable versioning if not already enabled.
Configure the destination bucket to accept cross-account replicated objects.

Step 4: Run Batch Operation Jobs:

Set Up Batch Operation Jobs:

Utilize the AWS Batch Operations feature to set up batch jobs for replicating Amazon S3 objects.
Create a job definition specifying the replication task, AWS IAM roles, and other relevant parameters.

Schedule Batch Jobs:

Define schedules for batch jobs based on your replication requirements.
Leverage Amazon CloudWatch Events for automated triggering of batch jobs.

2. Monitor Batch Operations:

Use the AWS Batch console to monitor the status and progress of batch jobs.
Implement Amazon CloudWatch Alarms for proactive monitoring and notifications.

3. Monitor Batch Operations:

Use the AWS Batch console to monitor the status and progress of batch jobs.
Implement Amazon CloudWatch Alarms for proactive monitoring and notifications.

Step 5: Verify and Monitor Replication:

Verification:

Upload a test object to the source bucket and ensure it gets replicated to the destination bucket.
Check version IDs to confirm the successful replication of versions.

2. Monitoring:

Utilize Amazon CloudWatch metrics and Amazon S3 access logs to monitor replication metrics.
Set up Amazon CloudWatch Alarms for notifications on replication failures or delays.

Best Practices

Fine-Grained Permissions: Apply the principle of least privilege by configuring AWS IAM roles with granular permissions for cross-account access.
Logging and Monitoring: Regularly monitor Amazon CloudWatch metrics and Amazon S3 access logs to ensure replication is functioning as expected.
Versioning: Enable versioning on source and destination buckets to track changes and ensure data integrity.
Error Handling: Implement proper error handling mechanisms and set up notifications for any replication failures.

Conclusion

Replicating Amazon S3 objects cross-account in AWS allows for secure and efficient data sharing between environments.

Following the outlined steps and best practices, organizations can establish a robust and controlled mechanism for replicating data across AWS accounts, ensuring data consistency and integrity.

Drop a query if you have any questions regarding Amazon S3 and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. What is the significance of enabling versioning on Amazon S3 buckets in cross-account replication?

ANS: – Versioning is crucial for tracking changes and maintaining a history of object modifications. It ensures data integrity and allows for accurate replication of different versions of objects.

2. Can I replicate Amazon S3 objects across AWS accounts without using batch operation jobs?

ANS: – Yes, replication can be achieved without batch operations, following the standard configuration of cross-account AWS IAM roles and replication rules. However, batch operations enhance automation and scalability in large-scale replication scenarios.