Amazon S3 Bucket Cross-Region and Cross-Account Replication Using Terraform

Overview

Ensuring disaster recovery and data redundancy is essential in today’s digital environment. Through object replication between various AWS regions, Amazon S3 cross-region replication improves data availability and durability. Managing this process across several AWS accounts can be difficult but Terraform makes it easier by automating resource provisioning and maintenance.
This blog post explains how Terraform can be used to build up Amazon S3 cross-region and cross-account replication. We will walk you through setting up replication rules, configuring AWS IAM roles for cross-account access, and generating Amazon S3 buckets in several locations. This blog will cover how to use Terraform to achieve Amazon S3 replication.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Introduction

The robust feature of Amazon S3’s cross-region replication lets you automatically duplicate Amazon S3 objects across various AWS regions. This is especially helpful for guaranteeing disaster recovery, fulfilling compliance requirements, and enhancing data availability.

You may safely duplicate data between Amazon S3 buckets in various AWS accounts when you combine this with cross-account replication, increasing the security and flexibility of your cloud infrastructure.

Step-by Step Guide

Cross-account role: First, we need a cross-account role allowing our source account to modify and create objects and buckets in the destination account. For that, we will run terraform in our destination account, creating a role for our source account. You must set up a provider to set up the region in the destination account.

provider "aws" {

region = "us-east-1"

}

Now we will create a role.

resource "aws_iam_role" "destination_assume_role" {
  name = "destination-account-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          AWS = "111111111111"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy" "destination_policy" {
  name = "destination_policy"
  role = aws_iam_role.destination_assume_role.id
  policy = jsonencode(
    {
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject",
          "s3:ListBucket",
          "s3:GetBucketLocation",
          "s3:ListAllMyBuckets",
          "kms:CreateKey",
          "s3:CreateBucket",
          "S3:PutBucketVersioning",
          "S3:GetBucketPolicy",
          "S3:GetBucketAcl",
          "s3:*"
        ],
        Resource = [
          "*"
        ]
      }
    ]
  })
}

resource "aws_iam_role" "destination_assume_role" {

name = "destination-account-role"

assume_role_policy = jsonencode({

Version = "2012-10-17"

Statement = [

{

Action = "sts:AssumeRole"

Effect = "Allow"

Principal = {

AWS = "111111111111"

}

]

})

}

resource "aws_iam_role_policy" "destination_policy" {

name = "destination_policy"

role = aws_iam_role.destination_assume_role.id

policy = jsonencode(

{

Version = "2012-10-17",

Statement = [

{

Effect = "Allow",

Action = [

"s3:GetObject",

"s3:PutObject",

"s3:DeleteObject",

"s3:ListBucket",

"s3:GetBucketLocation",

"s3:ListAllMyBuckets",

"kms:CreateKey",

"s3:CreateBucket",

"S3:PutBucketVersioning",

"S3:GetBucketPolicy",

"S3:GetBucketAcl",

"s3:*"

Resource = [

"*"

]

}

]

})

}

You can do “terraform apply” here to create this role in the destination account. Make sure to replace 11111111 with your source account ID and run this terraform script using destination account credentials.

Now, we will start the process of replication in our source account and destination account

2. Provider conf.

provider "aws" {

alias = "destination"

region = "us-west-2"

assume_role {

role_arn = aws_iam_role.destination_assume_role.arn

}

data "aws_caller_identity" "source" {}

data "aws_caller_identity" "destination" {

provider = aws.destination

}

3. AWS KMS Keys: We need some keys for both accounts. Our destination key will be a bit special because there will be a policy that will allow the source account to access it.

resource "aws_kms_key" "source" {

deletion_window_in_days = 30

enable_key_rotation = true

}

resource "aws_kms_key" "destination" {

provider = aws.destination

deletion_window_in_days = 30

enable_key_rotation = true

policy = data.aws_iam_policy_document.destination_kms_key.json

}

data "aws_iam_policy_document" "destination_kms_key" {

statement {

principals {

type = "AWS"

identifiers = [

data.aws_caller_identity.source.account_id

]

}

actions = [

"kms:Encrypt"

]

resources = ["*"]

}

statement {

principals {

type = "AWS"

identifiers = [

data.aws_caller_identity.destination.account_id

]

}

actions = [

"kms:*"

]

resources = ["*"]

}

4. Amazon S3 buckets: It is time to create buckets with new keys.

resource "aws_s3_bucket" "source" {
  bucket = "replication-test-source-buckettt"
  acl    = "private"

versioning {
    enabled = true
  }

server_side_encryption_configuration {
  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.source.arn
      sse_algorithm     = "aws:kms"
    }
  }
}

replication_configuration {
  role = aws_iam_role.replication.arn

rules {
    id     = "replicate"
    status = "Enabled"

source_selection_criteria {
      sse_kms_encrypted_objects {
        enabled = true
      }
    }

destination {
      account_id         = data.aws_caller_identity.destination.account_id
      bucket             = aws_s3_bucket.destination.arn
      storage_class      = "STANDARD_IA"
      replica_kms_key_id = aws_kms_key.destination.arn
    }
  }
}
}

resource "aws_s3_bucket" "destination" {
  provider = aws.destination

bucket = "replication-test-destination-buckettt"
  acl    = "private"

versioning {
    enabled = true
  }

server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = aws_kms_key.destination.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }
}

resource "aws_s3_bucket" "source" {

bucket = "replication-test-source-buckettt"

acl = "private"

versioning {

enabled = true

}

server_side_encryption_configuration {

rule {

apply_server_side_encryption_by_default {

kms_master_key_id = aws_kms_key.source.arn

sse_algorithm = "aws:kms"

}

replication_configuration {

role = aws_iam_role.replication.arn

rules {

id = "replicate"

status = "Enabled"

source_selection_criteria {

sse_kms_encrypted_objects {

enabled = true

}

destination {

account_id = data.aws_caller_identity.destination.account_id

bucket = aws_s3_bucket.destination.arn

storage_class = "STANDARD_IA"

replica_kms_key_id = aws_kms_key.destination.arn

}

resource "aws_s3_bucket" "destination" {

provider = aws.destination

bucket = "replication-test-destination-buckettt"

acl = "private"

versioning {

enabled = true

}

server_side_encryption_configuration {

rule {

apply_server_side_encryption_by_default {

kms_master_key_id = aws_kms_key.destination.arn

sse_algorithm = "aws:kms"

}

5. AWS IAM Roles: An AWS IAM role in our source account is required so that Amazon S3 can access the destination bucket.

resource "aws_iam_role" "replication" {
  name               = "replication-test-role"
  assume_role_policy = data.aws_iam_policy_document.replication_role.json
}

resource "aws_iam_role_policy" "replication" {
  name   = "replication"
  role   = aws_iam_role.replication.id
  policy = data.aws_iam_policy_document.replication_policy.json
}

data "aws_iam_policy_document" "replication_role" {
  statement {
    principals {
      type        = "Service"
      identifiers = ["s3.amazonaws.com"]
    }

actions = [
      "sts:AssumeRole"
    ]
  }
}

data "aws_iam_policy_document" "replication_policy" {
  statement {
    actions = [
      "s3:GetReplicationConfiguration",
      "s3:ListBucket"
    ]

resources = [
      aws_s3_bucket.source.arn
    ]
  }

statement {
    actions = [
      "s3:GetObjectVersionForReplication",
      "s3:GetObjectVersionAcl",
      "s3:GetObjectVersionTagging"
    ]

resources = [
      "${aws_s3_bucket.source.arn}/*"
    ]
  }

statement {
    actions = [
      "s3:ReplicateObject",
      "s3:ReplicateDelete",
      "s3:ReplicateTags"
    ]

resources = [
      "${aws_s3_bucket.destination.arn}/*"
    ]
  }

statement {
    actions = [
      "kms:Decrypt"
    ]

resources = [
      aws_kms_key.source.arn
    ]
  }

statement {
    actions = [
      "kms:Encrypt"
    ]

resources = [
      aws_kms_key.destination.arn
    ]
  }
}

resource "aws_iam_role" "replication" {

name = "replication-test-role"

assume_role_policy = data.aws_iam_policy_document.replication_role.json

}

resource "aws_iam_role_policy" "replication" {

name = "replication"

role = aws_iam_role.replication.id

policy = data.aws_iam_policy_document.replication_policy.json

}

data "aws_iam_policy_document" "replication_role" {

statement {

principals {

type = "Service"

identifiers = ["s3.amazonaws.com"]

}

actions = [

"sts:AssumeRole"

]

}

data "aws_iam_policy_document" "replication_policy" {

statement {

actions = [

"s3:GetReplicationConfiguration",

"s3:ListBucket"

]

resources = [

aws_s3_bucket.source.arn

]

}

statement {

actions = [

"s3:GetObjectVersionForReplication",

"s3:GetObjectVersionAcl",

"s3:GetObjectVersionTagging"

]

resources = [

"${aws_s3_bucket.source.arn}/*"

]

}

statement {

actions = [

"s3:ReplicateObject",

"s3:ReplicateDelete",

"s3:ReplicateTags"

]

resources = [

"${aws_s3_bucket.destination.arn}/*"

]

}

statement {

actions = [

"kms:Decrypt"

]

resources = [

aws_kms_key.source.arn

]

}

statement {

actions = [

"kms:Encrypt"

]

resources = [

aws_kms_key.destination.arn

]

}

6. Destination bucket policy: We need to allow our new AWS IAM role to be replicated in our destination bucket.

resource "aws_s3_bucket_policy" "destination" {

provider = aws.destination

bucket = aws_s3_bucket.destination.id

policy = data.aws_iam_policy_document.destination_bucket_policy.json

}

data "aws_iam_policy_document" "destination_bucket_policy" {

statement {

principals {

type = "AWS"

identifiers = [

aws_iam_role.replication.arn

]

}

actions = [

"s3:ReplicateDelete",

"s3:ReplicateObject"

]

resources = [

"${aws_s3_bucket.destination.arn}/*"

]

}

statement {

principals {

type = "AWS"

identifiers = [

aws_iam_role.replication.arn

]

}

actions = [

"s3:List*",

"s3:GetBucketVersioning",

"s3:PutBucketVersioning"

]

resources = [

aws_s3_bucket.destination.arn

]

}

We are good to go now, and you can do the final terraform apply to set up your cross-region and cross-account replication. After applying, you can test this setup by putting objects into your source bucket, and they will automatically get replicated to your destination bucket.

Conclusion

Using Terraform to set up Amazon S3 cross-region, cross-account replication improves compliance, disaster recovery, and data redundancy. You have now successfully created Amazon S3 buckets, configured AWS IAM roles, and defined replication rules using Terraform with this article.

You may improve availability and durability by automating this process and ensuring consistent data replication across AWS regions and accounts. Maintain a close eye on your replication status and expenses, and for best results, follow security best practices.

Drop a query if you have any questions regarding Amazon S3 cross-region, cross-account and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

Reduced infrastructure costs
Timely data-driven decisions

Get Started

About CloudThat

CloudThat is an award-winning company and the first in India to offer cloud training and consulting services worldwide. As a Microsoft Solutions Partner, AWS Advanced Tier Training Partner, and Google Cloud Platform Partner, CloudThat has empowered over 850,000 professionals through 600+ cloud certifications winning global recognition for its training excellence including 20 MCT Trainers in Microsoft’s Global Top 100 and an impressive 12 awards in the last 8 years. CloudThat specializes in Cloud Migration, Data Platforms, DevOps, IoT, and cutting-edge technologies like Gen AI & AI/ML. It has delivered over 500 consulting projects for 250+ organizations in 30+ countries as it continues to empower professionals and enterprises to thrive in the digital-first world.

FAQs

1. Can I replicate objects between Amazon S3 buckets in different AWS accounts?

ANS: – Yes, cross-account replication allows you to duplicate objects between Amazon S3 buckets in several AWS accounts. Enabling rights for replication across accounts entails configuring IAM roles and rules. Terraform streamlines this procedure by automating the development and setup of AWS IAM roles and policies.

2. How does Amazon S3 cross-region replication impact data transfer costs?

ANS: – Data transport expenses may be incurred when sending data between AWS regions for Amazon S3 cross-region replication. These expenses vary depending on the AWS regions and volume of data copied, among other things. When planning your replication arrangement, you must consider these costs and keep a close eye on them to save on expenditures.

3. Can I use encryption with Amazon S3 cross-region replication?

ANS: – Yes, you can use encryption to protect your replicated data while using Amazon S3 cross-region replication. Both client-side encryption, which encrypts data before uploading it to Amazon S3, and server-side encryption (SSE) are supported by Amazon S3. To ensure that duplicated objects are encrypted per your security standards, you can provide encryption settings while configuring replication rules with Terraform.

WRITTEN BY Shakti Singh Chouhan

Shakti Singh is a Cloud Engineer with over 3.5 years of experience in designing, deploying, and securing scalable AWS infrastructures. A DevOps enthusiast, he is passionate about automation, security, and cloud migration. Shakti enjoys sharing insights on cloud technologies, problem-solving, and fostering a culture of continuous learning.