AWS: S3 Cross-Region Replication with DeleteMarkers set up

By | 07/17/2019
0 Comment

At this moment I’m configuring a new CDN for our project.

Will use CloudFront and Cloudflare here so need to create two dedicated buckets with different names – cdn.cfr.example.com => CloudFront and cdn.cfl.example.com => Cloudflare.

To avoid coping data each time to both buckets – an AWS S3 Cross-Region Replication can be used, so data from a bucket-1 will be copied to a bucket-2.

Documentation – Cross-Region Replication.

CRR prerequisites:

  1. both buckets must have S3 Versioning enabled (see. Using Versioning)
  2. both buckets must be created in different regions
  3. both buckets must have permissions for replication between them

CRR limitations:

  1. already existing files in a source-bucket will not be copied after CRR will be configured (i.e. only new files will be replicated)
  2. can not create a replication-chain, i.e. Bucket-A => Bucket-B => Bucket-C

AWS S3 Cross-Region Replication set up

Create two buckets:

For both enable Versioning:

In a source-bucket bttrm-crr-source go to the Management > Replication, click on the Add rule:

Set replicate all from this bucket:

Click Next, set the receiver-bucket name:

Next – permissions and IAM role.

Chose the Create new IAM role, set its name:

Save:

Replication testing

Upload a test file to the source bttrm-crr-source bucket:

[simterm]

$ touch testfile.txt
$ aws --profile bm-backend --region us-east-2 s3 cp testfile.txt s3://bttrm-crr-source
upload: ./testfile.txt to s3://bttrm-crr-source/testfile.txt

[/simterm]

Check if it is present:

[simterm]

$ aws --profile bm-backend --region us-east-2 s3 ls s3://bttrm-crr-source
2019-07-15 15:36:38          0 testfile.txt

[/simterm]

And check the destination bucket:

[simterm]

$ aws --profile bm-backend --region us-west-1 s3 ls s3://bttrm-crr-dest
2019-07-15 15:36:38          0 testfile.txt

[/simterm]

The file is present, all good.

Deleting files from S3 Cross-Region Replication and DeleteMarkers

if you’ll just delete a file from the current source-bucket – it will be deleted there, but not in the destination bucket.

Remove it in the source:

[simterm]

$ aws --profile bm-backend --region us-east-2 s3 rm s3://bttrm-crr-source/testfile.txt
delete: s3://bttrm-crr-source/testfile.txt

[/simterm]

Check it in the destination:

[simterm]

$ aws --profile bm-backend --region us-west-1 s3 ls s3://bttrm-crr-dest
2019-07-15 15:36:38          0 testfile.txt

[/simterm]

The file is still here.

To delete objects from under S3 Versioning – AWS adds a special marker called DeleteMarker (see. Working with Delete Markers):

[simterm]

$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-source
{
    "Versions": [
        {
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Key": "testfile.txt",
            "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
            "IsLatest": false,
            "LastModified": "2019-07-15T12:36:38.000Z",
            "Owner": {
                "DisplayName": "example.aws",
                "ID": "9365528c1a92cd01abba171eb7b95c352a330a40ceb28b420c9462fac5891bd2"
            }
        }
    ],
    "DeleteMarkers": [
        {
            "Owner": {
                "DisplayName": "example.aws",
                "ID": "9365528c1a92cd01abba171eb7b95c352a330a40ceb28b420c9462fac5891bd2"
            },
            "Key": "testfile.txt",
            "VersionId": "PIr6ZOPcdi9oFP8y0rMXkRhrFuKpDQ4P",
            "IsLatest": true,
            "LastModified": "2019-07-15T12:39:24.000Z"
        }
    ]
}

[/simterm]

But this marker will not be copied to the destination bucket, see the What Does Amazon S3 Replicate:

  • If you make a DELETE request without specifying an object version ID, Amazon S3 adds a delete marker. Amazon S3 deals with the delete marker as follows:
    • If you are using the latest version of the replication configuration, that is, you specify the Filter element in a replication configuration rule, Amazon S3 does not replicate the delete marker.
    • If you don’t specify the Filter element, Amazon S3 assumes that the replication configuration is a prior version V1. In the earlier version, Amazon S3 handled replication of delete markers differently. For more information, see Backward Compatibility .
  • If you specify an object version ID to delete in a DELETE request, Amazon S3 deletes that object version in the source bucket, but it doesn’t replicate the deletion in the destination bucket. In other words, it doesn’t delete the same object version from the destination bucket. This protects data from malicious deletions.

Check the file in the destination bucket:

[simterm]

$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-dest
{
    "Versions": [
        {
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            ...
            "Key": "testfile.txt",
            "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
            ...
        }
    ]
}

[/simterm]

No DeleteMarkers present.

Check the replication settings:

[simterm]

$ aws --profile bm-backend --region us-west-1 s3api get-bucket-replication --bucket bttrm-crr-source  --query 'ReplicationConfiguration.Rules[*].DeleteMarkerReplication' --output text
Disabled

[/simterm]

DeleteMarkerReplication – "Status": "Disabled", okay.

IAM s3:ReplicateDelete

At first – check the IAM role used (see the Setting Up Permissions for Cross-Region Replication).

Find its name:

And find this role in the IAM:

Check it – the "s3:ReplicateDelete" in the Actions must be present:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Get*",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::bttrm-crr-source",
                "arn:aws:s3:::bttrm-crr-source/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::bttrm-crr-dest/*"
        }
    ]
}

S3 Bucket Policy

Now – export existing S3 policy:

[simterm]

$ aws --profile bm-backend --region us-west-1 s3api get-bucket-replication --bucket bttrm-crr-source > bttrm-crr-source-origin.json

[/simterm]

Check its content:

{
    "ReplicationConfiguration": {
        "Role": "arn:aws:iam::534***385:role/service-role/s3crr_role_for_bttrm-crr-source_to_bttrm-crr-dest",
        "Rules": [
            {
                "ID": "bttrm-s3-crr-replica",
                "Priority": 1,
                "Filter": {},
                "Status": "Enabled",
                "Destination": {
                    "Bucket": "arn:aws:s3:::bttrm-crr-dest"
                },
                "DeleteMarkerReplication": {
                    "Status": "Disabled"
                }
            }
        ]
    }
}

Now remove ReplicationConfiguration, "Filter": {}, "Priority": 1,  DeleteMarkerReplication parameters and add "Prefix": "", so this policy will be like Version 1:

{
        "Role": "arn:aws:iam::534***385:role/service-role/s3crr_role_for_bttrm-crr-source_to_bttrm-crr-dest",
        "Rules": [
            {
                "ID": "bttrm-s3-crr-replica",
                "Prefix": "",
                "Status": "Enabled",
                "Destination": {
                    "Bucket": "arn:aws:s3:::bttrm-crr-dest"
                }
            }
        ]
}

The problem is that CRR config version 2 still can’t do DeleteMarkers replication at this moment, the AWS Support reply was:

I did have a look at your blog and the suggestions provided by you seem correct. In fact, this would have been my suggestion as well. The internal team is working on bringing feature of Delete replication in V2 as well but it may take some time for that to happen. It is possible that the way V2 has been implemented in back-end is little different from V1 and using this feature directly may have been causing issues. Unfortunately, I don’t have any ETA on when this would finish. Till then, I’m sure your blog would be able to help AWS Users 🙂

Apply this config:

[simterm]

$ aws --profile bm-backend --region us-west-1 s3api put-bucket-replication --bucket bttrm-crr-source --replication-configuration file://bttrm-crr-source.json

[/simterm]

Upload some new file to the source bucket:

[simterm]

$ aws --profile bm-backend --region us-east-2 s3 cp test.file s3://bttrm-crr-source

[/simterm]

Delete it:

[simterm]

$ aws --profile bm-backend --region us-east-2 s3 rm s3://bttrm-crr-source/test.file

[/simterm]

Check the source bucket now:

[simterm]

$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-source
{
    "Versions": [
        {
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            ...
            "Key": "test.file",
            "VersionId": "0mkfsrIRdkGCgL.zKgMyMXjo_UydigIt",
            ...
        },
        {
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            ...
            "Key": "testfile.txt",
            "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
            ...
        }
    ],
    "DeleteMarkers": [
        {
            ...
            "Key": "test.file",
            "VersionId": "1kpKJlgOM7xpFnH4qsEp0EzqhCE1HFMX",
            ...
        },
        {
            ...
            "Key": "testfile.txt",
            "VersionId": "PIr6ZOPcdi9oFP8y0rMXkRhrFuKpDQ4P",
            ...
        }
    ]
}

[/simterm]

Great – both files found, both have DeleteMarkers.

And check the destination bucket:

[simterm]

$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-dest
{
    "Versions": [
        {
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            ...
            "Key": "test.file",
            "VersionId": "0mkfsrIRdkGCgL.zKgMyMXjo_UydigIt",
            ...
        },
        {
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            ...
            "Key": "testfile.txt",
            "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
            ...
        }
    ],
    "DeleteMarkers": [
        {
            ...
            "Key": "test.file",
            "VersionId": "1kpKJlgOM7xpFnH4qsEp0EzqhCE1HFMX",
            ...
        }
    ]
}

[/simterm]

There is still no DeleteMarker for the testfile.txt – it’s OK, as it was deleted before we made changes in the DeleteMarkers behavior.

But for the test.file – DeleteMarker already present, it was replicated from the source, so this file is “deleted” now as well:

[simterm]

$ aws --paws --profile bm-backend --region us-west-1 s3 ls s3://bttrm-crr-dest
2019-07-15 15:36:38          0 testfile.txt

[/simterm]

No test.file found.

Done.