Moving S3 bucket data to another bucket in real time - AWS Lambda?
Moving S3 bucket data to another bucket in real time - AWS Lambda?
We have a parent bucket, say "Bucket-1". Then under this Bucket-1, we have multiple folders, 1 for each customer (say cust1, cust2, cust3 and so on).
We upload Gb's of data into these folders for each customer. And this data is uploaded via multiple channels, via JS, via JAVA, via Swift IOS etc.
Requirement:
Now we've got a requirement from a specific customer to upload it's data(forthecoming) into some other dedicated bucket say "Bucket-2" (so that some special permissions can be provided to the customer for read access via another AWS account of that customer).
Solution(s) that I came up with:**
Code modifications in all the channels (JS, Java, Swift) to accommodate this change. But this solutions is a) time taking, b) tightly coupling an upload logic with a specific requirement.
Using AWS lambda to move data between buckets. This lambda will be triggered with any operation done to that specific client's bucket folder.
For now #2 seems the best fit to me. Any suggestions? Or any other solution that comes up to your mind?
Bucket-1
Double clicking on the requirement though, why do you need a separate bucket for special permissions, ACLs can be assigned per Object and IAM policies/users can be scoped to specific prefixes (folders) within a bucket.
– Cheruvian
Aug 20 at 18:52
Is the bucket (Bucket-2) in your account, or your customers?
– PrestonM
Aug 20 at 20:58
1 Answer
1
You can use Cross-Region Replication (CRR) - Amazon Simple Storage Service to automatically replicate an Amazon S3 bucket (or part of a bucket) to another bucket. This happens automatically but asynchronously.
The down-side is that it requires the other bucket to be in a different region (which is the primary use-case for CRR).
Alternatively, instead of copying the data to another bucket, you could use a Bucket Policy to grant the necessary access to the other account. This makes much more sense than duplicating read-only data purely to provide a different type of access. It is worth exploring this option before making a complicated option that would need on-going maintenance (and would cost more in storage costs).
By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
I agree with your second proposed solution as being the best. You should be able to trigger a Lambda function every time an upload is made to
Bucket-1. If the trigger cannot be made on the specific path then you can at least add that logic to the Lambda function so you only make the copy if the path is prepended with a specific string.– Joshua Kemmerer
Aug 20 at 18:44