Amazon S3 is one of the most popular object storage services that apps use today. Monitoring S3 buckets for activity can be very beneficial, depending on the reason the bucket stores data.

For example, the Kloudless File Picker provides an easy way for users to upload content to an app’s S3 bucket. Apps can monitor S3 for new files to process rather than write client-side logic to trigger data processing when a user completes an upload.

AWS docs describe monitoring activity in S3 using SNS. In this article, we will describe a generalized version that enables events to be consumed by multiple receivers via SQS instead. Kloudless adopts this approach to scale to multiple environments and downstream processors rather than restrict the notifications to a single receiver. This is critical since S3 buckets do not support multiple notification configurations.

There are three steps to receive notifications from buckets:

1. Create an SNS topic to receive notifications sent from a bucket.
2. Create an SQS queue for each receiver, and subscribe to the SNS topic.
3. Configure S3 buckets to publish events to the SNS topic.

We’ll use two AWS Account IDs represented by keywords to demonstrate the process:

• APP_ARN is the ID of the developer’s AWS account
  
• OWNER_ARN is the ID of the bucket owner’s AWS account

The developer’s app creates and configures the SNS topic and SQS queue in its own AWS account, and retrieves events from the queue. The app accesses buckets using the bucket owner’s account to configure the buckets to publish events to the SNS topic.

Create an SNS topic

The app first creates an SNS topic in its own AWS account with the name s3-event-OWNER_ARN. This represents a topic that will receive notifications from the bucket owner’s buckets. The presence of the bucket owner’s ARN ensures uniqueness. Set a policy similar to the one below on the SNS topic:

The Condition above specifies three S3 buckets that can publish to this SNS topic.

Create an SQS queue and subscribe to the SNS topic

Continuing to use the app’s own AWS account, create a queue with name s3-eventq-receiver-1-APP_ARN and set its policy similar to the one below:

Similar to the first policy, the Condition specifies an SNS topic that can send messages to the queue. The queue name includes receiver-1 to uniquely identify the recipient of messages in this queue. Create additional SQS queues for other recipients.

Then, set the queue to subscribe to the SNS topic. Here’s an example of how to do this using the popular Python library, boto3:

The developer app’s IAM account provides the key and secret for the API requests above.

Update the S3 buckets’ notification configurations

Using the bucket owner’s AWS IAM credentials, set the buckets to publish ObjectCreated and ObjectRemoved events to the SNS topic. Here is an example using boto3:

The bucket owner’s key and secret are used in this case.

The developer’s app can now poll the queue for new events! Here is a diagram of the event flow:

The queue’s presence ensures that multiple apps can subscribe to bucket notifications. For example, the subscriber described above, receiver-1, could be the developer’s production application, but still leave room for a development and staging application to receive the same events. As mentioned above, buckets do not natively support more than a single notification configuration, even though the API endpoint accepts a list.

Simplify S3 activity monitoring using Kloudless

The process described above is a simplified version. Real-world use involves complexities such as the ones below:

• Buckets in different regions
• Detecting and monitoring newly added buckets
• Cleaning up SNS topics intended for deleted buckets and/or accounts
• Conflicts with existing notification configurations on buckets

Check out Kloudless for an easy-to-use API that enables your application to monitor S3 as well as several other cloud apps for activity with a single integration.