S3 File Upload Notification System

In an event-driven or microservices architecture, the importance of S3 events cannot be overstated. In this blog, we will explore the implementation of a File Upload Notification System using Amazon Web Services (AWS). This system will monitor an S3 bucket for new file uploads, trigger a Lambda function to process the uploaded files, send a notification using Simple Notification Service (SNS), and store metadata in a Simple Queue Service (SQS) queue for further processing.

1. Set Up S3 Bucket

The first step is to create an Amazon S3 bucket to store the uploaded files. To create a bucket, log in to the AWS Management Console and navigate to the Amazon S3 service. Click on the "Create bucket" button and provide a unique and globally-unique name for the bucket e.g., notificationbucket2. Choose the appropriate AWS Region and leave the other options at their default values. Once the bucket is created, it will serve as the storage location for the uploaded files.

Note: We will be adding an S3 event rule to this bucket later.

Amazon Simple Notification Service (SNS) is a fully managed communication service that allows you to send notifications. To create an SNS topic:

Create a new SNS topic

In the AWS Management Console, navigate to the SNS service. Create a new SNS topic, e.g., notificationsystem, which will be used for sending notifications. Select Standard, does not care about the order

Subscribe and Activate:

Subscribe the topic to an email address. Choose the email protocol, enter the endpoint email.

Activate the subscription by confirming the email.

Note: This step ensures that notifications are sent to the specified email.

3. Set Up SQS Queue

Within the SQS dashboard, click on the "Create Queue" button. Create a new SQS queue, e.g., notificationqueue. Choose the queue type based on your requirements. In this case, select the "Standard Queue" option. Standard queues provide at-least-once delivery of messages, making them suitable for scenarios where occasional duplication of messages is acceptable.

Depending on your specific requirements, you may adjust additional settings such as the message retention period and visibility timeout. This queue functions as a dedicated repository for storing metadata associated with uploaded files.

4. Set up Lambda Function

Lambda is like a super helpful computer service from Amazon. It lets developers do their coding without worrying about setting up or taking care of actual servers. For our File Upload Notification System, making a Lambda function is a really important part. It's like creating a special tool that handles the files you upload and makes sure everything else works smoothly afterward.

4.1 Create a Lambda Function:

In the AWS Management Console, navigate to the Lambda service. Create a new Lambda function, e.g., notificationfunction, selecting the Python 3.12 runtime environment.

4.2 Grant Permissions

Lambda functions need specific permissions to interact with other AWS services. AWS provides execution roles to manage these permissions. Grant permissions to read from the S3 bucket, write to the SQS queue, and publish to SNS. Open the role (notificationfunction-role-0ffc7kxe) and add permissions for

AmazonSNSFullAccess
AmazonSQSFullAccess

Giving specific and limited permissions to your Lambda function makes it more secure. It means the function only gets the exact rights it requires, following the principle of least privilege

4.3 Write Lambda Function Code

When a new file is uploaded to an S3 bucket, this function is triggered. It extracts information about the uploaded file, such as the bucket name and file key. You can customize the processing logic for your specific needs. After processing, the function sends a success notification using Amazon SNS (Simple Notification Service) to a specified topic. Additionally, it stores metadata related to the processed file in an Amazon SQS (Simple Queue Service) queue for further handling or analysis. This Lambda function orchestrates the notification system, connecting various AWS services seamlessly.

import json
import boto3

s3_client = boto3.client('s3')
sns_client = boto3.client('sns')
sqs_client = boto3.client('sqs')

def lambda_handler(event, context):
    sns_topic_arn = 'arn:aws:sns:ap-south-1:928836847222:notificationsystem'
    #go to sns console and copy arn from there 

    # Define the SQS queue URL
    sqs_queue_url = 'https://sqs.ap-south-1.amazonaws.com/928836847222/notificationqueue'

    # Process S3 event records
    for record in event['Records']:
        print(event)
        # Extract S3 bucket and object information
        s3_bucket = record['s3']['bucket']['name']
        s3_key = record['s3']['object']['key']

        # Example: Sending metadata to SQS
        metadata = {
            'bucket': s3_bucket,
            'key': s3_key,
            'timestamp': record['eventTime']
        }

        # Send metadata to SQS
        sqs_response = sqs_client.send_message(
            QueueUrl=sqs_queue_url,
            MessageBody=json.dumps(metadata)
        )

        # Example: Sending a notification to SNS
        notification_message = f"New file uploaded to S3 bucket '{s3_bucket}' with key '{s3_key}'"

        sns_response = sns_client.publish(
            TopicArn=sns_topic_arn,
            Message=notification_message,
            Subject="File Upload Notification"
        )

    return {
        'statusCode': 200,
        'body': json.dumps('Processing complete')
    }

Deploy the Function: After defining the function code, deploy your Lambda function by clicking on the "Deploy" button.

5. Configure S3 Event Trigger

In the S3 bucket properties, configure an event to trigger the Lambda function. Navigate to Amazon S3 > Buckets > notificationbucket2 > Properties. Choose the Lambda function created earlier as the target for the S3 event.

Configuring the S3 event trigger establishes a seamless connection between your S3 bucket and the Lambda function. Whenever a new file is uploaded to the specified S3 bucket (notificationbucket2), the configured

6.Test the System

This step involves uploading a file to the S3 bucket and closely observing the workflow to ensure that each component performs its designated tasks seamlessly.

Navigate to S3 bucket (notificationbucket2) where the file uploads are monitored. Upload a file to the S3 bucket using the AWS Management Console or any preferred S3 upload method.

Confirm that you receive a notification, as configured, through the chosen SNS protocol (e.g., email).

7. Examine Logs Using AWS CloudWatch

To gain deeper insights into the execution of your Lambda function and identify potential issues or optimizations, AWS CloudWatch provides a comprehensive log monitoring solution.

Navigate to the CloudWatch service. Look for a log group related to your Lambda function. Log groups are organized by AWS service and function name. Open the log stream to view log events generated during the execution of your Lambda function. Each log event provides information about the function's behavior, including any print statements or errors.

Conclusion

The S3 File Upload Notification System project effectively integrates AWS services, orchestrating a seamless workflow from file uploads to notifications and metadata storage. Configuring S3 event triggers, testing the system, and examining logs using AWS CloudWatch ensures reliability and provides ongoing monitoring capabilities. This project offers a concise blueprint for building responsive and scalable systems on the AWS cloud infrastructure, catering to diverse applications across industries.

S3 File Upload Notification System

1. Set Up S3 Bucket

3. Set Up SQS Queue