Streamlined AWS Ops: Automating Lambda, Glue, DynamoDB Security

This blog outlines the process of setting up AWS Lambda for identifying unused Lambda functions, Glue jobs, and enabling deletion protection for DynamoDB tables.

Introduction:

In today’s dynamic AWS environment, efficient resource management is paramount to ensure optimal performance and cost-effectiveness. As organisations harness the power of various AWS services like Lambda functions, Glue jobs, and DynamoDB tables, it becomes essential to streamline operations and identify unused resources for optimisation. This blog explores the implementation of an automated solution using AWS Lambda, EventBridge, and SNS to identify and manage unused Lambda functions, Glue jobs, and enable deletion protection for DynamoDB tables.

Prerequisites:

IAM Basics: Understand AWS Identity and Access Management (IAM) for secure management of AWS services. Create roles and permissions for Lambda functions.
Lambda Function Basics: Understand the basic structure of a Lambda function written in Python. Lambda functions in Python are scripts executed in response to events.
EventBridge Basics: Understand the fundamental concepts of Amazon EventBridge, such as events, rules, and event buses. EventBridge facilitates serverless event-driven application connections.
SNS Basics: Understand the fundamental concepts of Amazon Simple Notification Service (SNS), covering topics, subscriptions, and messages.

AWS Lambda:

AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS) that allows you to run your code without provisioning or managing servers. It is a key component of the AWS serverless platform, enabling developers to build and deploy applications without the need to worry about infrastructure.

1. Identify the unused Lambda Functions.

To find Lambda Functions that aren’t being used, we have to create a new Lambda Function. Here are the steps:

Step 1: Create a Lambda function
1.1 Go to Lambda Console and click on “Create function”
Select the values “Author from scratch”, Name, Run Time as shown below.

1.2 Create an Inline policy to and attached to the Lambda execution role. created in step 1.1
Inline Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:ListFunctions",
                "sns:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogStreams",
                "sns:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "logs:GetLogEvents",
            "Resource": "*"
        }
    ]
}

Attach the policy to the Lambda Access Role as below

1.3 lambda function code:
Use the below code and update the lambda function code:

import boto3
from datetime import datetime, timedelta
def get_last_execution_time(lambda_function_name, region='eu-central-1'):
client = boto3.client('logs', region_name=region)
# Get the log group name for the Lambda function
log_group_name = f'/aws/lambda/{lambda_function_name}'
# Get the latest log stream within the log group
response = client.describe_log_streams(
logGroupName=log_group_name,
orderBy='LastEventTime',
limit=1,
descending=True
)
if 'logStreams' in response and response['logStreams']:
last_log_stream = response['logStreams'][0]
last_event_time = last_log_stream.get('lastEventTimestamp', 0) / 1000 # Convert to seconds
return datetime.fromtimestamp(last_event_time)
else:
return None
def is_more_than_32_days_ago(last_execution_time):
if last_execution_time is None:
return False # No execution time available
current_time = datetime.utcnow()
difference = current_time - last_execution_time
return difference.days > 32
def send_email_notification(subject, message, topic_arn):
sns_client = boto3.client('sns')
sns_client.publish(
TopicArn=topic_arn,
Subject=subject,
Message=message
)
def lambda_handler(event, context):
client = boto3.client('lambda')
# Get all Lambda function names in the region
functions = client.list_functions()
result = {}
for function in functions['Functions']:
function_name = function['FunctionName']
last_execution_time = get_last_execution_time(function_name)
if is_more_than_32_days_ago(last_execution_time):
result[function_name] = last_execution_time.isoformat() if last_execution_time else 'Not available'
print(result)
# SNS Configuration
sns_topic_arn = 'arn:aws:sns:eu-central-1:123456789:SNS-rules-snstopic'
if result:
# Prepare message for SNS
message = "Lambda functions with last execution more than 32 days ago in abcd (prod) account:\n\n"
for function_name, execution_time in result.items():
message += f"{function_name}: {execution_time}\n"
# Send email notification
send_email_notification("Lambda Functions Report", message, sns_topic_arn)
return {
'statusCode': 200,
'body': result
}

1.4 Test the code

Here, you will obtain the results of unused Lambda functions as an output that haven’t been used in the last 32 days.

Step 2: Create an Amazon EventBridge schedule to run the above Lambda function automatically to identify the unused Lambda functions.

Amazon EventBridge:

Amazon EventBridge is an AWS service that facilitates event-driven architectures. It acts as a central hub for routing events between different services and applications, allowing for decoupled and scalable architectures. With EventBridge, you can create rules to filter events and route them to various targets, such as AWS services, Lambda functions, or external endpoints. It’s a successor to Amazon CloudWatch Events, offering enhanced features and integrations for building modern cloud applications.

Amazon EventBridge schedule:

An Amazon EventBridge schedule is like setting a timer for actions to happen automatically. You tell EventBridge when you want something to occur, like every day at a certain time or every hour. When that time comes, EventBridge triggers the action you’ve specified, such as running a specific task or sending out a reminder. It’s handy for automating recurring tasks or scheduling events in your applications.

2.1 Go to Amazon EventBridge console.

2.2 Click on “Schedules” in the right side menu options and Click on “Create Schedule”

2.3 Specify Schedule details.

Provide ‘Schedule Name’, ‘Description’, ‘Schedule Group’ as below.

2.4 Schedule Pattern
Specify ‘Schedule pattern’, ‘Schedule type’ as below.

2.5 Create a schedule pattern for every 15th day of the month.

2.6 Select Target
Select target as ‘AWS Lambda’ as shown in the below.

2.7 Target Details Invoke.
Select the Lambda function which we have created in Step 1.

2.8 Target Configure all the settings as below.
Specify ‘Schedule state’, ‘Action after schedule completion’ as below.

Specify ‘Retry policy’, ‘Maximum age of event’ and ‘Retry attempts’ as below.

2.9 Permissions.
Specify the “Execution role” as below.

2.10 Review and create schedule.

2.11 Click on “Create schedule” then the schedule will be created as below.

Step 3: Created an SNS Topic to receive email notifications.

3.1 Go to AWS Simple Notification Service (SNS) console.

3.2 Click on ‘Topics Section’ and click on ‘Create topic’.
Provide the below details and create an SNS topic.

3.3 Create the required subscriptions for the above SNS topic to get the email notifications.

3.4 Update the Lambda function code with the SNS topic ARN created in step 3.2.

Now, the Lambda function has been created for the unused Lambda functions. Notifications will be received every 15 days for Lambda functions that have not been used in the last 32 days.

Step 4: Unused Lambda Functions Report/Result

You will get an email notification with the below report for every 15 days in the month with the unused Lambda functions.

2. Identify the unused Glue Jobs.

Introduction:

AWS Glue makes data integration easier by handling the heavy lifting of extracting, transforming and loading data without developers needing to manage the underlying infrastructure. It seamlessly works with other AWS services to create efficient data pipelines, which means businesses can gain insights faster while saving on operational costs. However, as data pipelines change over time, it’s important to make sure we’re using resources efficiently.

Identifying Unused Glue Jobs:

To use resources wisely, we need to find and check Glue jobs that aren’t being used. We can do this by looking at the records Glue keeps of job activity. By studying when jobs were last run, we can spot ones that haven’t been used in a while. These unused jobs might not be needed anymore, so we can either make them better or get rid of them to save money.

we can follow the same steps as we used to create the Lambda function in steps 1 to 3 to get unused glue job.

Lambda Function Code:

Here’s a simple code for a Lambda function that automatically finds Glue jobs that haven’t been used for more than 32 days. This code does the work of checking for inactive jobs, making it easier for developers and administrators to manage Glue resources effectively.

import boto3
from datetime import datetime, timedelta, timezone
def get_last_execution_time(glue_job_name, region='eu-central-1'):
client = boto3.client('glue', region_name=region)
# Get all runs for the Glue job
response = client.get_job_runs(
JobName=glue_job_name,
MaxResults=100, # Adjust MaxResults as needed
)
if 'JobRuns' in response and response['JobRuns']:
job_runs = response['JobRuns']
# Sort job runs by START_TIME in descending order
sorted_runs = sorted(job_runs, key=lambda x: x['StartedOn'], reverse=True)
last_run = sorted_runs[0]
last_execution_time = last_run.get('StartedOn')
return last_execution_time
else:
return None
def is_more_than_32_days_ago(last_execution_time):
if last_execution_time is None:
return False # No execution time available
current_time = datetime.now(timezone.utc)
# Ensure last_execution_time is timezone-aware
if last_execution_time.tzinfo is None or last_execution_time.tzinfo.utcoffset(last_execution_time) is None:
last_execution_time = last_execution_time.replace(tzinfo=timezone.utc)
difference = current_time - last_execution_time
return difference.days > 32
def send_email_notification(subject, message, topic_arn):
sns_client = boto3.client('sns', region_name='eu-central-1') # Specify the correct region here
sns_client.publish(
TopicArn=topic_arn,
Subject=subject,
Message=message
)
def lambda_handler(event, context):
client = boto3.client('glue')
# Get all Glue job names in the region
jobs = client.get_jobs()
result = {}
for job in jobs['Jobs']:
job_name = job['Name']
last_execution_time = get_last_execution_time(job_name)
if is_more_than_32_days_ago(last_execution_time):
result[job_name] = last_execution_time.isoformat() if last_execution_time else 'Not available'
print(result)
# SNS Configuration
sns_topic_arn = 'arn:aws:sns:eu-central-1:1234567896:SNS-rules-snstopic'
if result:
# Prepare message for SNS
message = "Glue jobs with last execution more than 32 days ago :\n\n"
for job_name, execution_time in result.items():
message += f"{job_name}: {execution_time}\n"
# Send email notification
send_email_notification("Glue Jobs Report", message, sns_topic_arn)
return {
'statusCode': 200,
'body': result
}

You will get an email notification with the below report for every 15 days in the month with the unused Lambda functions

3. Enable Deletion Protection for DynamoDB Table.

Introduction:

DynamoDB is a fully managed NoSQL database service provided by AWS, offering scalability, reliability, and low latency. It automatically distributes data and offers flexible capacity modes, making it a preferred choice for various applications such as web, mobile, gaming, and IoT. However, to ensure data integrity and prevent accidental data loss, it’s important to enable deletion protection for DynamoDB tables.

Amazon EventBridge rule

An Amazon EventBridge rule is like a set of instructions you give to Amazon’s event system. You tell it what to look for in events, like specific types of messages or when to expect them. When the system sees events that match your instructions, it does something you’ve told it to do, like sending you a notification or running a program. It helps automate tasks based on what’s happening in your apps or systems.

Enabling Deletion Protection:

To safeguard DynamoDB tables from accidental deletion, enabling deletion protection is crucial. This feature prevents users from deleting tables, helping maintain data consistency and availability. By following similar steps as before, we can enable deletion protection for DynamoDB tables.

We’ll utilise AWS services like Lambda functions and Amazon EventBridge rules to automate this process, ensuring tables remain protected without the need for manual intervention.

we can follow the same steps as we used to create the Lambda function in step 1 to enable deletion protection for the DynamoDB table. There’s no need to set up an SNS for notification. Just include the provided Lambda function code to enable deletion protection. Also, ensure to create an Amazon EventBridge rule instead of setting up a schedule.

Lambda Function Code:

Below is a Lambda function code snippet designed to enable deletion protection for DynamoDB tables. This code snippet simplifies the process of enabling deletion protection, making it easier for developers and administrators to safeguard their DynamoDB resources.

3.1 Here’s the Lambda function code to enable deletion protection for DynamoDB tables:

import boto3
def lambda_handler(event, context):
dynamodb = boto3.client('dynamodb')
# Get a list of all tables in the current region
existing_tables = dynamodb.list_tables(ExclusiveStartTableName='first DynamoDB table name', Limit=100)['TableNames']
# Enable deletion protection for existing tables
for table in existing_tables:
try:
response = dynamodb.update_table(
TableName=table,
DeletionProtectionEnabled=True
)
print(f"Deletion protection enabled for table: {table}")
except Exception as e:
print(f"Error enabling deletion protection for table {table}: {e}")
# Check for newly created tables using DynamoDB Streams events
if 'Records' in event and 'dynamodb' in event['Records'][0]:
new_table_name = event['Records'][0]['dynamodb']['NewImage']['TableName']['S']
# Enable deletion protection for the newly created table
try:
response = dynamodb.update_table(
TableName=new_table_name,
DeletionProtectionEnabled=True
)
print(f"Deletion protection enabled for newly created table: {new_table_name}")
except Exception as e:
print(f"Error enabling deletion protection for newly created table {new_table_name}: {e}")
return {
'statusCode': 200,
'body': 'Deletion protection enabled for existing and newly created tables.'
}

3.2 Amazon EventBridge schedule.
Navigate to Amazon EventBridge and create a rule by providing the necessary details.

When a new DynamoDB table is created, deletion protection will be turned on automatically.

Result: All the below DynamoDB table delete protection is enabled.

Conclusion:
In the constantly changing world of AWS, using automation is like having a superpower for managing resources well. When you set up automated systems using tools like AWS Lambda, EventBridge, and SNS, it's like having a team of helpers working behind the scenes. They help make operations smoother, find things you're not using anymore, and make sure your data stays safe. So, don't hesitate to use automation to make the most of your AWS setup and help your organisation thrive in the digital age.