How to Automate AMI Backup & Cleanups, using AWS Lambda

Is your Dev-ops or Infrastructure management team looking for options for cloud cost optimization and save time by eliminating human errors and the need for dedicated resources (like a standalone server) executing all tasks? Look no further! Automating AMI backups and cleanups using AWS Lambda helps you solve the above predicament to the T. We’ve seen it put to good use, and though we’ll bring it to you in this step-by-step guide to streamline the backup process. So, let’s get started!

What is AWS Lambda?

As Wikipedia says, AWS Lambda is an event-driven, serverless computing platform provided by Amazon Web Services. Introduced in 2014, AWS Lambda simplifies the process of building smaller, on-demand applications that are responsive to events and new information. It runs code in response to events and automatically manages to compute resources required by the code. You can start a Lambda instance within milliseconds! To top it all, it supports Node.js, Python, and Java, as of 2016. AWS Lambda can be managed through various methods, including the AWS Command Line Interface (CLI).

Why do I need AMI Backups and Cleanups? AMI makes it easier and faster to recover an instance in case of a disaster or failure of the instance, and therefore, automating this process is the way to go. In this blog-post, let me take you through the steps involved in automating the AMI backups and cleanups using AWS Lambda (also, automate AMI Backups and Cleanups with ELB tags, using AWS Lambda)

The process, generally comprises of the following steps:

Setup IAM Permissions
Create Lambda Backup Function
Create Lambda Cleanup Function
Schedule Functions
Tagging EC2 Instance

Understanding AMI and EBS Snapshots

Amazon Machine Images (AMIs) and EBS snapshots are two essential components of Amazon Web Services (AWS) that help ensure data backup and disaster recovery. Understanding the differences between these two concepts is crucial for effective data management and protection.

An Amazon Machine Image (AMI) is a comprehensive backup of an entire EC2 instance, including all attached volumes. Think of it as a snapshot of a virtual machine. AMIs are invaluable for disaster recovery scenarios where an EC2 instance becomes corrupted. They simplify deployment with pre-installed software, enable application deployment across different regions, and facilitate server upgrades with all necessary software and configurations intact.

On the other hand, an EBS snapshot is a point-in-time backup of a specific EBS volume. Unlike AMIs, EBS snapshots are incremental, meaning they only capture changes made since the last snapshot. This makes them efficient for regular backups of drives or volumes. EBS snapshots are also useful for changing the type of volume (e.g., from gp2 to io1 for increased IOPs) and creating new volumes based on existing data.

In summary, while AMIs provide a holistic backup of the entire EC2 instance, EBS snapshots offer a more granular backup of individual EBS volumes. Both are critical for ensuring robust data backup and disaster recovery strategies.

Setting Up AWS Backup and EBS Volumes

AWS Backup is a fully managed service that automates backups on AWS, ensuring your data is protected and recoverable. Here’s how to set up AWS Backup and EBS volumes:

Let's now take a closer look at each of them with some demos and screenshots to make it easier.

SETUP IAM PERMISSIONS

In IAM Dashboard, Click on Roles, and** Create New Role** with the Role Name: lamda-ec2-ami-role. Under AWS Service Roles, select AWS Lambda as the Role Type and then proceed to create a role. Go to Policies tab, click Create Policy and select Create your own policy (you can name the policy as lamda-ec2-ami-policy). Paste the content of the following JSON in Policy Document, and click on Create Policy.

IAM roles and policies can be used to manage permissions across multiple AWS accounts.

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “logs:” ], “Resource”: “arn:aws:logs:::” }, { “Effect”: “Allow”, “Action”: “ec2:”, “Resource”: “” } ] } Select the created policy, click on Policy Actions and Attach to select the role already created – lamda-ec2-ami-role and click on Attach Policy. We have just created a role for which we have allowed permissions to EC2 instances and view logs in Cloudwatch:

…and here’s the IAM Role (lamda-ec2-ami-role) with the attached policy (lamda-ec2-ami-policy)

CREATE AWS BACKUP LAMBDA FUNCTION

Now that we have created a role and a policy, we’ll have to create the first function that allows us to perform an EC2 backup for every instance in our account, which has a** “Backup”** key tag. We don’t have to indicate a value here. Here’s how the AMI backup script works:

The script will first search for all ec2 instances having a tag with “Backup” on it.
As soon as it has the instances list, it loops through each instance and then creates an AMI of it.
Also, it will look for a “Retention” tag key which will be used as a retention policy number in days. If there is no tag with that name, it will use a 7 days default value for each AMI.
After creating the AMI it creates a “DeleteOn” tag on the AMI indicating when it will be deleted using the Retention value and another Lambda function.

So here’s how you can create your first function. Go to Services, Lambda, and click Create a Lambda Function:

Click on Functions Menu on the left, and click on Create a Lambda Function
Select Blank Function and proceed with lambda
Give a name for it - lambdaAMIBackups
Select Python 2.7 as a Runtime option * You’ll have to write a code next. Refer this github page for a sample code
Select the previously created IAM role (lamda-ec2-ami)
Click Next and Create Function

While creating lambda function, make sure to choose the IAM role created earlier (lamda-ec2-ami-role) and have specified sufficient memory and timeout configurations.

CREATE LAMBDA CLEANUP FUNCTION FOR EBS SNAPSHOTS

Having successfully created the AMI using the previous function, we need to now remove them when not needed anymore. Here’s how the AMI cleanup script works:

The script first searches for all ec2 instances having a tag with “Backup” on it.
As soon as it has the instances list, it loops through each instance and reference the AMIs of that instance.
It checks that the latest daily backup succeeded then it stores every image that’s reached its DeleteOn tag’s date for deletion.
It then loops through the AMIs, de-registers them and removes all the snapshots associated with that AMI.
Additionally, the script ensures that the root volume is properly managed and cleaned up.

Using the same steps as before, create the function (lambdaAMICleanup) and refer this sample code. You will end up with something like this:

So, you have 2 working functions that will backup AMI and remove those when “DeleteOn” specifies. And now, it’s time to automate using the Event sources feature from Lambda.

SCHEDULE FUNCTIONS

We need to run at least once a day both. Login to your AWS Management console, Go to Services, and click on Lambda under Compute.

Click on Functions Menu on the left, and click on the function you want to schedule - lambdaAMIBackups for example.
Go to Triggers tab and click on Add Trigger
Select Cloudwatch Events - Schedule
Type Rule Name ( create-ami for example)
Go to Schedule Expression: select cron and modify accordingly with the schedule time.
Check Enable Trigger
Click Submit

You can use the AWS Backup Dashboard to monitor the status of your scheduled backup jobs. It provides real-time updates and visibility of backup and restore jobs, ensuring that your backup plans are running as expected.

Note that the schedule time is shown in in UTC format – something like this: Lambda AMI Backup Function scheduled to run at 18:45 UTC or 12.15AM IST and Lambda AMI Cleanup Function scheduled to run at 19:30 UTC or 12.45AM IST every day. You can now see the rules under Services » Management Tools » CloudWatch » Rules; after you create the schedule:

TAGGING EC2 INSTANCE

Having created AMI backup and cleanup functions and scheduling them, now it’s time to create a tag for the EC2 instance with a tag-key Backup with no value and Retention with retention days. Login to your AWS Management console, Go to Services, and click on EC2 under Compute.

Click on Instances menu
Select the Instance you want to tag (Linux-test, for example).
Go to Tags » Add Tags and add a tag with Backup with no value and with Retention value 4, for example.

Here, tag-key “Backup” is used to identify the instance for which the AMI has to be created and tag-key “Retention” with value “4” ensures that we retain AMI for 4 days and delete after 4 days. If the Retention tag is not used then, by default, it retains for 7 days. Now you can see the AMI created with tag-key “DeleteOn” with deletion date. Instance store volumes can also be tagged for backup purposes, but they are more suitable for temporary data.

This AMI will delete on the date shown in Value, only when there is a successful AMI created for that day. That’s it! We’ve successfully used AWS Lambda to automate AMI backups and cleanups. The process might look daunting at first, but I’m sure that this step-by-step guide would help keep it simple.

Is there a limitation?

Well, off-course. The process works very well for all standalone instances unless the instance terminates. In cases where you might have a load balancer serving many instances (different types of it), the tag attached to one instance may terminate for some reason, therefore, not creating the AMI. However, there's a workaround using ELBs. It fetches all the load balancers with pre-defined tags (Backup for example), puts them in an array and passes them through a loop, before picking one instance attached to it, to create the AMI.

There is a whole lot more in our forthcoming blogpost, but until then, please keep the comments and feedback coming!