Hello Engineers, In this article I am going to explore a CLI tool that will be a game changer for containerized applications and make it easy to deploy on Amazon Web Services. Recently I talked about AWS Copilot at the AWS Hungary Conference and shared my insights on this technology developed by the AWS Containers team.
I will be doing hands-on on this, so you can practice within your environment as you are reading this.
What is AWS Copilot
AWS Copilot is a Command Line Interface (CLI) tool that simplifies the process of deploying and operating containerized applications on AWS. It automates the creation of necessary infrastructure for running applications, such as load balancers, Amazon Elastic Container Registry, Amazon Elastic Container Service, and IAM roles. Copilot also manages your applications' deployment and operational tasks, allowing you to focus on writing code. With AWS Copilot, you can easily create, deploy, and manage your containerized applications on AWS.
AWS Copilot handles infrastructure provisioning and management for you where you don't manage EC2 instances. You define your application using simple YAML manifest files Copilot automatically creates services, load balancers, pipelines, etc.
Why AWS Copilot
Before AWS Copilot, To deploy and work with containerized applications engineers used AWS Elastic Container Service (ECS) which is a highly scalable and efficient container management service provided by Amazon Web Services (AWS). It allows you to run, manage, and scale containerized applications on a cluster of virtual machines within the AWS ecosystem. You have full control over infrastructure along with managing EC2 instances and registering them with the ECS cluster. You define task definitions that describe your containerized applications and might need to create services, load balancers, etc manually. Copilot makes it super easy to set up and deploy your containers on AWS - but getting started is only the first step of the journey.
You now understand the AWS copilot and its benefits, let's make our hands dirty by doing hands-on
Install AWS Copilot
To start playing with AWS Copilot, you first need to install it on your local machine. It ships as a separate package or software to one command install.
Below are commands to install it on different platforms
Linux:
curl -Lo copilot https://github.com/aws/copilot-cli/releases/latest/download/copilot-linux
chmod +x copilot && sudo mv copilot /usr/local/bin/copilot && copilot --help
MacOS:
curl -Lo copilot https://github.com/aws/copilot-cli/releases/latest/download/copilot-darwin
chmod +x copilot && sudo mv copilot /usr/local/bin/copilot && copilot --help
or you can use the brew package manager to install:
brew install aws/tap/copilot-cli
Initialized AWS Copilot for the Project
You can deploy any project on ECS by the copilot. If you don't have any project right now, here's a repo where you can practice all the stuff we are about to learn. After Installing the AWS Copilot and cloning the project if you don't have one, Go to the project that you want to deploy, and within your code directory run:
copilot init
The next thing we will do is answer a few questions from Copilot. Copilot will use these questions to help us choose the best AWS infrastructure for your service.
Once Copilot finishes setting up the infrastructure to manage your app, you’ll be asked if you want to deploy your service to a test environment type yes.
Now we can wait a few minutes ⏳ while Copilot sets up all the resources needed to run your service. After all the infrastructure for your service is set up, Copilot will build your image push it to Amazon ECR, and start deploying to Amazon ECS on AWS Fargate.
After your deployment completes, your service will be up and running and Copilot will print a link to the URL 🎉!
Copilot will handle the deployment process, creating AWS resources, and configuring the necessary infrastructure. Under the hood, Copilot bootstraps these AWS resources when setting up your application.
ECS Cluster
ECS Service
Fargate Tasks
ECR Repo
VPC
Subnets
Security Groups
Load Balancer
After the deployment of our application using Copilot, it creates a cluster on ECS where you can see or manage it from the AWS Console.
This also creates a Repository on ECR with the image provided from the project's docker file.
When working with AWS Copilot, there are some concepts that we should be aware of. Let's look into this one by one.
Manifest Files
After setting up and deploying your copilot application, in your project, you will see a new folder will be created named copilot contains some manifest files for the application services and environments. This manifest.yml file contains metadata that describes a service's, a job's, or an environment's architecture as infrastructure-as-code. It is a file generated from copilot init, copilot svc init, copilot job init, or copilot env init that gets converted to an AWS CloudFormation template. Manifest files are always stored under copilot/<your service, job, or environment name>/manifest.yml.
Application
An application refers to a top-level group composed of services, environments, and pipelines that are related to one another. It enables users to organize their services into an application, irrespective of whether it is composed of a single service that performs all functions or a constellation of microservices. The tool categorizes the services and environments into which they can be deployed, thereby creating an efficient and structured approach to service management.
You can see your deployed app details using the command:
copilot app show
You will see all your app details on the terminal.
Below are additional Copilot App Commands that are self-explanatory:
Environments
When you work with applications, you might need to create different versions of a service for different environments. For example, you might want to create a development environment (dev) and a production environment (prod). To make this process easier, you can use the copilot init command. This command allows you to create a test environment that contains all the AWS resources needed to provision a secure network (VPC, subnets, security groups, etc.), as well as other resources that can be shared between multiple services. These resources include an Application Load Balancer or an ECS Cluster. When you deploy your service into your test environment, it will use the test environment's network and resources. Your application can have multiple environments, and each one will have its networking and shared resources infrastructure.
In the above image, you can see the environment for the application we deployed earlier named Stage. We can have multiple environments according to our scenarios.
Let's create another environment for our app and name it prod.
copilot env init
After running this command, you'll see logs something like the below and you can verify your env by checking into this folder copilot/environments/prod/manifest.yml.
Service
Service is your actual application code and infrastructure needed to get it up and running on AWS. Remember when we got asked by Copilot after running copilot init about our service name and type where we selected Load Balanced Web Service? AWS has some pre-defined services to cater to almost every use case. As for our application, we wanted to deploy it on Amazon Load-Balancer so we selected that service type.
Public Facing Internet Service
If you want your service to serve public internet traffic, you have three options:
Request-Driven Web Service - will provision an AWS App Runner Service to run your service.
Static Site - will provision a dedicated CloudFront distribution and S3 bucket for your static website.
Load Balanced Web Service - will provision an Application Load Balancer, a Network Load Balancer, or both, along with security groups, and an ECS service on Fargate to run your service.
Backend Service
If you require a service that is only accessible from within your application and cannot be accessed externally, you can create a Backend Service. Copilot will set up an ECS Service that runs on AWS Fargate, but it won't configure any endpoints that are accessible from the internet.
Worker Service
Worker Service enables your application's microservices to communicate with each other asynchronously using a pub/sub architecture. By publishing events to Amazon SNS topics, your microservices can share information with a Worker Service that consumes these events. This allows for a more efficient and scalable application architecture, where each microservice can focus on its specific task without being burdened by the need to directly communicate with other services. With Worker Services, your application can achieve high performance and reliability while maintaining flexibility and modularity.
Run this command to see what services are we using for our app:
copilot svc show
Job
Amazon Elastic Container Service (ECS) provides Jobs, which are tasks triggered by an event. Copilot simplifies creating and deploying applications on ECS. Copilot supports only Scheduled Jobs, which run at a fixed rate or schedule.
Select the app and job type, then define the schedule or rate. Copilot automatically creates AWS resources and deploys the task definition to ECS.
Command to create a job:
copilot job init
or to list existing jobs:
copilot job ls
Currently, we don't have a job created with our environment or application, you might see a blank list.
Copilot Jobs offers logs that display the most recent activity of your job. You can track your logs in real-time using the --follow flag. This will show logs from any new instances of your job after you execute the command.
copilot job logs
Sample Logs:
copilot/myjob/37236ed Doing some work
copilot/myjob/37236ed Did some work
copilot/myjob/37236ed Exited...
copilot/myjob/123e300 Doing some work
copilot/myjob/123e300 Did some work
copilot/myjob/123e300 Did some additional work
copilot/myjob/123e300 Exited
Now after doing all of the above stuff, I recommend you delete all of your resources in AWS so that you don't get surprised at the end of the month.
For that just run the command:
copilot app delete
It will remove all the resources that have been created by Copilot along with all the services, stacks, environments, applications, etc.
If you like this article, do like, comment, share, and let me know if you have created something with AWS Copilot. And don't forget to follow me on Linkedin, Github, and Twitter.
Peace ✌🏻