AWS Devops, tooling and process

10th March 2022

DevOps is a combination of specific engineering practices and patterns, followed by cultural changes that increase an organization’s or team’s ability to deliver high-quality products quickly.  Most organizations have adopted DevOps to some extent, and over time, several practices emerged, which can be broadly categorized into:

  • continuous integration;
  • continuous deployment;
  • infrastructure as code (IaC); and
  • monitoring and logging.

The interest in DevOps and related technologies in recent years has pushed leading cloud providers, such as AWS, to provide all the necessary tools and technologies for organizations to implement and adopt DevOps practices successfully. Providers also look to increase value to their customers by delivering more reliable products at a faster pace than traditional deployment cycles.

AWS simplifies the process to build and manage infrastructure, ship code faster to applications and monitor the infrastructure and application, all while measuring application performance or telemetry. For each of the previously mentioned DevOps practices, AWS offers specific tools, such as:

Apart from these tools, AWS offers other tool sets, services and platforms organizations use to deploy to the cloud easily, consistently and quickly, like AWS Elastic Beanstalk and Amazon Elastic Kubernetes Service (Amazon EKS).

AWS offers a variety of services to organizations that practice DevOps and facilitate rapid build, test and delivery for complex environments that can scale. Below are some AWS developer tools and descriptions that can help organizations start their DevOps journey on AWS, categorized by practice.

Continuous integration and continuous delivery (CI/CD)

Continuous integration is a development practice to push and merge code changes into a centralized code repository. These are integrated continuously to detect changes and run a predefined set of automated actions, such as tests, scans and builds.

AWS offers several tools to help with the practice:

  • AWS CodeCommit. A source control service on AWS that hosts private Git repositories and works seamlessly with existing Git-based tools. Because it is available on AWS portal as a hosted platform, developers can also use the console version to collaborate and commit, branch and perform other operations on the code repositories from a web UI.
  • AWS CodeBuild. A continuous integration service that builds and compiles source code, runs a predefined set of tests and generates artifacts as deployable software.
  • AWS CodeArtifact. An artifact repository service that enables DevOps teams to store, publish and share software packages built during various DevOps stages.

Continuous delivery, as the name suggests, focuses primarily on deploying code or configuration changes automatically when a change is detected in a source code repository. This ensures all changes developers push — regardless of how small — are deployed, tested and validated immediately. Because the small changes are deployed continuously in case of any issues or bugs, it is easier to detect the bug and revert back those changes. AWS DevOps tools that support this process include:

  • AWS CodeDeploy. A deployment service that automates software packaging and deployments to various AWS providers and services, such as AWS Simple Storage Service (S3), Elastic Compute Cloud (EC2), Lambda or any custom service or location, such as on-premises instances. It eliminates manual deployments to ease deployments of software packages to a chosen instance and release features quickly with minimal or no downtime.
  • AWS CodePipeline. A continuous delivery service that enables IT teams to define the steps and workflows required to release software. For example, deploy software packages to dev and staging environments first, where a series of tests and scans are run, before it’s released to production instances.
  • AWS CodeStar. A service that further speeds up the development process and overall experience with CI/CD tools in AWS. It creates a unified CI/CD interface that enables developers to set up the entire continuous delivery toolchain within minutes.

Infrastructure as code

Infrastructure as code is one of the most central principles of DevOps, where infrastructure is treated as software, including configurations and templates. IaC enables developers to rapidly provision, configure and manage AWS infrastructure in a consistent and secure way. This method replaces manual deployments and configurations, which not only saves time, but incorporates infrastructure as part of the application build and deployment.

CodeCommit stores infrastructure as code around which DevOps teams build pipelines to provision infrastructure via CodePipeline. AWS offers the following services to define infrastructure as a code:

  • AWS CloudFormation. A service that enables developers to provision AWS resources and infrastructure in a consistent and predictable manner using an array of ready-to-use templates or by writing custom templates in specific syntax and structure.
  • AWS Cloud Development Kit (CDK). A development framework that enables developers to provision infrastructure using programming languages like Python, .Net and Java, without leaving the comfort of an integrated development environment and rich text editor features like autocomplete.
  • AWS OpsWorks. A configuration management service that helps developers automate server configurations on EC2 instances or on-premises servers, by providing managed Chef and Puppet instances.
  • AWS Systems Manager. A service that helps automatically collect software inventory, apply OS patches, create system images and configure Windows and Linux OSes. Systems Manager is an end-to-end management tool that improves visibility and control into systems against any configurations drifts or noncompliance.
  • AWS Config. A policy-as-code tool that enables more secure hybrid cloud environments by continuously evaluating configurations and activity against a set of predefined policies managed by security administrators.

Monitoring and logging

Observability into infrastructure health and application performance to detect failures, outages and performance bottlenecks is critical for every organization. To ensure everything is running at desired levels and state, AWS provides two different services for monitoring and logging.

  • Amazon CloudWatch. A monitoring and observability service that automatically collects data and metrics from most of AWS’ services, like EC2, S3 and Amazon Relational Database Service instances. Developers can also define custom metrics to be collected from any AWS resource. CloudWatch enables users to set up alarms, as well as auto scale and run an AWS Lambda function if an alarm is triggered.
  • AWS X-Ray. A service that enables developers to analyze distributed traces in distributed applications — like microservices — to understand and debug any underlying performance issues or general failures.

Popular third-party DevOps tools

A single cloud provider cannot always provide the best tools for a domain as vast as a DevOps ecosystem, which AWS acknowledges via integrations with most third-party tools to give organizations the flexibility to choose what arrangement of tools, services and platforms works for them.

The following are some third-party tools that work well with AWS, broken into various DevOps categories:

  • Continuous integration. For code repositories: Bitbucket, GitHub and GitHub Enterprise. For testing: Gremlin, BlazeMeter and Selenium for web testing.
  • Continuous deployment. For building: Jenkins and JetBrains TeamCity.
  • Infrastructure as code. Terraform, CFT.
  • Monitoring and logging. Datadog, Prometheus, Grafana and Splunk Observability Cloud.
  • Configuration management. Puppet, Chef and Ansible.
  • Containerization. Docker, Docker Swarm and Kubernetes.

How to create a CI/CD pipeline in AWS

One can create a simple CI/CD pipeline in AWS.

  1. Set up a code repository on CodeCommit with a static HTML file.
  2. Create an S3 bucket to host a static website using the HTML file.
  3. Set up CodePipeline, which is continuously integrated to CodeCommit to deploy any new changes in the HTML file to the S3 instance.
  4. Test the pipeline.

This pipeline will be triggered each time there is a new commit record in the CodeCommit repository. Once deployment completes, the static website will display any changes.

Source