Your Best Friend For CI/CD - What Is CircleCI?
Automation and CircleCI come in here – a top CI/CD tool. Below, we explore what is CircleCI, plus more.
Automation helps speed up work, and a big part of that is CI/CD.
The first part, CI, stands for Continuous Integration. This concept encourages developers to regularly integrate code they’re working on with a shared repository – as frequently as possible. When the code merges, that triggers automated tests and builds.
The second part, CD, is ambiguous. It stands for Continuous Delivery or Continuous Deployment. Both mean further automation of the previous integration process. Continuous Delivery is about the automated creation of images in software applications.
These images are present in some kind of repository and are ready for manual deployment. With Continuous Deployment, that manual step is also covered by the automated process.
What we’ve outlined may seem like a lot of unnecessary automation fuss, but these practices are worthwhile investments. With CI/CD platforms, software teams are notified about bugs faster, helping deliver a more stable product while increasing team confidence and productivity. To give you a better idea, let’s take a look at CircleCI Continuous Integration tool.
What is CircleCI?
CircleCI is a CI/CD solution that helps improve your team’s efficiency. It provides a managed cloud service, and also has a self-hosted option. You can access CircleCI via its website – there’s a free tier, so it’s easy to try out.
Over the years, CircleCI has become one of the most popular CI/CD platforms. Below, we take a look at its features and example pipelines to see why so many companies choose CircleCI to automate their workflows.
Components of CircleCI
In general, most CI tools have a similar way of working. Despite that, it’s important to familiarize yourself with the CircleCI ecosystem, so you understand the example code snippets later in this post.
Steps
Steps are the fundamental building blocks of the workflows. Most of the time, they consist of executable bash commands. Other times, it’s possible to use predefined steps provided by CircleCI, covering common use cases – for example, checking out code from a repository or handling pipeline cache.
Jobs
Jobs have two configuration pieces. First, they need a list of steps. These steps are run as a single unit, in order, when a job is executed. Second, executing a job requires an executor. CircleCI supports many platforms – Docker, Windows, macOS, and virtual machines.
#...
jobs:
build: # Job name
docker: # Job executor
- image:
auth:
username: mydockerhub-user
password: $DOCKERHUB_PASSWORD # context / project UI env-var reference
steps:
- checkout # Special step to checkout your source code
- run: # Run step to execute commands, see
# circleci.com/docs/configuration-reference/#run
name: Running tests
command: make test # executable command run in
# non-login shell with /bin/bash -eo pipefail option
# by default.
#...
Job and steps example
Workflows
Workflows sit at the top of the hierarchy. They consist of jobs, each of which has a list of steps.
To mitigate long execution times, jobs execute in different ways – sequentially, parallel, or manually. We take a more detailed look at optimization techniques later on.
#...
workflows:
build_and_test: # name of your workflow
jobs:
- build1 # predefined job name from the jobs section
- build2:
requires:
- build1 # wait for build1 job to complete successfully before starting
# see circleci.com/docs/workflows/ for more examples.
- build3:
requires:
- build1 # wait for build1 job to complete successfully before starting
Workflow example
Orbs
Workflows are implemented from the ground up. To avoid duplicate configurations, orbs come into play. Orbs provide reusable snippets of code, covering common use cases like creating and uploading a Docker image to Docker Hub. CircleCI provides an orbs repository that’s easy to find and use, and anyone can browse.
source: circleci.com
Context
It’s well-known that pipeline tools provide ways to create configuration options. CircleCI offers this via contexts that hold a set of environment variables. These values are available across projects in your organization.
Let’s say you have a secured API that all services access. You can build a context and configure the API key as an environment variable, and each team references the context in their pipelines. At runtime, CircleCI injects the variables into the workflow jobs.
Pipeline optimization
After a while, it’s common for build application times to increase. With that in mind, CircleCI provides a list of features to make processes faster and more straightforward.
Concurrency
In CircleCI terms, concurrency refers to job execution. If there are many jobs in a workflow and they aren’t dependent on each other, they’ll run concurrently. Note: there’s a soft limit on this.
From the previous section, we know that jobs can have various executor environments. On top of that,you can fine-tune the executors based on CPU and memory needs. It’s possible to influence those factors by providing a resource class like small, medium, or large.
The limitation factor comes into play if you’re using a lot of CPU and memory power. If that happens, CircleCI limits concurrency to make sure the platform stays stable. But fear not, CircleCI’s premium support can raise the limit for you.
As a rule of thumb,don’t define requirements for each job in a workflow if possible, so they run concurrently.
Parallelism
This is interpreted at job level, indicating how many executors are needed to complete the specific job. We use this feature to spread out test executions – it’s less complicated than it sounds.
CircleCI has a command line tool, helping slice test cases into more manageable piles. For splitting logic, use one of three methods: time, name, or file size.
Data persistence
During builds, many files are generated and handled. There are dependencies, test reports, compiled code, and images – to name a few. At some point, you’ll likely need to access and use that data, so you need a place where you can store it. CircleCI offers the following ways:
- Cache – store a file or directory with a key. Jobs save to the cache or retrieve data from it via the key. For example, use this to speed up dependency managers.
- Workspace – these are workflow-aware storage spaces. Use them to pass generated version numbers between jobs or compiled code that you need later on.
- Artifacts – a storage class that persists data after a workflow completes. These are useful for storing jar files and test reports.
Docker Layer Caching
Docker is regularly used in pipelines. For example, we use Docker executors for jobs or have steps that generate some kind of Docker image. This CircleCI feature caches common Docker layers. They persist through workflow runs and speed up the build process of the images.
Extra features of CircleCI
To round up our introduction of CircleCI, we have a few more features to mention.
- Branch-specific configuration – it’s possible to exclude or include jobs based on the current branch the pipeline is currently executing.
- Conditional execution of steps - you can write conditions, and if they test true, the step is executed. You can see examples if you look at the orb repository, where you can view the source code.
- Commands – these define a sequence of steps and are a useful feature to make codes reusable.
- Parameters – defined under jobs, commands, and executors, parameters are best used with conditions to define dynamic workflows.
Examples of CircleCI for Java
So far, we’ve covered theory. Now, let’s outline code examples.
To create a CircleCI workflow, you need to define the following:
- List of orbs you’d like to use
- List of defined jobs, if something isn’t supported by an orb
- List of workflows that will execute the jobs
Put this information in the yml file .circleci/config.yml of your project and CircleCI will pick it up.
Gradle simple workflow
version: 2.1
orbs:
gradle: circleci/gradle@3.0.0
aws-ecr: circleci/aws-ecr@8.1.2
workflows:
build-and-push:
jobs:
- gradle/run: # <--------------- 1.
executor:
name: gradle/default
tag: 17.0.4
post-steps: # <--------------- 2.
- persist_to_workspace:
root: build
paths:
- libs
- aws-ecr/build-and-push-image: # <--------------- 3.
repo: ng-java-circle-test-backend
attach-workspace: true
workspace-root: ./build
tag: ${CIRCLE_BRANCH}.${CIRCLE_SHA1}
requires:
- gradle/run
Key highlights:
- The above is a job from the official Gradle orb. By default, it’ll run the build command. It also uses a cache to store the dependencies. We made a small change to the default settings, because the base job is currently configured to use Java 13. Using the tag parameter, we changed it to a newer version that’s present in the CircleCI image repo.
- We defined pre- and post-steps for the job to add new functionality. Here, we wanted to carry the jar file to the next job so we could push it to the Elastic Container Registry (ECR). To make this happen, we also used the workspace feature.
- This is a job from the AWS-ECR orb. We only needed simple parameters to ensure the jar file was available from the previous job. For tagging, we used CircleCI’s built-in environment variables. Beneath the surface, the orb uses extra configurations – a list of environment variables – that we need to communicate with AWS:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_ECR_REGISTRY_ID
- AWS_REGION
Gradle parallel workflow
version: 2.1
jobs:
build-with-caches-and-parallel:
parallelism: 2
environment:
GRADLE_OPTS: "-Dorg.gradle.daemon=false -Dorg.gradle.workers.max=2"
docker:
- image: cimg/openjdk:17.0.4
steps:
- checkout
- restore_cache:
key: v1-gradle-wrapper-
- restore_cache:
key: v1-gradle-cache-
- run:
name: Run tests in parallel
command: |
cd src/test/java
# Get list of classnames of tests that should run on this node
CLASSNAMES=$(circleci tests glob "**/*.java" \
| cut -c 1- | sed 's@/@.@g' \
| sed 's/.\{5\}$//' \
| circleci tests split --split-by=timings --timings-type=classname)
cd ../../..
# Format the arguments to "./gradlew test"
GRADLE_ARGS=$(echo $CLASSNAMES | awk '{for (i=1; i<=NF; i++) print "--tests",$i}')
echo "Prepared arguments for Gradle: $GRADLE_ARGS"
./gradlew test $GRADLE_ARGS
- save_cache:
paths:
- ~/.gradle/wrapper
key: v1-gradle-wrapper-
- save_cache:
paths:
- ~/.gradle/caches
key: v1-gradle-cache-
- run:
name: Assemble JAR
command: |
if [ "$CIRCLE_NODE_INDEX" == 0 ]; then
./gradlew bootJar
fi
- run:
name: Create build folder on all nodes
command: |
mkdir -p ./build/libs
- persist_to_workspace:
root: build
paths:
- libs
workflows:
build-and-push-with-caches-and-parallel:
jobs:
- build-with-caches-and-parallel
- aws-ecr/build-and-push-image:
repo: ng-java-circle-test-backend
attach-workspace: true
workspace-root: ./build/libs
tag: ${CIRCLE_BRANCH}.${CIRCLE_SHA1}
executor: # <------- Extra Note
name: aws-ecr/default
use-docker-layer-caching: true
requires:
- build-with-caches-and-parallel
We copied this example from the official Java guide, and it has most of the steps described.
We have one extra note: The orbs that include jobs have an executor assignment. As shown in the example, it’s possible to change that if you require something else. In this case, we configured it to use the Docker Layer Caching feature.
Terraform simple workflow
version: 2.1
orbs:
terraform: circleci/terraform@3.1.0
common: # <------------ 1.
terraform-common: &terraform-common
path: terraform
tag: 1.2.8
workflows:
deploy_infrastructure:
jobs:
- terraform/validate:
<<: *terraform-common
checkout: true
- terraform/plan: # <------------ 2.
<<: *terraform-common
var_file: vars/test.tfvars
persist-workspace: true
requires:
- terraform/validate
- wait-for-apply-approval:
type: approval
filters: # <------------ 3.
branches:
only: main
requires:
- terraform/plan
- terraform/apply:
<<: *terraform-common
attach-workspace: true
filters:
branches:
only: main
requires:
- wait-for-apply-approval
We should automate infrastructure-related changes as well. Using Terraform, we can manage that via code. Here’s an example pipeline:
Key highlights:
- Example for YAML code reuse. The section is anchored, which can be referenced later.
- These are jobs from the Terraform orb. We added common steps to the config file, and extended them where needed. Here, Terraform uses the provided var file to fill up the defined variables. Once it has a plan, it persists to the workspace, ensuring the apply step executes what we approved.
- Branch filtering in action. We only deploy the infrastructure from the main branch; for every other branch, the last two steps aren’t visible.
Netguru best practices for CircleCI
CircleCI is the CI/CD solution we encourage developers to use at Netguru. Many engineering teams use it for internal and client projects.
We try to adhere to the following guidelines:
- Use a greater number of smaller jobs
- Identify where parallelism could help
- Make code reusable as much as possible
- Use orbs if appropriate
- Integrate Rollbar to track deployment status
- Integrate CodeClimate for test reports
CircleCI: the way forward
This guide will help you get started more easily with Java projects using the CircleCI Continuous Integration tool. We answered the question “what is CircleCI?” and outlined how it helps development teams create an efficient workflow for their daily tasks. If you’d like more information, feel free to get in touch.
