CircleCI 2.0 - Concepts' Explanation By Example

Updated Mar 4, 2024 • 12 min read

CircleCI is a widely used and extremely flexible continuous integration tool.

The new version, 2.0, was presented in July 2017. A support drop for the previous release has also been scheduled: 31 August 2018 is going to be the date. Until then, each project should be migrated to the new version. This won't be an easy task at all. This post is designed to help you out with this process by making it smoother and easier to tackle.

I will focus mainly on the new concepts introduced in CircleCI 2.0. Understanding them is crucial if you want to proceed any further. Buckle up!

CircleCI 2.0 Essentials

Migrating from 1.0

If you already have a CircleCI 1.0 config then you can head to the official config file translator and read through some official guides about the migration. The tool, unfortunately, is far from perfect so eventually, you will need to understand concepts behind the new release to conduct the migration properly. If you are either migrating or composing a file from scratch, take a look below. I will try to explain everything you might want to know when starting to work with CircleCI 2.0.

Basic Concepts

CircleCI 2.0 operates on a differently structurized YAML file located under ~/.circleci/config.yml. You can use all the tricks and syntax available in YAML files. It's safe to say that every config should contain at least three keywords: version, jobs, workflows. In short:

Version: a simple indicator of CircleCI version.
Jobs: a section that contains definitions of so-called jobs, which are independent tasks that can be run in parallel or sequentially.
Workflows: used to orchestrate job execution order, relations and their inner dependencies. This keyword allows many things, among which you can find: preparing different flows for different repository branches, running selected jobs based on outputs from previous ones in flow order.

You can always read more about basic concepts .

Now, let's take a closer look at the jobs section. It should contain a hash with job names as keys. Each job should have an executor (docker, machine, macos) and steps. A pretty self-explanatory example looks like this:

jobs:
 deploy_to_staging:
 working_directory: ~/project
 docker:
 - image: circleci/ruby:2.5.1-stretch-browsers
 - image: circleci/postgres:9.6.8-alpine-ram
 environment:
 POSTGRES_USER: postgres
 POSTGRES_DB: project_test
 POSTGRES_PASSWORD: ""
 steps:
 - checkout
 - attach_workspace:
 at: ~/project
 - run: cd deployment
 - run: bundle install
 - run: bundle exec cap staging deploy

Please, pay special attention to the executor. For most use cases, it would probably be docker, indicating that we will operate on predefined docker images. The first image on the list is the one in which each command will be executed by default. In the steps part, you can define as many run commands as you want, and they will be executed one by one.

While jobs are simple musical notes which live on their own, workflows are overtures which bring the notes together to compose something greater than the sum of each part independently. I will explain workflows by showing you an example.

workflows:
  version: 2
  build_test_deploy:
    jobs:
      - build_rails_dependencies
      - run_audits:
          requires:
            - build_rails_dependencies
      - run_unit_tests:
          requires:
            - build_rails_dependencies
      - deploy_to_staging:
          requires:
            - run_audits
            - run_unit_tests
          filters:
            branches:
              only: master

I think that if you read them out loud step by step, everything will become clear to you. We can define job precedence and their relations.

workflows

Above, you can see a simple workflow in action. If one action in the chain fails, the execution of the rest will stop. Later in the section, I will show a whole file structure example.

New tools

The 2.0 release introduced very interesting concepts that help to finish a build a lot faster than before. The main strengths are: data sharing and parallelism.

Data persistence

There are three terms associated with data persistence:

Workspace: it usually represents a directory with files that you want to share across build steps (jobs). This is basically the working directory of a build. You would usually want to start by checking out the repository code and attach it to a workspace with this simple job:
```
- checkout_code
  working_directory: ~/project
 docker:
    - image: circleci/node:8.2.1
  steps:
    - checkout
    - persist_to_workspace:
        root: ~/project
        paths:
          - .
```
Then, in other jobs, each directory that you want to leave for reuse and attach to the workspace needs to be attached by a command:
```
- persist_to_workspace:
 root: ~/project
    paths:
      - node_modules
```
As above, for example, npm install should output node_modules directory which might be reused later with a test or deploy command. More about workspaces here.
Cache: this one is extremely useful. Remember the times your build installed dependencies? Gems, packages? Anyone? Say no more, the build time spent on that has been cut down to the minimum now. Once the build fetches the needed data, it can be cached for any other build and reused in the job, as simple as that. The data state can be recognised by the SHA, so anytime anything changes in the specified file like yarn.lock, the job will store cache (which may take some time). In any other case, files will be loaded from cache and reused, which means no more waiting for redundant and repetitive tasks to finish. The syntax looks as follows:
```
- restore-cache:
 key: yarn-
- run: yarn install
- save-cache:
 key: yarn-
 paths:
 - ~/project/node_modules
```
As you may have realised, this tool can be very powerful. To get to know more about caching, look here. It's even more versatile than what I described above.
Artifacts, in simple terms, are the scraps, the output of the jobs. Artifacts can be virtually anything, but especially, coverage reports, screenshots, compressed applications (the deploy result of, for example, an Android project). They are available for upload to services (like Codebeat) or simple download via CircleCI artifact's tab or even an API with token authorization. They are stored on Amazon S3 cloud and the only limit is their size, which is 3GB. The basic command for storing artifacts is:
```
steps:
 - run:
 name: Creating Dummy Artifacts
 command: |
 echo "my artifact file" > /tmp/artifact-1;
 mkdir /tmp/artifacts;
 echo "my artifact files in a dir" > /tmp/artifacts/artifact-2;
 - store_artifacts:
 path: /tmp/artifact-1
 destination: artifact-file
 - store_artifacts:
 path: /tmp/artifacts
```

Parallelsim

I assume that there was a time when your test suites took a lot of time to finish. Now, it may be a matter of the past. It's possible to cut down the build time by running tests in parallel. We can let Circle determine the specs' example order and split them in packs to maximise the test suite efficiency. Typically, after one test run, the test results are saved and stored with relevant execution time. Next time, they will be split evenly by the number of processes. We can achieve this as follows:

- run:
 name: Run unit tests
 command: |
 TEST_DIRS="controllers,jobs,lib,mailers,models,policies,queries,resources,tasks,uploaders,validations,validators,workers"
 TEST_COMMAND="$(circleci tests glob "spec/{$TEST_DIRS}/**/*_spec.rb" | circleci tests split --split-by=timings)"
 bundle exec rspec --exclude-pattern "spec/features/**/*_spec.rb" --format RspecJunitFormatter --out test_results/rspec.xml --format progress $TEST_COMMAND
- run: bundle exec codeclimate-test-reporter
- store_test_results:
 path: test_results