End to End Tests on CircleCI with Docker - Rails, Capybara, Selenium
This solution, even though it has many pros, it brings in some difficulties as well. One of which are end to end tests.
It may be really tricky to execute those tests in a proper way, especially when we don't have that much of a control over the process on external hosts, for example during CircleCI build.
I decided to share with you what I've learned - how to set end to end tests on CircleCI. The instructions below can be easily applied to any framework combination other than Rails and React. However, article, in current form, refers to configuration like so:
-
two separate applications for backend (Rails) and frontend (React - the type doesn't really matter) on separate repos
-
tests in Rails are executed by RSpec with Capybara + Selenium + ChromeDriver
-
CircleCI configuration assume incorporating Docker and docker-compose
-
setup process was conducted on existing project with current CircleCI setup and dockerized app for staging and production
I'd like to fit into the form of a simple article, yet I'll try cover as many "What If's" and "Wait, what's" as possible, that may come up during the setup. Most of the lines in showed files that are essential to this setup are commented and explained.The big picture here is to allow writing integration tests (located in spec/features) on backend, test them locally without Docker and run during frontend app build on CircleCI.
Local tests - Backend
Environment setup
Make sure to add proper gems to Gemfile:
...
group :test do
...
gem "capybara", "~> 2.7", ">= 2.7.1"
gem "chromedriver-helper", "~> 1.0"
gem "rspec_junit_formatter" # Preparing proper output for CircleCI test metadata
gem "selenium-webdriver", "~> 2.53", ">= 2.53.4"
end
Then after bundle install let's add configuration for Capybara and Selenium in two files:
require "capybara/rails"
require "selenium-webdriver"
...
Capybara.register_driver :chrome do |app|
Capybara::Selenium::Driver.new(
app,
browser: :chrome,
desired_capabilities: { "chromeOptions" => { "args" => %w[window-size=1024,768] } },
)
end
Capybara.register_driver :selenium do |app|
Capybara::Selenium::Driver.new(app, browser: :chrome)
end
Capybara.configure do |config|
config.default_max_wait_time = 10
config.default_driver = :selenium
end
Capybara.app_host = "http://localhost:3000"
Capybara.javascript_driver = :chrome
Capybara.server_port = 5001 # We don't want it to collide with standard rails server on port 5000
Capybara.server_host = "0.0.0.0" # Start server on localhost as meta-address
Capybara.server = :puma, { Silent: true } # Supress puma STDOUT in console
...
require "capybara/rspec"
Since our tests are going to be placed in: spec/features our test suite on CircleCI is going to fail. We don't want to configure e2e tests on backend's CircleCI. This is why we need to override test command, in circle.yml:
machine:
services:
- redis
environment:
ES_JAVA_OPTS: "-Xms2g -Xmx2g"
_JAVA_OPTIONS: "-Xms1024m -Xmx2048m"
CONTINUOUS_INTEGRATION: true
dependencies:
post:
- if [[ ! -e elasticsearch-5.5.1 ]]; then wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.1.tar.gz && tar -xvf elasticsearch-5.5.1.tar.gz; fi
- elasticsearch-5.5.1/bin/elasticsearch: { background: true }
test:
post:
- bundle exec codeclimate-test-reporter
- bundle exec bundle-audit check --update
- bundle exec brakeman -f plain
override: # ADD this section
- bundle exec rspec --exclude-pattern "spec/features/**/*.rb" # We override Circle's test command here to explicitly exclude features specs
deployment:
master:
branch: master
commands:
- cd deployment; bundle install; bundle exec cap staging deploy
Local tests - Frontend
Environment setup
This one is simple - almost none setup needed.
The only thing you have to do is to point requests to specific backend address (in our case it's http://localhost:5001 as stated in rails_spec.rb). This is because we want the frontend app to use the same database as is used by rspec during tests - only in this case tests are viable.
This step depends on the technology you use in your project. It's safe to say that most of the javascript projects have package.json file. Usually we started local server by command:
$ yarn start
since, we don't wan't to mess up development environment we can add custom start-test command in package.json:
"scripts": {
"build": "react-scripts build",
...
"start": "react-scripts start",
"start-test": "REACT_APP_API_BASE_URL=http://localhost:5001 react-scripts start",
...
Our frontend app depends on the env variable REACT_APP_API_BASE_URL which is used to build reference links to rails API. With the above we can now use:
$ yarn start-test
In your project you must figure out how to set this up. Possibilities:
-
any other env variable consumed at some point by the app - which sets proper environment in which server is started ex.:
APP_ENV=testorNODE_ENV=test -
setup .env file with environment variable same as above
-
some kind of
if clausein API related service / component
Is it alive?
If you've set everything correctly at this point you are able to run e2e tests locally. Simple is it working? test case:
require 'rails_helper'
feature 'Home page', js: true do
scenario 'visit home page' do
visit '/'
expect(current_path).to eq '/'
expect(page.first('span').text).to eq("About Us")
end
end...
After we started frontend's local server like stated in previous section we are going to run specs manually just by:
$ rspec spec/features
So now let's move to the Crème de la crème - CircleCI setup!
CircleCI - Prerequisites
Let's start with identifying the files you will be working on:
Dockerfile - file used by docker which contains every information and command that is needed to build specific container by docker (containers are like little environments with their own dependencies, variables and configurations). Look at your Dockerfile now. It does not have any extension. IMPORTANT check if your current Dockerfile has two (or more) FROM instructions. If yes, then this is a problem in terms of CircleCI. See, this is called multi-stage build, which is great (it uses one image, takes something from it, adds something from other image and builds one image from which you set up your container. In single-stage you have to build separate images from the scratch - they take up a lot more space and time), but not for CircleCI. Multi-stage build requires Docker v17.05 and above (standard Docker on Circle 1.0 is much older). We can force that version, but only on CircleCI v2.0. So basically, if you have CircleCI v1.0 and multi-stage build Dockerfile - you have a problem to solve. Dockerfile reference
circle.yml - if you have file named like this then you use CircleCi v1.0. This version is way easier to setup. For v2.0 there is .circleci/config.yml file and it's structure is different. If you want you can migrate from 1.0 to 2.0 following steps in: Migrating from 1 to 2 but I personally don't recommend unless your really know what you are doing. However, using 2.0 have many pros - newer version of Docker and docker-compose which support commands like exec which might be useful. Circle.yml reference
docker-compose.yml - file used by, surprisingly, docker-compose tool. You can think of this file as of list of containers (specified by reference to Dockerfile or by image from Docker Hub) with respective config. This is the most important file in which you have to place every part of setup you need: rails, front, postgres, redis, etc, etc. Docker-compose.yml reference.
CircleCI - Backend
Setup Environment
In my opinion best approach here is to add specific environment to rails app, tailored specifically for end-to-end tests on CircleCI. I picked name: e2e. It's just convenient, it can be virtually any name. So to start:
-
Duplicate
config/environments/test.rbasconfig/environments/e2e.rb -
In
Gemfileto each group specified for:testadd:e2eas well in the group definition. -
If you have secrets then you should make a namespace for
e2einconfig/secrets.yml(copy definitions from development) and add propersecret_key_basein encrypted keys (in our project we edit secrets byEDITOR=nano rails secrets:edit) -
Remember to have in mind that in places where you use
Rails.env.test?or something similar to setup / check anything you might want to specify how to behave in case ofe2eenvironment as well -
If you use any kind of requests mocking / suppressing / mimicking (like webmock or vcr) you will need to whitelist containers names in virtual network for example
frontend,elasticsearch,postgres,redisetc.
Create new file config/database.e2e.yml:
e2e:
adapter: postgresql
host: postgres
encoding: unicode
database: your_project_test
pool: 5
username: postgres
password: your_password
Create Dockerfile.e2e(or duplicate other Dockerfile you have in project). Our is placed in docker/Dockerfile.e2e. Your Dockerfile may look like the one below. You can also create one from scratch using Dockerfile reference.
FROM quay.io/netguru/baseimage:0.10.1 # Image containing all dependecies like imagemagick etc
ENV RUBY_VERSION 2.4.2 # To get proper ruby
## Install Ruby & Dependencies
RUN \ # Install ruby
apt-get update -q && \
apt-get install -q libcurl3 && \
apt-get install -q -y cron && \
ruby-install --system --cleanup ruby $RUBY_VERSION -- --disable-install-rdoc && \
gem install bundler
## Copy Gemfile & bundle
ADD Gemfile* $APP_HOME/ # Gemfile installation
RUN bundle install --jobs=8 --retry=3
## Add rest of code
ADD . $APP_HOME/ # Copy to APP_HOME folder
ENV RAILS_ENV e2e # Set env to e2e
ENV WEB_CONCURRENCY 2 # Puma concurrency
ENV AVAILABLE_MEMORY 1200 # How much memory the container can use
ENV RAILS_SERVE_STATIC_FILES true # Serve static assets
ENV REDIS_URL redis://redis:6379/0 # Set Redis url and port
ADD ./config/database.e2e.yml /app/config/database.yml # Copy database config
RUN mkdir /app/rspec_output # Create directory for rspec results
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* # Cleaning up
# Elasticsearch
ENV ELASTICSEARCH_URL=http://elasticsearch:9200 # Our case - setup ElasticSearch url and port
EXPOSE 5001 # Container will listen on this port
Add e2e specific configuration to rails_helper.rb locally, ENV is 'test', while on CircleCI it's 'e2e'. We don't want to start test server automatically. It will be started in a different way:
Capybara.register_driver :chrome do |app|
Capybara::Selenium::Driver.new(
app,
browser: :chrome,
desired_capabilities: { "chromeOptions" => { "args" => %w[window-size=1024,768] } },
)
end
if Rails.env.test?
Capybara.register_driver :selenium do |app|
Capybara::Selenium::Driver.new(app, browser: :chrome)
end
Capybara.configure do |config|
config.default_max_wait_time = 10
config.default_driver = :selenium
end
Capybara.app_host = "http://localhost:3000"
Capybara.javascript_driver = :chrome
Capybara.server_port = 5001
Capybara.server_host = "0.0.0.0"
Capybara.server = :puma, { Silent: true }
else
args = ["--no-default-browser-check", "--start-maximized"]
caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => { "args" => args })
Capybara.register_driver :selenium do |app|
Capybara::Selenium::Driver.new(
app,
browser: :remote,
url: "http://selenium:4444/wd/hub",
desired_capabilities: caps,
)
end
Capybara.configure do |config|
config.default_max_wait_time = 15
config.default_driver = :selenium
end
Capybara.app_host = "http://frontend:3000" # Containers communicate by aliases, hence 'frontend'
Capybara.javascript_driver = :selenium
Capybara.run_server = false # To ensure everything is in sync don't start puma automatically
end
It's very good idea to keep all commands you need to start specs within one script. We can create one in docker/e2e.sh. This bash script will be started in container by a docker-compose.
#!/bin/bash
echo "Setting up and running e2e tests!"
# Create database and seed (since we might want to include some data needed for our app to work properly like oauth clients
rails db:setup
# Start puma server in e2e environment on meta-address host on port 5001 in detached mode
RAILS_ENV=e2e puma -b tcp://0.0.0.0:5001 -d
# Start feature specs and output results to xml file
rspec --format progress --format RspecJunitFormatter --out ./rspec_output/rspec.xml spec/features
That's it! We're done with backend.
CircleCI - Frontend
Setup Environment
Create new docker-compose.ci.yml file in project's main directory or just duplicate and rename current one for ex. docker-compose-staging.yml . You will need to setup few things: proper dependencies for backend, volumes, ports, aliases etc. Everything is explained below. In the end this file should more or less look like this:
version: '2' # Version of composer
services:
backend: # Backend container
build:
context: ~/backend # Backend should be pulled by git to this location
dockerfile: docker/Dockerfile.e2e # The location of Dockerfile, relative to context above
depends_on: # The containers below will be resolved and loaded before backed
- redis
- postgres
- frontend
- selenium
- elasticsearch
ports:
- "5001:5001" # Expose host_port:container_port
volumes:
- "/rspec_output:/app/rspec_output" # Mount /rspec_output (on host) to /app/rspec_output (in container)
environment:
- RAILS_ENV=e2e # This env var will be available to any container instance - used to run rspec in e2e env
- RAILS_MASTER_KEY # If no value is passed this var is going to be read from machine ENV - very useful. Just set RAILS_MASTER_KEY for secrets in CircleCI build config page
networks: # Virtual network for containers
main: # Name of the network
aliases:
- backend # Alias for reference purposes
command: bash -c "bash /app/docker/e2e.sh" # Start container by a bash script (container will be alive as long as this script is running)
frontend: # Frontend container
build:
context: .
dockerfile: Dockerfile.e2e
command: nginx
logging: # Disable spamming irrelevant messages from this container
driver: none
networks:
main:
aliases:
- frontend # Container name in virtual network that can be addressed
ports:
- "3000:3000"
volumes: # Used to mount ./tmp on machine to /app/dist in container
- "./tmp:/app/dist"
selenium: # Selenium container
image: selenium/standalone-chrome
ports:
- "4444:4444"
logging: # Disable spamming irrelevant messages from this container
driver: none
networks:
main:
aliases:
- selenium # Container name in virtual network that can be addressed
elasticsearch: # Elasticsearch container (not required) - we used it in project
image: docker.elastic.co/elasticsearch/elasticsearch:5.5.1
volumes:
- "~/es_data:/usr/share/elasticsearch/data"
environment:
- "xpack.security.enabled=false"
- "transport.host=localhost" # This line might be needed to disable errors with circleCI 1.0 container memory amount
- "bootstrap.system_call_filter=false" # This line might be needed to disable errors with circleCI 1.0 container memory amount
ports:
- "9300:9300"
logging: # Disable spamming irrelevant messages from this container
driver: none
networks:
main:
aliases:
- elasticsearch # Container name in virtual network that can be addressed
postgres: # Postgres container
image: postgres:9.5 # Postgres version, installed from image on Docker Hub
environment:
- POSTGRES_PASSWORD=password # There can't be 'blank' password, use one specified in database.e2e.yml
logging: # Disable spamming irrelevant messages from this container
driver: none
networks:
main:
aliases:
- postgres # Container name in virtual network that can be addressed
redis: # Redis container
image: redis:4.0.6 # Redis version, installed from image on Docker Hub
logging: # Disable spamming irrelevant messages from this container
driver: none
networks:
main:
aliases:
- redis # Container name in virtual network that can be addressed
networks: # Set virtual network up
main:
If you need another service (in container) just add it to the list in a similar way, watch out for the order.
Now to the frontend Dockerfile. Same as before - copy current Dockerfile or create one in project's root. Unfortunately we had multi-stage build Dockerfile and CircleCI v1.0 so I had to convert multi-stage to single-stage. Let me use an example. At first we had this file:
## specify node version
FROM quay.io/netguru/ng-node:6 as builder # First 'FROM', as builder is used as reference below
## add necessary environments
ENV NODE_ENV staging # Set env var
## add code & build app
ADD . $APP_HOME # Copy / add external source to image's filesystem
RUN yarn install # Install all dependencies for frontend app
RUN yarn build # Compile frontend build
## Real app image
FROM nginx:alpine as app # Second 'FROM', now it's multi-stage build file
## Copy build/ folder to new image
COPY --from=builder /app/build /app/dist # Copy file from one image to another by reference - we will have to get rid of it
COPY nginx.conf /etc/nginx/nginx.conf # Copy nginx config to nginx directory
EXPOSE 3000 # Container will listen on this port at runtime
Now it have to be split up in two files:
## Real app image
FROM nginx:alpine # Remove 'as app'
COPY nginx.conf /etc/nginx/nginx.conf
EXPOSE 3000
## specify node version
FROM quay.io/netguru/ng-node:6 # Remove 'as builder'
## add necessary environments
ENV NODE_ENV e2e # Set env
ENV REACT_APP_API_BASE_URL http://backend:5001 # Set env var to point to backend within build. It can be also done by specific command in package.json
## add code & build app
ADD . $APP_HOME
RUN yarn install # Install dependencies
RUN yarn build
If you look carefully you see that I deleted COPY --from=builder /app/build /app/dist - this will be done in a different way.
The above change will force two new steps. In circle.yml we have to add in dependencies/pre after pyenv rehash:
dependecies:
pre:
...
- pyenv rehash
- docker build -t frontend_img -f Dockerfile.e2e.build . # Builds image from Dockerfile.e2e.build (with our front app)
- docker create --name frontend_pre frontend_img # Creates container from image
- docker cp frontend_pre:/app/build ./tmp # Copies /app/build directory (compiled app) from a container to ./tmp in CircleCI machine
...
In case your config is:
multi-stage+v1.0- convert multi-stage to single-stagemulti-stage+v2.0- make sure that you force proper Docker version andEdgebuild of CircleCIsingle-stage+v2.0- do nothing, just use thatDockerfilesingle-stage+v1.0- do nothing, just use thatDockerfile
The last step is to configure circle.yml and setup precedence of commands and indicate usage of Docker. In the end circle.yml should look like:
machine:
node:
version: 8.2.1
pre:
- mkdir ~/.cache/yarn
- curl -sSL https://s3.amazonaws.com/circle-downloads/install-circleci-docker.sh | bash -s -- 1.10.0 # Download Docker 1.10.0 (linked to docker-compose v2) for CircleCI
- >-
GIT_SSH_COMMAND='ssh -i ~/.ssh/backend-e2e-tests-deploy-key'
git clone git@github.com:netguru/project.git ~/backend && cd ~/backend # Download backend to ~/backend. The SSH for backend had to be placed in CircleCI build setup page
- set -o allexport; source "${HOME}/${CIRCLE_PROJECT_REPONAME}/.env"; set +o allexport
environment:
PATH: "${PATH}:${HOME}/${CIRCLE_PROJECT_REPONAME}/node_modules/.bin"
services:
- docker # Enable Docker for this build
dependencies:
pre:
- sudo mkdir /rspec_output && mkdir $CIRCLE_TEST_REPORTS/rspec && mkdir ~/es_data # make directories for mounting volumes and directory for test metadata. es_data is related to elasticsearch container
- pip install docker-compose # Update docker-compose
- pyenv rehash # Refresh docker-compose link
- docker build -t frontend_img -f Dockerfile.e2e.build . # Build first part of former multi-stage Dockerfile
- docker create --name frontend_pre frontend_img # Create container from image above
- docker cp frontend_pre:/app/build ./tmp # Copy compiled frontend app build from container to machine ./tmp dir
- docker-compose -f docker-compose.ci.yml build # Build all containers in docker-compose.ci.yml
cache_directories:
- ~/.cache/yarn
override:
- yarn install
test:
override:
- docker-compose -f docker-compose.ci.yml up --exit-code-from backend # Start containers from docker-compose. This will run in foreground and output exit code of bash script
- ? case $CIRCLE_NODE_INDEX in # Standard frontend tests
0) yarn test --runInBand ;;
1) yarn eslint && yarn stylelint ;; esac
: parallel: true
- cp /rspec_output/rspec.xml $CIRCLE_TEST_REPORTS/rspec/rspec.xml # Copy rspec results to CircleCI test metadata folder
deployment:
master:
branch: master
commands:
- cd deployment; bundle install; bundle exec cap staging deploy
This is it, at this point everything should be safe and sound. If not and your build is failing ssh to CircleCI and play around.
Useful commands
$ docker ps # show running containers
$ docker ps -a # show all containers
$ docker ps -n=-1 # show n last created containers
$ docker start # start container
$ docker stop # stop container
$ docker cp : # copy file/directory from container to destinated dir in machine
$ docker cp : # copy file/directory from machine to container (don't get deceived, read 'understanding volumes' article)
$ docker images # list all images
$ docker rm -f # remove container
$ docker rmi # remove image
$ docker-compose run # run a command in container specified in docker-compose file. Watch out - this command creates new containers each time. Moreover it doesnt create a network!
$ docker-compose exec # execute a command in container that is already running (unavailable in older versions of Docker (CircleCI v1.0))
$ docker-compose up # start containers from docker-compose file by specified commands (creates networks). Containers are running as long as commands which run them
Useful links
article about multi-stage build
another article about multi-stage build
yet another article about multi-stage build
understanding volumes in containers
FAQ
Is it hard to setup?
It may be. Everything depends on CircleCi version you have, which limits interactions with Docker (old versions) and how many dependent containers you need for your backend. Some of them might be really tricky to setup correctly.
Should I consider migrating to CircleCI v2.0?
Definitely! This build takes much time to execute. With newer version of Circle you might speed it up significantly and use new Docker (less bugs, some commands actually work, not just fail).
What if I have CircleCI v2.0 file?
Then you should be happy. Everything you see above is still valid, only that you don't have to worry about multi-staging. Just follow this guide Multi-stage docker build with with CircleCI v2.0 and rewrite our circle.yml additional lines using 2.0 style CircleCI v2.0 reference
Summary
I hope that I've managed to describe everything to enable you to set end to end tests on your own in your project.
Please feel free to ask if anything is unclear. Maybe article is unclear or needs improvements? Point it out as well. I believe the best way we learn is by a mistake, we can both benefit from it.
Later, I'm planning to prepare a follow-up for CircleCI v2.0, so stay tuned!
