Optimizing Reporting in Automated Testing: Playwright, CircleCI, and Allure Reporter on AWS

How to Optimize Automated Testing?

With Playwright at your disposal, you can replicate how real users interact with your application, making sure everything works perfectly across different browsers and devices. Combine this with CircleCI, and you've got yourself a robust setup that pushes updates automatically, keeping your deployment game strong and steady.

Then there’s Allure Reporter, which turns the nuts and bolts of test data into insightful, easy-to-understand reports. This kind of detailed reporting is essential, not just for catching bugs early but also for understanding how your tests perform over time. It’s all about getting the right information to the right people quickly, so decisions can be made swiftly and confidently.

We’ll also cover how to host your Allure test reports on AWS as static web pages. This way, they’re always there, easy to access for everyone in the team, from developers to project managers. This isn’t just about automating your processes—it’s about making them more transparent and connected, which is crucial in a fast-paced development environment.

Ready to see how all these pieces fit together to help you run your projects more smoothly? Let's get started on upgrading your development process.

First, you will need a project in Playwright and a created account on CircleCi.

• If you are not currently working on any project where you want to implement CI/CD with Allure, simply clone the GitHub repository which contains basic playwright setup boilerplate

After cloning the project, in the root directory run the npm install command

• Create a free account on CircleCi https://circleci.com/vcs-authorize/

If you already have the Playwright project configured, let's start by creating two very simple tests that will serve as input to our project (they can be any tests, but it is worth making them very short, which will improve the configuration process).

For demonstration purposes, we will use saucedemo.com

In your playwright project, in the /tests/ path, create a file that will contain the tests and name it e.g. login.spec.ts

Then create two basic tests in it:

import { test, expect } from "@playwright/test";

test("has title", async ({ page }) => {
   await page.goto("https://saucedemo.com");
   await expect(page).toHaveTitle(/Swag/);
});

test("get started link", async ({ page }) => {
   await page.goto("https://saucedemo.com");
   await page.locator("#user-name").type("standard_user");
   await page.locator("#password").type("secret_sauce");
   await page.locator("#login-button").click();
   await expect(page.locator("#shopping_cart_container")).toBeVisible();
});

In the terminal, type npx playwright test to check if tests run correctly (to have clean test results you can remove the tests from the test-examples/ location)

Once you have added the tests, it's time to install Allure Reporter.

In the terminal, enter the command: npm install --save-dev allure-playwright

Then we will streamline the process a bit and add a script command in package.json that will run our tests and return the result in the standard format and the allure format.

Open the package.json file and in the "scripts" section add:

"scripts": {
       "test": "playwright test --reporter=line,allure-playwright"
   },

From now we will run our tests from the terminal using the npm test command.

In your terminal run the npm test and check if it works.

Then we need to install the Allure command line in the project to be able to perform operations using Allure.

To install the Allure command line in the terminal type: npm install -g allure-commandline

To generate the allure report, type allure generate --clean

Once the report has been generated, you can view it using Allure Open which will open a browser with the locally hosted report.

Good job, we have already set up the project in Playwright, written basic tests, and added the Allure reporter.

Now add CI/CD to our project.

In your project, create a folder called .circleci
Inside this folder, create a config.yml file

For now, CircleCi doesn't know where the configuration file is or what it contains, so your project is not yet integrated with CircleCi.

To integrate your project with CircleCi, run the following commands in your terminal:

git add .

git commit -m “your commit message”
git push --set-upstream origin <your_branch_name> # (main branch name by default)

Now it's time to configure CircleCi

If you have logged in to CircleCi using GitHub, your repositories will immediately be visible as projects.

If you have chosen a different form of account creation, here you will find instructions on how to add a project in CircleCi https://circleci.com/docs/getting-started/.

After creating a project you will see your project in the Projects tab

Click “Set Up Project” -> Select your project -> In the “From which branch” field provide your branch name

If everything is set correctly you should see:

Under the “From which branch” field.

Now click “Set Up Project” and you’re good. The playwright-circleci configuration is done.

Now it's time to add some instructions to our config.yml file so that our CI/CD pipeline starts automatically after changes are made to our branch.

Open the config.yml file and type:

version: 2.1
executors:
 docker-executor:
   docker:
     - image: mcr.microsoft.com/playwright:v1.43.0-jammy

This configuration sets up a CircleCI job environment that uses a specific Docker image tailored for Playwright testing, ensuring that all necessary dependencies and frameworks are pre-installed in the environment where the CI jobs will run.

For our config.yml file to be able to instruct the pipeline, we need to somehow access our code. The first important step in the config.yml file will be checkout - downloading the source code. To use the checkout command, you must have an SSH key, which you will place in your CircleCi project.

To the config.yml file add:

jobs:
 build:
   executor: docker-executor
   steps:
     - checkout
     - add_ssh_keys:
         fingerprints:
           - "your-fingerPrint-here"

Explanation:

• The checkout step is a built-in CircleCI step that checks out your code from the version control system (like GitHub) you have linked with your CircleCI account. This is the first step in most jobs, ensuring that the latest code is used for subsequent steps.

• • The add_ssh_keys step is used to attach SSH keys stored in CircleCI to your job. This step is crucial when your pipeline needs to authenticate with external systems, such as deploying code to a server or accessing private dependencies.
  • fingerprints: This key allows you to specify which SSH keys to add to the build, based on their fingerprints. The fingerprint is a unique identifier for each SSH key, ensuring that the correct key is used.

How to add an SSH Key?

Creating and Using SSH Keys with CircleCI

Step-by-Step Guide

• First, you need to generate an SSH key pair. This can be done on your local machine using the following command:
  ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
• When prompted, provide a file path to save the key, or press enter to use the default location (~/.ssh/id_rsa).
• Set a passphrase for the key when prompted or leave it blank for no passphrase.

2. Add the Public Key to the Server

• Copy the public key (id_rsa.pub) to your server or the service that requires SSH access. The method to do this can vary based on the server or service provider. Generally, you can add the public key to the ~/.ssh/authorized_keys file on the server.

• Log in to CircleCI and select your project.
• Go to Project Settings > SSH Keys.
• Click on "Add SSH Key".
• Paste the private key (contents of id_rsa) into the text box.
• Optionally, specify the hostname for which the key will be used if it's not a generic key.
• Click "Add SSH Key" to save it.

• To use the SSH key in your pipeline, you must reference it by its fingerprint in the config.yml file.

Now, using the "run" commands, we instruct the pipeline to install project dependencies on the machine and run tests, i.e. our npm test command

- run:
         name: Install Dependencies
         command: npm install
     - run:
         name: Run Playwright Tests
         command: npm test

This workflow configuration indicates that when the workflow is triggered (usually by a push to the repository unless configured otherwise), it will execute the build job. This is a fundamental setup in CircleCI, allowing you to automate parts of your development process.

Now just use again:

git add .

git commit -m “your commit message”

git push

And go to your CircleCi pipeline. The build should be triggered and passed.

Ok, the basic configuration is done. You are able now to trigger your tests in the CI/CD pipeline on push to your branch.

Then, after completing the test run on CircleCi, we will want to generate an allure report and host it on AWS as a static website, so that everyone, even people without an AWS account, can access it.

This process will consist of two stages:

1. Creating an AWS account and configuring the S3 bucket
2. Adding instructions to config.yml

Let’s start with the AWS part.

First, let’s start by creating an account on AWS.

Create your account on the eu-central-1 region (this can be changed later).

Here you will find a very handy tutorial on how to do it Creating an AWS account by Travis Media . Create your account, and follow Travis also with creating an admin account. You should never perform the type of operations we will do from the root account on AWS. Create and always use an Administrator account for all tasks. More details can be found in Travis' video.

Everything we will do in this guideline will not require you to pay any fees on AWS.

Nevertheless, when using an AWS account, I recommend setting a budget setup, which will not allow you to exceed certain costs and will trigger an email alert in the event of a potential cost overrun.

You can find out more about this in this short but excellent video by Stephan Marek.

Once your AWS account is ready, we can start creating and configuring S3, where we will host our test report.

Create S3 bucket. In the AWS console, follow these steps:

1. Search for the S3 service
2. Open the S3 service
3. Press the "Create Bucket" button
4. Add your bucket name (the name should be unique)
5. Set ACLs enabled
6. Untick "Block all public access"
7. Tick the "Turning off block all public access" warning
8. Turn on bucket versioning
9. Click Create bucket

Now go to your buckets and open your recently created bucket. Your bucket is empty now.

Inside your bucket, go to Properties and click Edit

Select “Enable” Static Website Hosting and in the Index document field type “index.html”

Now inside your bucket go to Permissions, and in the Policy tab add the following policy, and click “save changes”.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "PublicReadGetObject",
           "Effect": "Allow",
           "Principal": "*",
           "Action": "s3:GetObject",
           "Resource": "arn:aws:s3:::/*"
       },
       {
           "Sid": "PublicListBucket",
           "Effect": "Allow",
           "Principal": "*",
           "Action": "s3:ListBucket",
           "Resource": "arn:aws:s3:::"
       }
   ]
}

Your bucket should be ready.

Now it's time to edit our config.yml file so that the tests that run send the result files to AWS S3.

Before we start performing various operations on AWS from our config.yml, we need to add the secret key and id from AWS to CircleCi. To do this follow these steps:

1. Open the IAM (Identity and Access Management) Service:
   • Navigate to the Services menu at the top of the AWS console and select "IAM" located under "Security, Identity, & Compliance."
2. Find and select your user:
   • In the IAM dashboard, click on "Users" in the left navigation pane.
   • Search for and click on the user name to whom you want to add the access keys.
3. Manage the user’s access keys:
   • Once you're on the user's summary page, find the "Security credentials" tab.
   • Scroll to the "Access keys" section.
4. Create a new access key:
   • Click the “Create access key” button.
   • A new access key ID and secret access key will be generated.
   • AWS will offer you the option to download the key pair as a .csv file. It’s important to download and save this file because you won’t be able to access the secret access key again once you leave this dialog.
5. Securely store the new keys:
   • Keep the downloaded .csv file in a secure location, or securely note down the access key ID and secret access key if you prefer not to download them.
   • It's crucial to handle these credentials carefully to avoid unauthorized access to your AWS resources
6. Configure CircleCI with your AWS credentials:
   • In your CircleCI project settings, go to "Environment Variables."
   • Add the AWS Access Key ID and Secret Access Key as environment variables. Typically, you might name them AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Now that you have created the secret key and id and added them to the CircleCi, let's go to our config.yml file and start editing it.

After lines, we have already created

- run:
         name: Install Dependencies
         command: npm install

Add

- run:
         name: Install Dependencies
         command: npm install
     - run:
         name: Install Allure Commandline
         command: npm install -g allure-commandline
     - run:
         name: Install Java
         command: |
           apt-get update
           apt-get install -y openjdk-11-jdk
     - run:
         name: Set JAVA_HOME
         command: echo 'export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"' >> $BASH_ENV
     - run:
         name: Update PATH
         command: echo 'export PATH="$JAVA_HOME/bin:$PATH"' >> $BASH_ENV
     - run:
         name: Install AWS CLI
         command: |
           export DEBIAN_FRONTEND=noninteractive
           ln -fs /usr/share/zoneinfo/Etc/UTC /etc/localtime
           apt-get update
           apt-get install -y awscli tzdata
           dpkg-reconfigure --frontend noninteractive tzdata
     - run:
         name: Configure AWS CLI
         command: |
           aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
           aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
           aws configure set default.region eu-central-1
     - run:
         name: Define Timestamp
         command: echo 'export TIMESTAMP=$(date +%Y%m%d%H%M%S)' >> $BASH_ENV
     - run:
         name: Create Allure Results Directory
         command: mkdir -p allure-results
     - run:
         name: Download Allure History from S3
         command: |
           aws s3 cp s3:///allure-reports/latest/history ./allure-results/history --recursive

And then after lines with run test add:

- run:
         name: Run Playwright Tests
         command: npm test
     - run:
         name: Generate Allure Report with History
         command: |
           allure generate allure-results --clean
     - run:
         name: Upload New Allure Report to S3 with Timestamp
         command: |
           UNIQUE_REPORT_DIR="build-$(echo $CIRCLE_BUILD_NUM)-${TIMESTAMP}"
           aws s3 sync ./allure-report s3:///allure-reports/$UNIQUE_REPORT_DIR --acl public-read
     - run:
         name: Update Latest Report Reference with History
         command: |
           aws s3 rm s3:///allure-reports/latest --recursive
           aws s3 cp --recursive ./allure-report s3:///allure-reports/latest/
           aws s3 cp --recursive ./allure-report/history s3:///allure-reports/latest/history/
     - run:
         name: Display Allure Report URL with Timestamp
         command: echo "Allure report is available at https://.s3.eu-central-1.amazonaws.com/allure-reports/${UNIQUE_REPORT_DIR}/index.html"

In the file below I will include a few commands that may seem complicated at first glance. However, I will try to describe them in the comments.
I would like to draw your attention to the fact that Allure has a built-in function of comparing reports to previous - historical ones.

Some commands in the file below that are related to report history are not necessary to run and host Allure reports on AWS but may be useful when we want to approach reporting reliably. History commands are also a great resource for tracking results and artifact history within the S3 bucket itself.

Your complete file now should be:

version: 2.1
executors:
 docker-executor:
   docker:
     - image: mcr.microsoft.com/playwright:v1.43.0-jammy

jobs:
 build:
   executor: docker-executor
   steps:
     - checkout
     - add_ssh_keys:
         fingerprints:
           - "yourFingerPrint"
     - run:
         name: Install Dependencies
         command: npm install
     - run:
         name: Install Allure Commandline
         command: npm install -g allure-commandline
     - run:
         name: Install Java # Installs Java Development Kit, which is needed to run Java applications.
         command: |
           apt-get update  # Updates the list of available packages and their versions.
           apt-get install -y openjdk-11-jdk
     - run:
         name: Set JAVA_HOME # Sets an environment variable JAVA_HOME that specifies the location of Java.
         command: echo 'export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"' >> $BASH_ENV
     - run:
         name: Update PATH # Updates the system PATH so the shell can find the Java binaries.
         command: echo 'export PATH="$JAVA_HOME/bin:$PATH"' >> $BASH_ENV
     - run:
         name: Install AWS CLI
         command: |
           export DEBIAN_FRONTEND=noninteractive  # Prevents the command line from asking for input.
           ln -fs /usr/share/zoneinfo/Etc/UTC /etc/localtime  # Sets the server's timezone.
           apt-get update  # Refreshes the repository index.
           apt-get install -y awscli tzdata  # Installs the AWS command line interface and timezone data.
           dpkg-reconfigure --frontend noninteractive tzdata  # Reconfigures installed packages.
     - run:
         name: Configure AWS CLI
         command: |
           aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID 
           aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY 
           aws configure set default.region eu-central-1
     - run:
         name: Define Timestamp # Creates a timestamp for when the job runs, useful for naming files or directories uniquely.
         command: echo 'export TIMESTAMP=$(date +%Y%m%d%H%M%S)' >> $BASH_ENV
     - run:
         name: Create Allure Results Directory
         command: mkdir -p allure-results # Creates a directory to store Allure test results.
     - run:
         name: Download Allure History from S3 # Copies the Allure test history from an S3 bucket to the local machine.
         command: |
           aws s3 cp s3:///allure-reports/latest/history ./allure-results/history --recursive
     - run:
         name: Run Playwright Tests
         command: npm test
     - run:
         name: Generate Allure Report with History # Generates a visual report, with the clean option removing old data first.
         command: |
           allure generate allure-results --clean
     - run:
         name: Upload New Allure Report to S3 with Timestamp # Creates a unique directory name for this build's report. / # Syncs the new report to your S3 bucket, making it publicly accessible.
         command: |
           UNIQUE_REPORT_DIR="build-$(echo $CIRCLE_BUILD_NUM)-${TIMESTAMP}" 
           aws s3 sync ./allure-report s3:///allure-reports/$UNIQUE_REPORT_DIR --acl public-read
     - run:
         name: Update Latest Report Reference with History # Deletes the current 'latest' directory in your S3 bucket. / # Copies the new report and history data to the 'latest' directory. /
         command: |
           aws s3 rm s3:///allure-reports/latest --recursive 
           aws s3 cp --recursive ./allure-report s3:///allure-reports/latest/ 
           aws s3 cp --recursive ./allure-report/history s3:///allure-reports/latest/history/
     - run:
         name: Display Allure Report URL with Timestamp # Prints the URL where the report can be accessed.
         command: echo "Allure report is available at https://.s3.eu-central-1.amazonaws.com/allure-reports/${UNIQUE_REPORT_DIR}/index.html" > allure-report-url.txt
     - store_artifacts:
         path: allure-report-url.txt
         destination: allure-report-url

workflows:
 version: 2
 build_and_deploy:
   jobs:
     - build

Now everything should work and you should see:

Report URL hosted on AWS in the Artifacts section in the CircleCi job

History of reports on AWS

https://.s3.eu-central-1.amazonaws.com/allure-reports/latest/index.html url.

You may have noticed that the pipeline in CircleCi does a lot of repetitive things with each test run. This process is neither efficient nor effective.

That’s why we can go a step further and try to dockerize our environment in a basic way.

What do you need?

• Docker installed, registered, and logged in account on dockerhub.
• Added an access token from Docker in CircleCi environment variables

Explanation:

Generate a Docker Access Token

1. Log in to Docker Hub: Go to Docker Hub at hub.docker.com and log in with your credentials.
2. Access Security Settings: Navigate to your profile settings by clicking on your username, then go to "Security".
3. Create a New Token: Click on "New Access Token". Provide a descriptive name for the token to indicate where it will be used (e.g., "circleci-deployment").
4. Copy the Token: Once the token is generated, make sure to copy it immediately, as you won't be able to see it again after closing the window.

Add Docker Token to CircleCI

1. Log in to CircleCI
2. Project Settings: Navigate to the settings page of the project for which you want to use the Docker token.
3. Environment Variables: Find the "Environment Variables" section in the settings menu.
4. Add New Variable: Click on "Add Variable". Enter a name for your variable (DOCKER_LOGIN - login, and DOCKER_PASSWORD - token) and paste the Docker token you copied earlier.

If you have met these requirements, you can proceed to the project.

In the root create a Dockerfile file and put the below content into it

FROM node:16-buster

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y openjdk-11-jdk

ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64
ENV PATH $JAVA_HOME/bin:$PATH

RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
   && unzip awscliv2.zip \
   && ./aws/install

RUN npm install -g allure-commandline

RUN npm install -g playwright@1.43.0

RUN apt-get install -y libgbm1 libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdbus-1-3 libxrandr2 \
   libxshmfence1 libxcomposite1 libxdamage1 libxkbcommon0 libwayland-client0 libpango-1.0-0 libpangocairo-1.0-0 \
   libx11-xcb1 libx11-6 libxcb1 libxfixes3 libxext6 libxcb-dri3-0 libdrm2 libgbm1 libasound2


RUN npx playwright install

WORKDIR /app

COPY . /app

As you may have noticed, some of the commands that were in config.yml are now in the Dockerfile. What does it mean? Based on the Dockerfile, we will create an image that we will send to the Dockerhub "remote image store" and from now on, each of our test runs will use this image as a base to create the environment and it will not be necessary to install all the tools and dependencies each time.

Once you've created your Dockerfile, you need to build the image and send it to Dockerhub's "remote image store".

Use commands:

docker build -t /: . - to build your image

docker push /: - to push your image

Now you need to use your image in the config.yml file as the base for your job, and the remaining commands that were not included in the Dockerfile remain unchanged.

version: 2.1
executors:
 docker-executor:
   docker:
     - image: /:
       auth:
         username: $DOCKER_LOGIN
         password: $DOCKER_PASSWORD

jobs:
 build:
   executor: docker-executor
   steps:
     - checkout
     - add_ssh_keys:
         fingerprints:
           - "yourFingerPrint"
     - run:
         name: Define Timestamp
         command: echo 'export TIMESTAMP=$(date +%Y%m%d%H%M%S)' >> $BASH_ENV
     - run:
         name: Create Allure Results Directory
         command: mkdir -p allure-results
     - run:
         name: Download Allure History from S3
         command: |
           aws s3 cp s3:///allure-reports/latest/history ./allure-results/history --recursive
     - run:
         name: Run Playwright Tests
         command: npm test
     - run:
         name: Generate Allure Report with History
         command: |
           allure generate allure-results --clean
     - run:
         name: Upload New Allure Report to S3 with Timestamp
         command: |
           UNIQUE_REPORT_DIR="build-$(echo $CIRCLE_BUILD_NUM)-${TIMESTAMP}"
           aws s3 sync ./allure-report s3:///allure-reports/$UNIQUE_REPORT_DIR --acl public-read
     - run:
         name: Update Latest Report Reference with History
         command: |
           aws s3 rm s3:///allure-reports/latest --recursive
           aws s3 cp --recursive ./allure-report s3://allure-poc-wojtek/allure-reports/latest/
           aws s3 cp --recursive ./allure-report/history s3:///allure-reports/latest/history/
     - run:
         name: Display Allure Report URL with Timestamp
         command: echo "Allure report is available at https://.s3.eu-central-1.amazonaws.com/allure-reports/latest/index.html" > allure-report-url.txt
     - store_artifacts:
         path: allure-report-url.txt
         destination: allure-report-url
workflows:
 version: 2
 build_and_deploy:
   jobs:
     - build

From now on, CircleCi will use your image that has the necessary tools and dependencies installed instead of having to install them every time. What benefits do we get from this? First of all, the speed of building the testing environment and the repeatability and isolation of the environment.

Conclusions

As we wrap up this guide, it's clear that integrating Playwright with CircleCI, coupled with the dynamic reporting capabilities of Allure and the robustness of AWS for hosting, creates a potent combination for any development team looking to elevate their testing framework. This setup not only streamlines the deployment pipeline but also provides a transparent and insightful view into the health of your software, ensuring that each release meets the high standards expected in today's competitive environment.

By following the steps outlined in this article, you've laid the groundwork for a system that not only detects issues early but also keeps your entire team informed about the state of the application through beautiful, accessible reports. Remember, the key to successful implementation lies in continuous refinement and adaptation to the unique needs of your projects and team dynamics.

Whether you're a seasoned developer or new to automated testing, the journey toward more efficient, effective, and transparent testing practices is an ongoing process. Embrace these tools, experiment with their configurations, and continue to push the boundaries of what your CI/CD pipeline can achieve. Happy testing, and may your development cycles be smooth and your deployments successful!