Which JavaScript UI Testing Framework to Use in 2020?

Nowadays, most web pages are based on JavaScript.

Therefore it would be logical to use the same language to write tests. Frontend developers can maintain tests, execute scripts, and interact naturally with the page, but which JS framework should you use?

With so many tools available on the market, plenty of you have probably asked this question in your heads. There are old and well-known solutions that can require a lot of configuration to start, as well as new and fancy technologies tempting us with a five-minute setup and clean documentation.

I will do my best to review some of the most popular and provide a summary with the pros and cons of each one. I will be focusing on end-to-end (E2E) UI tests, as this is my area of expertise.

Selenium WebDriver + JavaScript

You’ve probably already heard about Selenium WebDriver. It’s the most popular open-source testing framework, very often powering other automation frameworks. Some of its biggest strengths are that it is versatile, robust, and can be used with any browser. Besides this, it has a vast community of active users and you can easily find answers to problems on the internet. On the other hand, it is not very fast and requires much more knowledge to set it up correctly. To get pretty, wrapped functions, you need to write them by yourself.

Pros

• Multiple browsers support
• Big community
• Highly configurable

Cons

• Steep learning curve
• Requires a lot of setting up
• Not the fastest one

Example including logging in

it('User can log in', async () => {
  await driver.get('http://automationpractice.com/index.php');
  await driver.findElement(By.xpath('//div[@class="header_user_info"]')).click()
  await driver.wait(until.elementLocated(By.id('email')));
  await driver.findElement(By.id('email')).sendKeys('example@netguru.com')
  await driver.findElement(By.id('passwd')).sendKeys('password', Key.RETURN)
  await driver.wait(until.titleContains("My account"));
  driver
    .getTitle()
    .then(title => {
      expect(title).toEqual('My account - My Store');
    });
});

Cypress

Cypress is quite fresh on the market as it appeared in 2017, and its popularity is steadily growing. It offers an all-in-one solution, requires very minimal setup, and is not based on Selenium. Reading its documentation, you can tell that it was made to be user-friendly. There is an excellent, minimalistic runner on which you can watch your tests going through pages, and you can even peek at each step. One of the most significant trade-offs compared to Selenium is being restricted to only Chromium based browsers and Firefox for which support was added recently. Some functions are still not built-in and require workarounds and libraries, like file upload, for example.

Pros

• Easy to set up and write
• Test runner makes debugging straightforward
• Good documentation

Cons

• Only supports Chromium based browsers and Firefox
• Lacks some functionalities
• You need to pay to unlock the full version

Example

it('User can log in', () => {
  cy.visit('http://automationpractice.com/index.php')
  cy.contains('a', 'Sign in').click()
  cy.get('#email').type('example@netguru.com')
  cy.get("#passwd").type('password{enter}')
  cy.title().should('include', 'My account - My Store')
})

WebdriverIO

WebdriverIO was first introduced back in 2015. It offers a custom implementation of the Selenium WebDriver API. The main advantage is support for both web and mobile by using Appium. Setup is pretty easy when following the well-written documentation explaining each step. The website lists many boilerplates, including most of the popular frameworks like Cucumber, Jasmine, and Mocha. It also supports all of the browsers Selenium does. However, it is not as popular as the other frameworks covered here, and not everything can be found on Google. The API documentation is incomplete.

Pros

• Easy to setup

• Dedicated community

• Appium support

Cons

• Incomplete documentation

• Not growing in popularity

Example

it('User can log in', () => {
    browser.url('http://automationpractice.com/index.php')
    $('a=Sign in').click()
    $('#email').setValue('example@netguru.com')
    $('#passwd').setValue('password')
    browser.keys('Enter')
    const title = browser.getTitle()
    assert.strictEqual(title, 'My account - My Store')
})

CodeceptJS

Utilising many of the other frameworks, CodeceptJS is probably the most versatile tool on this list. Configuration allows running tests with WebDriverIO, Protractor, Nightmare, and Puppeteer helpers. All share the same API, so the written code is the same. The first thing that you’ll notice is the specific syntax; tests are scenario-driven out of the box. Code can be read and understood by a non-technical person, allowing them to help in writing tests. Everything is well documented, and, similar to Cypress, there is even a test UI where you can see snapshots of each test step and debug it easily.

Pros

• Highly versatile

• Good documentation

• Easy to write and read tests

Cons

• You are restricted to the syntax

• Small community

Example

Scenario('User can log in', (I) => {
  I.amOnPage('http://automationpractice.com/')
  I.click('Sign in')
  I.fillField('#email', 'example@netguru.com')
  I.fillField('#passwd', 'password')
  I.pressKey('Enter')
  I.seeInTitle('My account - My Store')
});

Nightwatch.js

Nightwatch.js is a framework that grabbed my attention when I got into E2E testing. At the time, the setup and clean syntax were the main advantages for me. Nowadays, the installation is still pretty quick, but the documentation doesn’t include all of the information you need and other frameworks have done it better. It also uses Selenium Webdriver, so you have access to all browsers. The framework provides all of the functionality needed but all at an average level. Depending on your use case, you may want to use another tool.

Pros

• Quick setup

• Supports multiple browsers

Cons

• Weak documentation

• Limited functionality

• Slow performance

Example

'User can log in': function (browser) {
  browser
    .url('http://automationpractice.com/')
    .waitForElementVisible('body')
    .click('a.login')
    .setValue('#email', 'example@netguru.com')
    .setValue('#passwd', 'password')
    .click('#SubmitLogin')
    browser.assert.title("My account - My Store")
    .end()
}

Protractor

Protractor is one of the oldest frameworks mentioned in this article. It was made by the AngularJS creators specifically to test Angular applications but it can be used for other pages as well. Once well-liked and recommended, nowadays it lacks popularity in favor of other tools. It uses Selenium, therefore it offers multiple browser support and can even run those tests in parallel. Suited for Angular applications, it includes helpful selectors and functions. Setup is easy but debugging can take some time and might be harsh for a newcomer. It doesn't have functions like built-in wait, which other frameworks offer. Personally, at this time, I would not bother with it if your app is not using Angular, and even then, I would consider other options.

Pros

• Angular specific commands

• Runs on multiple browsers in parallel

Cons

• Hard to debug

• Decreasing popularity

Example

it('User can log in', function() {
  var EC = protractor.ExpectedConditions;
  browser.waitForAngularEnabled(false);
  browser.get('http://automationpractice.com')
  element(by.linkText('Sign in')).click()
  browser.wait(EC.presenceOf(element(by.css('body'))), 5000)
  element(by.id('email')).sendKeys('example@netguru.com')
  element(by.id('passwd')).sendKeys('password')
  element(by.id('SubmitLogin')).click()
  expect(browser.getTitle()).toEqual('My account - My Store')
});

TestCafe

TestCafe is similar to Cypress; pretty fresh compared to the other frameworks. Also, just like Cypress, it is not built on Selenium. It offers a test runner called TestCafe IDE, but it's much more than just a Cypress runner. You can build tests there and skip writing code completely. Unfortunately, it is a paid solution – you get 30 days free, then the cost is US$250 per year.

In contrast to Cypress, TestCafe offers multiple browser support, including mobile versions, and it works pretty much out of the box. I didn't have to download or install any browser drivers or specific browser versions. I just ran a command with a different parameter, and it worked. There is also the Smart Assertion Query Mechanism, which automatically waits for changes to take place and retries with checkups. When tests are run, a large pointer shows which element is being interacted with. The only downside can be syntax and the fact that you still need to think of JS asynchronicity, where Cypress wraps it neatly, allowing you to write synchronous code.

Pros

• Easy to set up

• Multiple browser support not based on Selenium

• Automatic waits

Cons

• Full version is paid and quite expensive

Summary

All frameworks have their pros and cons. For me, the most important thing is accessibility. With modern tools like Cypress and TestCafe, you can feel they were made to address the pains of the previous generation of testing software. Another thing I value is the community and support of my testing peers. Let's take a look at the graph below which shows the number of stars for each repository on GitHub over time.

As you can see, the blue line representing Cypress is going up quickly, even moving ahead of Selenium recently. In my opinion, this proves its usefulness combined with approachability. It is because Cypress is widely accepted and liked by both JavaScript consultantsand testers. Therefore, it is also my go-to recommendation for almost everyone. If you don't require support for more browsers, you should give it a try.

Second place goes to TestCafe because it also skips Selenium entirely, which, as you may or may not know, happens to be flaky at times. TestCafe feels like a robust and complete package. The IDE is a nice touch, but it's not necessary to write tests.

Third place, I’d say, is Codecept because of its unique approach and support for both browsers and mobile applications. The only thing that worries me is its lack of popularity.

Is it the end of the Selenium era? Not yet, but I think we are getting there. I don't recommend Selenium in the summary because I think it's better used with other languages like Java or Python. Although, if you are looking for a low-risk enterprise solution, it might be for you.

Happy automating!

Photo by Daan Stevens on Unsplash