Ranked #6 / 6 in Browser testing
— Last updated: 2021-12-14
Selenium is an open source browser testing framework used by many other browser testing tools because of its maturity and power. However, it requires significant technical knowledge to set up - including managing your own test devices - so we do not recommend it for most users.Visit Selenium
Selenium is free and open source (Apache License 2.0).
Device & browser support3/5
Selenium can be configured to run on all versions of Windows and macOS. Many Linux distributions are directly supported and some determined users can even get it to work on BSD-based distributions. Selenium 4 is W3C compliant and browser vendors like Google, Microsoft, Mozilla, and Apple provide Selenium compatible drivers for their browsers. Selenium does not support testing on mobile devices directly, but it is possible to work around this limitation.
Selenium can be integrated to work with many CI/CD tools like Azure Pipelines, Jenkins, GitLab CI, etc. Unfortunately, support for such integrations is community-driven and development can be slow. One example of this is the Selenium plugin for Jenkins - despite having a known security issue since 2020 it still gets over 2,000 monthly downloads.
As it is an open source project, users can develop their custom plugins to meet their needs, and the possibilities for integrations are endless. A vast number of community plugins and libraries are available, but the quality is variable and many have been completely abandoned.
Debugging & error reporting1.5/5
Selenium is designed only to automate browsers, not to report on the status of test cases run. By default, all results are printed to standard output - sharing findings and collaborating in a large team is difficult. A few third-party solutions such as JUnit, TestNG, and Extent reports are available to simplify the test case management procedure and generate GUI reports that cannot be accessed other than through the raw output.
What we like
First introduced in 2004, Selenium has evolved into an industry standard. Selenium 4 is W3C compliant and many cross-browser testing tools are built on top of it. Browser testing vendors like BrowserStack, LambdaTest and Sauce Labs actively contribute to the development of Selenium and provide Selenium Grid as a service on their respective testing platforms.
Selenium supports the Chrome DevTools Protocol and is working with browser vendors to add support for WebDriver BiDirectional Protocol. Together these features greatly enhance the testing capabilities and allow better control over browser properties, thus increasing possibilities for browser manipulation. Some of the noteworthy features are:
- Intercepting HTTP responses - parameters like URL, response headers, response code, etc. can be accessed.
- Network speed emulation and blocking certain URLs.
- Automating the input of basic auth credentials.
- Mutation observation - listen to events when a specific DOM element is updated.
- Mocking geolocation.
- Capturing console logs.
- Work with browser cache and override user agent string.
Selenium IDE is a low-code testing tool that records and plays back browser interactions. This allows novice developers to conduct automated testing. It is possible to design intricate test cases by adding conditional logic and looping to tests. The Selenium IDE also supports plugins that further enhance its functionality. Web Test Recorder is a similar offering by Ghost Inspector and it is possible to run tests generated by Selenium IDE in the Ghost Inspector cloud environment.
Selenium Grid, which allows executing tests on multiple physical or virtual devices, can be used to run tests on multiple versions of different browsers on numerous operating systems. Grid consists of several components which makes it very flexible, for example, assembling a grid of four machines each running three instances of Firefox 70.0 would allow 12 tests to run concurrently.
The ability to distribute tests makes it possible to scale tests from a single machine to parallel executions on servers all over the world. The number of tests that could be executed concurrently is only limited by the capabilities of the hardware.
Since the tester owns the testing infrastructure, it is possible to exercise greater control over data handling and storage policies. Tests can be executed without an internet connection, thereby minimizing the network costs and security concerns.
Selenium provides some advanced capabilities such as manipulating browser cookies, drag & drop mouse actions, support for GraphQL queries, transfer of files from the client machine (hub) to the remote server (test node), tracing the request lifecycle, capturing screenshots, switching window tabs and setting browser profiles in Firefox. Performance metrics like Resources, Documents, JSHeapUsedSize, etc. can be collected when running automated tests. This makes Selenium a good choice for tasks other than testing, like web scraping.
What we don’t like
Compared to the other SaaS browser testing tools we reviewed, Selenium has some major limitations because it was not created to perform visual testing or generate user friendly test reports. Tests are focused on whether or not the correct content is displayed rather than whether it is displayed correctly. This is a big restriction in today’s fast-paced world where developers run A/B testing on the tiniest of UI elements to increase conversion rates and engagement.
Other tools like BrowserStack and Sauce Labs provide modern solutions to perform visual regression tests whereas the Selenium team believes that the user interface is a secondary consideration for users.
Test case management and error reporting with Selenium is an arduous task. It does not generate test reports or notify about the results, other tools provide these features out of the box. In contrast to Sauce Labs, which provides advanced services to detect patterns in error reports, Selenium reports all results to console output by default. This makes collaborating with team members difficult. Third-party software must be used for organising test cases which adds more complexity to the tech stack.
Selenium does not explicitly support testing on mobile devices. Two other solutions, namely Appium and Selendroid, are available to solve this by implementing mobile tests using Selenium. Selenium BiDi API is under development and Chrome DevTools Protocol does not have a stable API - the functionality is highly dependent on the version of the browser.
Selenium IDE, a no-code solution to record tests, is only available for Chrome and Firefox browsers. It does not provide all the features supported by Selenium and users must resort to code, which requires a good grasp of one of the supported programming languages. When writing tests with Selenium, there is a steep learning curve. The only technical assistance available is through community forums.
Selenium Grid requires the availability of real or virtual desktop and mobile devices connected to a network during testing. Setting up a testing grid is a complex process and requires a large capital investment upfront. Regularly updating machines is a gargantuan task and adding the latest devices on launch further increases the costs.
Managing your own device lab is a questionable use of time and money when SaaS options are available. Alternatives we reviewed provide Selenium Grid as a service that already consists of thousands of machines spread across different geographic regions. There are no upfront costs associated with this and testers can start being productive within minutes. The time and effort spent in setting up infrastructure for small projects is impractical for all but the largest teams - even then it may be more efficient to pay a provider to do this for you. Some providers like Sauce Labs and AWS Device Farm even allow purchasing real private devices in addition to their already large collection of public devices.
Console developer perspective
Browser testing tool comparison
In our review of the best browser testing tools for developers, we highlight the best 6 browser testing tools for developers in 2022. See how they compare to Selenium in the table below.
Interesting tools by email
Our free weekly newsletter picks out the 2-3 most interesting tools. See the latest email.