SmartBear CrossBrowserTesting review
Ranked #3 / 6 in Browser testing
— Last updated: 2021-12-08
SmartBear CrossBrowserTesting provides both automated and live tests across multiple platforms and devices going back to Windows XP and iOS 4. The large number of integrations and frameworks makes it particularly powerful, but it is limited by high test latency and the lack of accessibility testing features.Visit SmartBear CrossBrowserTesting
Smartbear CrossBrowserTesting offers plans starting at $39/m for live testing and $169/month for automated testing.
Device & browser support4.5/5
CrossBrowserTesting offers a good selection of desktop and mobile devices to choose from. All the versions of Windows, macOS, Android, and iOS ranging from Windows 11 to Windows XP, “macOS 11.0” to “Mac OSX 10.10”, Android 11 to Android 4.2, and iOS 14.0 to iOS 4.3.3 respectively are available. On desktop, each OS is available with different browsers (approximately 50 versions of each) and display resolutions (800x480 to 2560x1920). On older Android devices, less popular browsers like Opera Mobile and UC Browser are also available. Only mobile Safari is available for most iOS devices but Chrome is supported on more recent iPhones.
CrossBrowserTesting integrates with a vast set of tools which makes it possible to create complex workflows. Integrations with CI/CD tools (Jenkins CI, TeamCity, VS Team Services, etc.) and testing services (Lighthouse, TestComplete, Katalon Studio, etc.) are available. Testing frameworks such as TestNG, NUnit, Mocha, Jest, Cucumber, Jasmine, etc. are also supported. CrossBrowserTesting provides a powerful API that enables customizing website-testing workflows and allows implementing browser testing services via HTTP. Any action which can be performed via dashboard can also be performed via an API endpoint.
Debugging & error reporting4/5
CrossBrowserTesting allows adding notes and tags to test reports - this makes it easier to filter test reports and share. Test reports can be shared in bulk with team members via a public link, Slack, Jira, or email.
All test executions show test steps with relevant details like execution logs, detailed network logs, video recordings and screenshots. This makes understanding the results and debugging process easier.
What we like
CrossBrowserTesting offers visual testing services for identifying inconsistencies in user experience across devices. An automated screenshot tool allows capturing screenshots on a large number of desktop and mobile browsers. It can perform regression testing to identify layout differences against a baseline run.
Tests can be scheduled to run on set intervals or triggered from either dashboard or API. Advanced configuration can be set up to perform actions like HTTP authentication, form login, waiting for a set duration, executing a Selenium script before capturing screenshots, etc. Multiple screenshot tests are run concurrently which reduces wait time and it is possible to run tests on locally hosted websites.
Results from the test can be notified via email or can be sent to Applitools for inspection. Live testing sessions (with identical testing configuration) can be launched with one click from the screenshot test reports to quickly interact with malfunctioning interfaces.
Standout features offered by CrossBrowserTesting include:
- Support for uncommon browsers like Opera Mini, Samsung Internet, Microsoft Edge Mobile, etc.
- A vast number of real devices are available for live interaction.
- Remote developer tools are available for debugging.
- Screenshots, videos & logs can be accessed after testing.
- Network logs for each request are displayed in an easy-to-understand gantt chart to identify performance bottlenecks.
Some of the features mentioned above are not available in the alternative browser testing tools we tested. For example, BrowserStack does not provide test reports for live tests, LambdaTest does not offer real devices (currently in beta), and Ghost Inspector and Selenium do not support live testing at all.
The record & replay feature allows recording website interactions and replaying them as tests without touching a single line of code. Tests can be combined to build suites, which can then be replayed or scheduled to do “automated-manual” testing. This feature performs on par with similar offerings by other vendors such as Ghost Inspector Web Test Recorder and Selenium IDE.
Remote testing servers can be connected to local machines (via a secure web socket tunnel) or any proxy service to test websites behind a firewall without needing to expose them to the internet. It is also possible to share a directory on a local machine with read-only access to testing servers. Local tunnels can be established via Chrome extension, NodeJS module, desktop application, or the command line.
What we don’t like
CrossBrowserTesting does not provide solutions to perform automated accessibility testing. The Lighthouse report generated during screenshot testing has a section on accessibility, but nothing more is offered. This came as a surprise because it is a basic feature and is already present in mainstream browsers such as Chrome. Specialized browser testing tools should provide more advanced, feature-rich solutions for accessibility testing.
When compared to other tools, network latency during live testing is high. Having to wait >1 second on average for each keystroke to appear on screen while typing is unreasonable and makes testing a tedious task. This issue is even worse when using a tunnel or a proxy to test locally. During live testing, sound is available, but it has a low bitrate quality.
CrossBrowserTesting used to provide an option to perform functional testing on headless browsers on Ubuntu - this feature has since been silently removed. We also found that the latest macOS and iPhone devices announced in September 2021 are still not available after 2 months. On older iOS devices, only Safari is supported. Chrome is available only on a few recently released iPhones.
Testing localization settings of websites are not possible directly - all testing traffic originates from servers in the US. Users can set up a proxy that routes all testing traffic from the desired location, however this induces unnecessary overhead and additional costs.
We encountered some issues with the record & playback functionality. Tests
generated are not perfect and sometimes need to be corrected by the tester
before they can be used. We discovered that any scroll action performed during
recording is recorded as
Scroll to (0,0) and using a laptop trackpad resulted
in sporadic clicks. Tests generated by this method can not be exported or used
anywhere else, thereby enforcing a vendor lock-in. Ghost Inspector Web Test
Recorder, a competing product, allows exporting tests in Selenium IDE format -
this makes it possible to store tests alongside code and execute them anywhere.
Console developer perspective
In our review of the best browser testing tools for developers, we highlight the best 6 browser testing tools for developers in 2022. See how they compare to SmartBear CrossBrowserTesting in the table below.
Interesting tools by email
Our free weekly newsletter picks out the 2-3 most interesting tools. See the latest email.