Selenium or Cypress or WebDriver or Playwright oh my! Overwhelmed with options or no idea where to start when choosing a test automation framework? Then this one is for you.

Below I explain the key considerations for choosing and successfully implementing an automated test framework. I will be focussing on UI test automation frameworks. The good news is the rules apply just the same for Unit, API and Load tests.  

Language

Choosing a language should be your first decision to help narrow the field. Automation frameworks exist in every imaginable language. It’s also one of the most important decisions to help you successfully implement any automation framework. Why? Because if your engineering team mostly writes in C# and TypeScript and you chose to write tests in Ruby, you are much less likely to get developer buy in and support then you would be if you chose to write tests in TypeScript. 

Having said that..

Don’t assume anything!

You shouldn’t assume that engineers who write the front end in a given language will want to use that language for writing tests. Depending on your teams structure, you might have engineers who are far more comfortable writing code in the language used for writing the back end. You should have a conversation with the team to get an understanding from them what language they would be most comfortable with. While you are at it, try to get a picture for what they would want from a test automation framework. 

Test Automation Framework Features 

As a starter for 6, you’ll want to consider things like:

  • Browser compatibility
  • Mobile testing support
  • Test parallelisation
  • Documentation and support
  • Test reporting
  • CI pipeline integration 

Let’s dive into these a bit more.

Browser compatibility

When it comes to browser compatibility, not all test automation frameworks are born equal. There are a range of ways that test automation frameworks will interact with a browser and this affects compatibility. The key consideration here is which browsers do you want to support with your testing? This doesn’t have to be every browser your users use for every test.

If your user base does use a wide range of browsers, you might want to consider running tests against more than one. You could also look into additional features like visual testing. Tools like Percy give you confidence that your website looks correct on various browsers and at different resolutions.

Mobile testing support

Does your project provide a different mobile web user experience? If so, mobile testing support may be a key consideration for your test automation framework. Not all frameworks support this out of the box. You would need to consider if this something you need.  Many frameworks that don’t support mobile testing natively can be extended to do so. Although this adds an additional layer of complexity. 

Woman holding iphone. Always remember to consider mobile testing when considering a test automation framework

Test parallelisation

Are you going to be writing a lot of tests? You may want to consider a framework that allows you to run multiple tests in parallel. Running tests in parallel can make test runs much quicker. Test will need to be written in a “thread safe” way. The total the number of threads will be limited by the hardware the tests are being run on. 

Documentation and support

This should be a big part of your decision making process. You need to be sure when making your choice that the framework offers you what you need, not only in features, but also in support. Is the documentation clear and easily understood by you and your team? Is there a good community of support around the framework?

thoughtworks Technology Radar can offer an excellent insight into a test automation framework and whether it might be well suited to your team. Not everything is included in each release, but you can search for the framework you are interested in to find it in historical issues of the Technology Radar. Just remember to keep in mind how things may have changed since it was last in featured the Technology Radar. 

Test reporting

I’m not aware of any test automation frameworks that don’t offer this in some way, but they all offer something different. Consider who the audience for the report is and be sure that they are happy that it contains the information they need and how that is presented. 

CI pipeline integration 

Once you have your tests automation framework up and running, it’s likely that you will want to run the tests as part of a pipeline. Although most frameworks can be made to work with most pipeline tools, it is still worth some time upfront to make sure that your shortlist of options will work well with your pipeline tooling.

Proof of concept

Once you have narrowed down your choices to one or two options, it is worth investing some time to build a proof of concept. Get hands in with the code and automate a simple journey. This gives you a chance to experience how it will be to use the test automation framework and also do some testing of it. If you have more than one test framework on your list still then by building out a proof of concept for each you can compare test performance as well as developer experience. 

To BDD or not to BDD

Using BDD/cucumber/gherkin in the test automation framework comes down to two things:

  • Team practice
  • Team preference 

If as a team you work in a true BDD style, as exceptionally well described by Aslak Hellesøy over on the Cucumber blog, then making use of Cucumber/Gherkin syntax just makes sense. 

If you don’t, then it comes down to preference. It can aid readability in a team where you have manual testers or non-technical QA who you want to introduce to automated testing. However, it does add an extra layer to your tests to achieve it. So a more technically experienced team may feel they don’t need or want it. Even if they already work in the BDD fashion. 

Summary

Hopefully now you have a clearer understanding of how to approach how to choose a test automation framework. If you have any questions still, feel free to leave them as a comment and I will be happy to help. 

Still not sure where to start? Why not try out my guide to Writing your first Test Framework in C# or Valentino Gagliardi’s Cypress Tutorial for Beginners