Email: sales@altexsoft.com

Phone: (877) 777-9097

Request for Proposal

The Good and the Bad of Katalon Studio Automation Testing Tool

Reading time: 7 minutes

Katalon Studio is one of the emerging tools for automated testing. Along with Selenium, Ranorex, TestComplete, and Apache JMeter, it’s worth discussing as more and more testing engineers opt for Katalon. The platform facilitates running automated tests for Web interfaces, APIs, and mobile apps (both iOS and Android), provides test recording, and analysis reports.

In this article, we’ll look at the pros and cons of Katalon Studio. If you’re looking for a broad overview of the main test automation providers, have a look at our dedicated article.

What is Katalon Studio?

Katalon Studio is a free-license tool released in January 2015 with a Selenium-based engine. Mostly, Katalon is designed to create and reuse automated test scripts for UI without coding. Katalon Studio allows running automated tests of UI elements, including pop-ups, iFrames, and wait-time. The tool can be launched on Microsoft Windows, macOS, and Linux.

katalon key features

Katalon Studio key features

Katalon’s main advantage is that it’s easy to deploy and has a wider set of integrations compared to Selenium, the market leader. Katalon has dual scripting interfaces for users with different programming skills. This means that testers with limited technical knowledge can use a simpler user interface that doesn’t require writing code.

The mode for more proficient users has access to scripting with syntax highlighting, code suggestion, and debugging. Katalon Studio has pre-defined artifact structure: a number of templates for test cases, test suites, test objects, and reports. Katalon Studio supports local and remote testing, as well as parallel and sequential executions. It runs on Groovy (Java) scripting language.

This solution has a quick setup and a number of pre-installed templates that allow repeating some testing patterns. Katalon Studio is a cross-browser tool that supports Web, mobile, and API testing. These solutions go with analytics and recording modules.

API testing

Unlike Selenium and Ranorex, with Katalon Studio you can test APIs without additional integrations. This inbuilt module allows testers to conduct end-to-end API testing, automate scripting, and maintain their tests. Main features of this module are code auto-completion, code inspection, code snippeting, references, and debugger. The software supports all types of REST, SOAP/1.1 and SOAP/1.2 requests and multiple data sources (XLS, XML, databases with dynamic mapping to maximize test coverage).

The API testing mode allows users to import tests from such API testing and editing tools as Swagger, Postman, and WSDL. Katalon API testing also has an inbuilt response viewer with auto-formatting and search for access to artifacts.

Web testing

Katalon offers a complete web testing solution with inbuilt Continuous Delivery/Continuous Integration and DevOps integrations. It supports testing of Web, Android, and iOS applications. The tool has code-assist utilities like built-in object spy, code refactoring, and in-context references. Native plugins integrate with Jenkins, Git, JIRA, and qTest.

Katalon’s web-testing execution mechanism uses multiple configurations and datasets. It also allows for customizing execution workflow and running it automatically. The module features importing external libraries to improve the automation functions.

Mobile testing

This module allows users to test mobile Web, iOS, and Android apps. With native integration by means of Appium, a test automation tool for mobile applications, the module supports testing for the latest mobile platforms and devices without additional installations. Katalon speeds up mobile testing by detecting and storing the objects.

Katalon Analytics

Unlike Selenium, Katalon Studio provides testing analysis and reports. This module of Katalon Studio shows test results and maintains test history. Katalon Analytics provides interactive dashboards and reports. Reports prioritize testing areas, highlighting the most crucial parts. All execution patterns are recorded on screenshots and videos, and test results are displayed as specific KPIs on dashboards. Analytics in Katalon Studio features email failure notifications that arrive with reports.

Katalon Recorder

Katalon is shipped with a Selenium-based recording module for Chrome and Firefox. It records tests, allows testers to perform debugging, automate test management, and export testing scripts into C#, Java, Ruby, Python, Groovy, and Robot Framework. Katalon Recorder has a record and playback feature for testers without programming skills and saves the methods previously used in testing.

Katalon support

Katalon Studio is a free tool that offers technical support plans for different prices. These types are:

  • Business (respond within 24 hours, 1 Helpdesk account, and 1 support project) – $2,500 per year
  • Enterprise (respond within 12 hours, 3 Helpdesk accounts, and 3 support projects) – $5,000 per year
  • Enterprise Premium (multiple accounts, exclusive support) – customizable price.

Katalon Studio

Download Katalon Studio

Documentation

So what are the main Katalon Studio pros and cons? Among the strengths are a large number of integrations, the toolset, and relative simplicity of use. As for the main disadvantages, we can spot the lack of scripting languages supported and some performance issues. Now, let’s take a closer look at them.

Pros of using Katalon Studio

+ Integrated with CI/DevOps workflow

Katalon support is compatible with a number of tools, used in Continuous Integration and DevOps. Katalon Studio supports native integrations with Git, qTest Manager, JIRA, and Slack. Also, it’s integrated with Jenkins and TeamCity, cloud CI tools like 12 CircleCI, Travis CI, SauceLabs, BrowserStack, Selenium Grid, and Kobiton.

Katalon Studio integrations
Katalon Studio integrations

+ Intuitive analytics dashboard and reports

An inbuilt, data-driven testing module allows users to analyze tests based on recorded scenarios. Katalon displays the analytic results in the form of built-in reports that can be exported in PDF, HTML, Excel, or CSV.  The reports are visually intuitive and simple to work with.

test execution report katalon
Test execution report in Katalon Analytics
Source: 
Katalon Studio tutorials

+ Supports several types of testing

In Katalon Studio you can run the following test types:

Keywords-driven. This approach works for testers who aren’t proficient in scripting. Based on keywords that represent users’ actions on the AUT (Applications Under Test), a tester can run an automated test. Keywords can be customized.

Data-driven. A test execution strategy that features reading test scripts from various data-sources. The data is taken from data pools, Excel files, ADO objects, CSV files, and ODBC sources.

API-testing.  It supports XLS, XML data sources, and databases with dynamic mapping for better test coverage. Also, the tests are compliant with Cucumber, a BDD-based (behavior driven development) framework for acceptance tests of web applications.

Cross-browser. Katalon supports all popular browsers and allows running automated tests in Chrome, Firefox, Internet Explorer, Safari, Edge, Remote, Headless, and custom environments.

+ Wide toolset and integrations

This quality assurance tool arrives in the box with ALM (Application Lifecycle Management) integrations, frameworks, and plugins that can be installed on Windows, macOS, and Linux. As a result, Katalon doesn’t require additional integrations to run tests. It supports parallel execution on multiple environments and can be adjusted to many frameworks. Integration with qTest, a centralized test management tool that facilitates the communication of a QA team, allows for controlling test requirements, test case planning, and defect management.

+ User-friendly GUI

Katalon’s UI offers rich graphics with tree views, tables, and menus which allow a user to manage test artifacts easily. Testers find the tool’s interface convenient, user-friendly, clean, and easy to grasp even for non-technical users.

katalon interface
Katalon Studio interface
Source: 
Katalon Studio docs

+ Mild learning curve + learning materials

Katalon Studio is famous for its simplicity, both of installation and usage. It’s easy to set up thanks to its many integrated packages. It’s good both for beginners and advanced users, who can use the whole IDE and write advanced scripts. Developers have access to a number of training videos and documentation for users: Katalon has GitHub repositories, a list of online courses on Udemy, and video courses on YouTube.

Katalon Studio short guide
Source: 
Katalon Studio docs

+ Free

Katalon is a free tool, which is considered an advantage by most of the users. However, the support works on a paid basis.

Cons of using Katalon Studio

– Lack of scripting languages

Unlike Selenium and TestComplete, the only scripting language supported by Katalon is Groovy. This scripting language is in the Java family so anyone who knows Java can use it. But, users would like to see more languages supported.

– Small community

Since Katalon was developed in 2015, the community is small, and you may experience a lack of support from peer testers. Users must rely on the paid customer support team or Katalon forum replies. To put this in perspective, Ranorex has 104 Capterra reviews, while Katalon Studio has only 11.

– Not open source

The tool has a closed source code, resulting in a smaller number of developers in the community. Selenium, one of the main competitors to Katalon, is an open-source tool that allows engineers to customize it or use community-built packages. However, Katalon Studio has some open components and is planning to introduce an open-source plugin platform.

– Documentation update lags

As much as the users like the number and variety of training materials for Katalon, they hate the documentation. It’s too long, complicated, and causes confusion as it’s not updated along with new version releases. So, anticipate that something won’t work as expected.

– Performance issues

Users report some bugs that hamper testing, slowing it down. For example, sometimes the tool freezes or may start lagging, the text and object verification within iframes is problematic. Mobile testing takes more time due to the necessity to capture and write code.

Who uses Katalon Studio?

Katalon Studio is used by companies from different industries.

Care Logistics, a company that provides healthcare logistics software and services to hospitals and health centers, faced the challenge of a quick verification of system stability. To verify the system’s functionality, the testing team had to create over 8,000 manual test cases and execute all of them before deployment. With Katalon Studio, the team managed to convert and automate these tests. As a result, the team could verify more issues and had 30 percent of products automated. Also, they achieved a 50 percent reduction in regression testing lifecycle.

Tunaiku, an Indonesian fintech lending venture, had to make sure that user data and transaction information was safe. This required a number of quality assurance tests in CI workflow, which involves Jenkins and Docker integration. Also, they needed to generate automation test scripts effectively. Tunaiku solved these quality assurance team problems by creating automation test scripts and customized code infrastructure in Katalon. This helped them reduce scripting time by 60 percent.

Angler, an Internet and technologies provider from India, used Katalon Studio for API testing. Their task was to test APIs and generate API scripts, as well as detect bugs in the development phase and prevent them from getting into APIs. As a result, Angler covered over 80 percent of functional testing with Katalon Studio.

Final word

Katalon Studio is a good choice for small and medium businesses. It’s an evolving solution with many integrations that allow you to cover a variety of testing types with a single tool. It comes with all the necessary installations out of the box to run various types of tests, including API testing. It’s free and easy to use. Katalon can be used by specialists with different QA engineering roles and varied programming skills, making it an attractive solution for teams with testers of different levels. Katalon Studio is mostly used to automate functional testing to reduce time spent on scripting. That said, because it has downsides like minor bugs and support for only one scripting language, it’s not the best enterprise solution.

4 Comments

Comments

avatar
Sort by: newest | oldest | most voted
Ozzy Sandoval
Jun 20, 2019

Hi,

Thank you for the great article!

I would like to point out that at the moment, Katalon Studio has 431 reviews on Capterra
https://www.capterra.com/p/157828/Katalon-Studio/

Ranorex has 111
https://www.capterra.com/p/173144/Ranorex-Studio/

Sorry to be “that guy” 🙂 but I feel that con needs to be updated

AltexSoft
Jun 20, 2019
AltexSoft

Hi Ozzy, thanks for pointing this out! We’ll make sure to update this. 🙂

Leo
Mar 8, 2019
Leo

Hi!
What’s the best enterprise solution for you?
Thanks!

AltexSoft
Mar 12, 2019
AltexSoft

Hi Leo. We usually apply Selenium WebDriver for our testing purposes, you can find our review of the tool here https://www.altexsoft.com/blog/engineering/the-good-and-the-bad-of-selenium-test-automation-tool/