We have already written in-depth on what an API is. We’ve also elaborated on the basics of software quality management. This time we connect the dots under the topic of API testing.
What is API testing?API testing is a set of quality assurance actions that include sending calls to the API, getting output, and validating the system's response against the defined input parameters, in particular, the accuracy of data and data's format, HTTP status codes, and error codes.
Usually, API testing is performed on APIs produced by the in-house development team. We don’t test third-party APIs, but we can test the way our software accepts their requests.
The approach to API testing largely depends on the API type. There are web APIs aka web services, database APIs that connect applications with DB management systems, operating systems APIs, and remote APIs for accessing resources located outside the device requesting them.
In this article, we’ll focus on in-house web APIs and their quality assurance.
API in terms of the three-tier architectureClient, server, and database are the three independent tiers of software architecture. Usually, a client is a web browser or a mobile application that presents the requested info, while business logic is realized on the server side. Client and server communicate through requests based on different transfer protocols. The system and logic of protocol-dependent requests are the API itself.
The role of API in the three-tier architectureIn the most popular software architectural style for data transfers - REST - clients make HTTP/HTTPS requests to the server. The server retrieves the requested data from the database, transforms it according to the business logic, and returns it to the client as a response in JSON, XML, or other formats. Finally, the UI displays it to the user.
Another type of web API is SOAP API - a legacy web communication protocol that is still in use. SOAP isn’t limited to HTTP/HTTPS protocol but supports many others including TCP, SMTP, and FTP, but it works with XML format only. You can find out more about SOAP and how it differs from REST in our dedicated article.
And now, let’s see how it works from the QA side.
Why API testing is importantA basic concept in automated testing is a test pyramid. It demonstrates how to tackle automated testing for a project: what parts to test first as they form the base of the pyramid, and what to leave for the final stages - the pyramid’s tip.
Following the pattern of the testing pyramid, the first is the unit test layer - code testing often performed by developers while writing it. Then comes the server layer where API testing belongs. Later, when the front end is finished, UI testing takes place.
Three layers of the test automation pyramidAs we already mentioned, API testing is often overlooked. But why is it so important?
UI tests can’t cover every aspect of back-end testing and turn out to be insufficient for verifying functional paths and API services. This could leave you with bugs rooted in the server or even at the unit level. But now, the cost of mistakes is way higher as it may require rewriting a considerable amount of code and derail the release.
The solution here is to start testing at the early stages. If the request doesn’t return the needed value at the server layer, it won’t be displayed at the UI layer either. Not relying on UI to be ready, API tests can be performed early in the development cycle. This way, they’ll kill at least half of the bugs before they get serious.
In the Agile processes, where instant feedback is needed, unit and API testing are preferred over GUI testing as they are quicker to create and easier to maintain. While the latter requires considerable rework to keep pace with frequent change.
But don’t think we dismiss UI testing. Certain defects can only be caught by GUI tests and it’s the only test type that truly checks the user experience. We just encourage you to pay attention to API testing as well and apply it to smooth the overall development workflow.
Types of API testing: What they focus onQA engineers approach API testing from different angles. They test whether it returns correct outputs in the expected format, whether it delivers responses within an acceptable timeframe, and how well it integrates with presentation layer software. Testers check an API’s reactions to edge cases (e.g. failures, unexpected or extreme inputs) and potential security attacks. So let’s address each of these aspects.
Functional API testsFunctional testing is the assessment of specific functions within the codebase. It makes sure the API actually works within the expected parameters, namely:
- returns the desired output for a given input; and
- handles errors when the results are outside of the expected parameters.
API performance testsTo ensure the API can handle the expected or higher load, QA engineers validate its functionality and performance by artificially creating or simulating API calls. We'll outline the types of API performance testing.
Load testing. The point of load testing is to measure where the limit of system performance under high load lies. That’s why we measure response times, throughput, server conditions, etc., while increasing the number of calls.
Soak testing. Load tests that run over a long period of time can reveal system instabilities like API memory leaks. So when you have a weekend ahead, leave automated soak tests running. On Monday, it will show you whether any unwanted behavior has emerged.
Stress testing. The idea is to gradually increase the count of virtual users to find the point at which the API starts throwing errors, slows down, or stops responding.
Spike testing. Unlike stress testing, here an API undergoes a sudden and extreme increase or decrease in the number of users over a short period of time. Spike testing checks whether the API is able to stabilize and return to normal functioning after that.
Scalability testing. You want to be sure that your system performance scales according to the changing load. To do so, increase the number of incoming requests and monitor whether it causes a proportional increase in response time.
API security testsSecurity, penetration, and fuzz testing are the components of the security auditing process aimed at testing an API for vulnerabilities from external threats.
Security testing. It validates whether security requirements are met. This includes authentication, permissions, and access controls, namely:
- what type of authentication is required to use the API;
- how well sensitive data is encrypted; and
- what authorization checks are set for resource access, etc.
Fuzz testing. The last step in the security audit tests the API at its absolute limits. Forcibly inputting massive amounts of random data, it tests whether the API will stand it or end up with negative behavior like a forced crash or overflow.
API integration and reliability testsBeing at the center of integrations between internal or third-party services, APIs need to pass the following tests.
Integration testing. Focused on communication between APIs, it ensures they are well-connected and don’t cause bugs in other APIs.
Integration API testing, Source: SoapUIReliability testing. When synced with various devices, an API should be tested for any possible disconnections.
How to go about testing the APIAPI testing is performed by submitting requests to the software through the API and then checking if it returns the expected data. When automated, API testing can be easily performed on a regular basis. The main advantage of automation is speed - less lag time between development and QA and less time spent on debugging in production.
QA engineers perform automated API testing using a testing tool that drives the API. We will discuss the most popular solutions in the next section.
API testing step by step, Source: Capital TerefeStep 1: Create API testing requirements. To be able to plan API tests, first, you need to determine testing boundaries and requirements which include:
- the purpose of the API and its target consumer,
- the workflow of the application,
- the features and functions of the API,
- other APIs this API interacts with,
- aspects of testing,
- the problems we’re testing for,
- priorities in testing,
- the desired output, and
- the definition of Pass and Fail.
Step 3: Make a trial API call. Before diving into thorough testing, make an API call just to check that nothing is broken and the API is operational.
Step 4: Define the input parameters. Plan all possible input combinations. You’re going to use them in your test cases, and later - for the authentication of the results to determine whether the API performs as expected.
Step 5: Create API test cases. After all the preparations are done, you can write and execute test cases, and, in the end, compare actual results with the expected ones. A good practice is to group them by test category. Examples of API test cases are:
- testing value in response to the input condition: You define the input and authenticate the output, which can be any type of data or status (e.g. Pass or Fail);
- checking the behavior of the API in the system when there is no return value in response;
- tracking events or interruptions triggered by the output;
- authenticating the effect on the system after data structure updates; and
- validating resources modified by the API call.
API testing tools: top choicesAPI testing isn’t very different from any other testing. But, it requires specific tools, dedicated to automating it. Among other things, those instruments vary in the programming language they’re based on.
Using an API testing tool, QAs usually either take advantage of out-of-the-box solutions or develop a customized framework from its components.
Verifying the status code to be 201 via PostmanPostman advantages. The tool is praised for ease of use and speed, tons of features, and many integrations. It has a friendly UI for constructing requests and reading responses, allowing automated tests to be created quickly. Postman runs on local machines so that you can stay in control of your data. Using its command-line tool Newman, you can integrate these tests in continuous integration (CI) environments.
However, Igor also warns that it’s hard to maintain extensive API collections and its built-in versioning system is weak, so, “It is quite painful to work on the API tests together with the whole team.” Igor’s recommendation for large projects is to create powerful API tests with additional libraries like Requests (Python) or REST Assured (Java), which we’ll describe in a bit.
Postman pricing. Postman has four pricing plans:
- Free version for small projects with up to three users
- Basic version (from $12 per user/month) for fast API designing, development, and collaborative team testing;
- Professional version (from $29 monthly) for the centralized management of the API workflow; and
- Enterprise version ($99 monthly) for API-first development.
Requests: a human-friendly HTTP library in PythonRequests, an open-source Python HTTP library, was released under the Apache2 License with the aim of simplifying HTTP requests.
Testing the response time to be less than the timeout via RequestsRequests advantages. The Requests library saves time and effort by fully automating keep-alive and HTTP connection pooling. No need to manually add query strings to URLs and form-encode POST data. Igor Pavlenko considers Requests a powerful library with easy-to-understand documentation, simple syntax, and rich functionality.
“It supports such features as OAuth 2 and Open ID Connect Authentication, various Certificates and much more. So, the choice is simple: If you have Python Requests is the best decision for you.”
REST Assured: a Java library supporting XML and JSON requestsREST Assured is a Java library for creating a REST API testing tool script. So, to get started with it, you’ll have to set up a new Java project first, and then include it as a library for your project. Its documentation is stored inside the GitHub repository. REST Assured is the best fit for functional testing of REST API services and it requires strong coding skills to create tests.
REST Assured based test to check a GET API response, Source: Saurabh AgarwalREST Assured advantages. API testing is pretty hard in Java, and Rest Assured was designed to simplify things. With it, you don’t have to create boilerplate code to test and validate complex REST responses. The library is open-source and has an active development community - another big pro.
JMeter: a Java-based tool measuring SOAP/REST API performance and moreApache JMeter is a time-tested tool initially used for load testing. These days, Apache JMeter also supports functional, regression, and stress tests on different protocols. It’s a free open-source tool with custom scripting functionalities that require advanced programming skills.
Using JMeter for testing, Source: Saurabh AgarwalJMeter advantages. JMeter's pluggable timers allow for simulating various types of loads. Its plugins add extensibility and customization, while built-in functions enable dynamic inputs to a test. Thanks to its efficient UI design and lightweight component support, JMeter executes tests fast and provides accurate timing. It caches test results and data providing their offline replay and analysis.
Oleksandr Fomenko, a Senior Test Automation Engineer at EPAM Systems, chose JMeter to measure the API performance for its convenient way of multi-threading emulation. “You can create test cases using JMeter’s record and playback technology, extract and reuse data from response, verify it using JSON Assertion, debug it, and build a comprehensive report. With these functions in stock, JMeter can be easily used as a stand-alone framework for testing REST APIs.”
RestSharp: simple RESTful API testing in C#Available as a NuGet package, RestSharp is a popular and convenient open-source HTTP client library for . NET. It’s easy to install and add to your C# project. RestSharp parses both XML and JSON formats.
API test using RestSharp: Check whether the HTTP GET call returns status code 200, Source: BasRestSharp advantages. RestSharp’s functionality allows for straightforward test creation, serialization, and deserialization. It can handle synchronous and asynchronous requests with a wide list of ready-made authenticators. Uploading files and forms in multiple parts, RestSharp cuts down on upload times.
EPAM’s Oleksandr Fomenko highlights its powerful features and considers RestSharp the best solution for processing HTTP requests. He also accents its clean implementation: “RestSharp allows you to easily implement any type of HTTP client.”
SoapUI and ReadyAPI by SmartBear: end-to-end testing tools for REST, SOAP, GraphQL, and other web servicesSoapUI is an open-source Java-based API testing tool by SmartBear for functional, security, load, and interoperability testing of SOAP, REST, GraphQL, and JMS (Java Message Service) web services.
Its commercial version - ReadyAPI (former SoapUI Pro) - covers more types of functionality tests (load, stress, and spike). It also supports the validation of event-driven microservices powered by Apache Kafka. Ready API comes pre-integrated with Git, Docker, Jenkins, TeamCity, and other popular DevOps tools. (We have already described another SmartBear product used for UI testing - TestComplete - in our article on automated testing tools)
SoapUI and ReadyAPI advantages. Both open-source and pro tools feature an easy-to-use drag-and-drop interface enabling scriptless creation and running of even the most complex test scenarios. They cover the full API testing spectrum. Yet, ReadyAPI has more extensive functionality including reporting and analytics. Also, as we mentioned before, it comes with out-of-the-box plugins for popular CI/CD tools.
ReadyAPI pricing plans. ReadyAPI offers three types of annual subscriptions called modules, each covering a different set of features:
- API Test Module starts at €755 (around $796) and includes end-to-end functional and security testing. It also gives access to a limited number of functional testing and virtualization instruments;
- API Performance Module starts at €6,010 (around $6,339). It you to validate the speed, scalability, and performance of APIs, servers, and network resources. It doesn’t include automated security testing, endpoint scanning, and some other instruments available in the API Test Module;
- API Virtualization Module starts at €1,131 (around $1,192). It allows you to virtualize critical APIs to reduce dependencies and delays throughout the delivery cycle. At the same time, the module lacks many testing features.
Katalon Studio: an automation tool with an inbuilt API testing modeDesigned primarily for UI testing, Katalon is also an integrated environment meant to easily generate and execute API tests.
Katalon advantages. Its edge over other automation tools like Selenium or Ranorex is that it doesn’t need additional integrations to test APIs. Katalon has an inbuilt API testing mode with the following capabilities:
- code auto-completion, inspection, and snippeting;
- references and debugger;
- inbuilt response viewer with auto-formatting;
- search for access to artifacts; and
- tests importing from API testing and editing tools (e.g. Swagger, Postman, WSDL).
Katalon dashboard for testing APIKatalon pricing. There are three pricing plans:
- Free - for learning and experimentation. It covers 2,000 monthly test results;
- Premium - for production-grade testing workloads (from $25 monthly); and
- Enterprise - for large companies with mission-critical testing workloads, producing over 30,000 test results a month. Contact Katalon Studio for more pricing information.
API testing framework vs ready-made API testing toolChoosing among the tools we described, keep in mind that some of them are better for writing an automation framework (RestSharp, REST Assured, and Requests), while other tools (Postman, SoapUI, Katalon, and JMeter) can be utilized as they are.
So, which path to take?
Writing your own framework gives you more freedom as you’re not limited to the capabilities of a tool and its existing plugins. You can customize it your way: Add any library suitable for the chosen coding platform, introduce complicated logic in your tests, and build a reporting standard convenient to you. You also get more flexibility in managing your test data. But you enjoy these perks only if you possess strong coding skills.
Otherwise, you’d better look towards a ready-made tool. Its user-friendly interface and low-threshold coding requirements will be a relief for you. The pitfall here is that they usually address general API testing issues. But the good thing is that their community can easily help you solve your more-or-less common problems.
To sum up, a ready-made tool like Postman is good for starters. Developing your own automation framework is a more advanced level, but it’s also rewarding in terms of functionality you can add to your testing solution.