Could ChatGPT be a tester?
Have you ever wondered if a machine, like ChatGPT, could replace a human tester? Can ChatGPT effectively design and implement automated tests? I wanted to find out for myself, so I gave ChatGPT (the Jan9 version) a simple task. This post will share what I learned.
WHAT I WAS TRYING TO CHECK?
The idea was straightforward: First, I provided ChatGPT with a short description of software and asked for test cases. Then, I asked ChatGPT to prepare test automation. Finally, I checked if the output was correct.
For the software example, I chose an e-commerce shop based on Magento. It is a very popular software based on PHP, and I assumed that ChatGPT would have a general understanding of such software, so a detailed description would not be necessary.
Regarding the testing code, my expectations were:
- ChatGPT should propose test cases based on the general description of the project.
- ChatGPT should propose some Java code.
- Tests should use Selenium, Cucumber, and Allure.
- Tests should be run from Jenkins.
Let’s see how it turned out.
CONVERSATION WITH CHATGPT
Question:
Assuming that I have an e-commerce shop based on Magento, propose some test conditions and test cases and write it down in Gherkin language. Assume that the shop has versions for several countries (Poland, Germany, United Kingdom) and there are specific methods of payment, methods of delivery for each country.
Answer:
Test Conditions:
* The e-commerce shop is based on Magento.
* The shop has versions for several countries (Poland, Germany, United Kingdom).
* There are specific methods of payment and methods of delivery for each country.
Test Cases (Gherkin language):
Scenario: Check available methods of payment
Given the user is on the “Payment Method” page
When the user selects “Poland” as the country
Then the available methods of payment should be “Credit Card” and “PayPal”Scenario: Check available methods of delivery
Given the user is on the “Shipping Method” page
When the user selects “Germany” as the country
Then the available methods of delivery should be “Standard Delivery” and “Express Delivery”Scenario: Check order confirmation
Given the user has completed the checkout process
When the user selects “United Kingdom” as the country
Then the order confirmation should display the correct selected payment method and delivery method for “United Kingdom”Scenario: Check if the user can place order with different currency
Given the user is on the “Payment Method” page
When the user selects “Poland” as the country and the currency is “PLN”
Then the user should be able to place the order with “PLN” currencyScenario: Check if the user can place order with different language
Given the user is on the “Payment Method” page
When the user selects “Poland” as the country and the language is “Polish”
Then the user should be able to place the order with “Polish” languageNote: Gherkin is a tool used to write acceptance test, this is just an example and you may need to adjust based on your actual requirement.
I was not satisfied with the proposals I received, so I attempted to generate something more sophisticated.
I would like to add that, according to ISTQB terminology, a test condition is an item or event of a component or system that can be verified by one or more test cases, such as a function, transaction, feature, quality attribute, or structural element. I had expected test conditions such as logging into the shop, account creation, order cancellation etc. In this context, “test condition” was understood as a kind of assumption.
Additionally, the scenarios were not very detailed. I had asked for something more elaborate.
Question:
Ad at least one end-to-end test from entering the shop page to the confirmation of successful payment for ordered good.
Answer:
Test Case (Gherkin language):
Scenario: Place an order and confirm successful payment
Given the user is on the “Home” page of the e-commerce shop
When the user clicks on a product and adds it to the cart
And the user navigates to the cart and proceeds to checkout
And the user fills in all the required information including shipping address and payment method
And the user submits the order
Then the user should be redirected to the payment gateway
And the user should enter the correct payment details
And the user should be redirected to the “Order Confirmation” page
And the order confirmation message should include “Payment Successful”
And the user should receive an email confirmation of the orderNote: You may need to adjust based on your actual requirement and it’s advisable to test it on different browser and devices.
Still not fully satisfied, but not giving up.
Question:
Change it into Scenario Outline and an Examples section.
Answer:
Test Case (Gherkin language):
Scenario Outline: Place an order and confirm successful payment
Given the user is on the “Home” page of the e-commerce shop
When the user clicks on a product and adds it to the cart
And the user navigates to the cart and proceeds to checkout
And the user fills in all the required information including shipping address and payment method \
And the user submits the order
Then the user should be redirected to the payment gateway
And the user should enter the correct payment details
And the user should be redirected to the “Order Confirmation” page
And the order confirmation message should include “Payment Successful”
And the user should receive an email confirmation of the orderExamples:
| payment_method |
| Credit Card |
| PayPal |
| Bank Transfer |Note: You can add different examples for different payment methods that your shop supports. And you may also want to cover more test cases, like covering different shipping address, different products, different prices and different languages.
Question:
Propose some Java code to implement a framework with Cucumber, Selenium with Chromedriver and Allure reporting. Assume that the Chromedriver is already installed. At this moment do not implement above steps but prepare some Java class for it.
Answer:
Here’s some sample code to demonstrate how you might implement a framework using Cucumber, Selenium with Chromedriver, and Allure reporting in Java:
1. Cucumber Test Runner: A JUnit runner class that runs Cucumber features.
import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;
@RunWith(Cucumber.class)
@CucumberOptions(
plugin = {"pretty", "html:target/cucumber-html-report", "json:target/cucumber.json", "allure:target/allure-results"}
features = "src/test/resources/features",
glue = "com/yourcompany/tests",
tags = "@place_order"
)
public class CucumberTestRunner {
}2. Step Definition: A class that implements the step definitions for the Cucumber feature.
import io.cucumber.java.en.Given;
import io.cucumber.java.en.When;
import io.cucumber.java.en.Then;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
public class StepDefinitions {
private WebDriver driver;
@Given("the user is on the Home page of the e-commerce shop")
public void navigateToHomePage() {
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
driver = new ChromeDriver(options);
driver.get("https://yourshop.com");
}
@When("the user clicks on a product and adds it to the cart")
public void addProductToCart() {
// Code to add a product to the cart
}
@When("the user navigates to the cart and proceeds to checkout")
public void proceedToCheckout() {
// Code to navigate to the cart and proceed to checkout
}
@When("the user fills in all the required information including shipping address and payment method {string}")
public void fillInShippingAndPaymentInfo(String paymentMethod) {
// Code to fill in shipping and payment information
}
@When("the user submits the order")
public void submitOrder() {
// Code to submit the order
}
@Then("the user should be redirected to the payment gateway")
public void verifyRedirectToPaymentGateway() {
// Code to verify that the user is redirected to the payment gateway
}
@Then("the user should enter the correct payment details")
public void enterPaymentDetails() {
// Code to enter payment details
}
@Then("the user should be redirected to the Order Confirmation page")
public void verifyRedirectToOrderConfirmationPage() {
// Code to verify that the user is redirected to the Order Confirmation page
}
@Then("the order confirmation message should include Payment Successful")
public void verifyPaymentSuccessful() {
// Code to verify that the payment was successful
}
@Then("the user should receive an email confirmation of the order")
public void verifyEmailConfirmation() {
// Code to verify that the user receives an email confirmation of the order
}
@After
public void tearDown() {
driver.quit();
}
}3. Allure configuration: A class that configures Allure reporting.
import io.qameta.allure.Allure;
import io.qameta.allure.Attachment;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
public class AllureConfig {
@Attachment(value = "Screenshot", type = "image/png")
public static byte[] takeScreenshot(WebDriver driver) {
return ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES);
}
public static void addScreenshot(WebDriver driver) {
Allure.addAttachment("Screenshot", "image/png", takeScreenshot(driver));
}
}Question:
Propose how to integrate the project with Jenkins.
Answer:
To integrate your project with Jenkins, you can follow these steps:
1. Install the Jenkins CI server on your machine or use a Jenkins instance hosted on a remote server.
2. Install the following Jenkins plugins:
* Cucumber Reports plugin
* Allure Jenkins plugin3. Create a new Jenkins job for your project.
4. In the “Source Code Management” section, configure your repository (e.g. Git, SVN) and provide the URL to your project’s repository.
5. In the “Build Triggers” section, configure your build triggers (e.g. build periodically, build when a change is pushed to the repository)
6. In the “Build” section, add a “Invoke top-level Maven targets” build step. In the “Goals” field, specify “clean test” to run your tests.
7. In the “Post-build Actions” section, add the “Publish Cucumber test result report” build step and provide the path to your Cucumber report files (e.g. target/cucumber.json).
8. Add “Allure Report” build step and provide the path to allure-results (e.g target/allure-results).
9. Save the job and trigger a build.
10. Once the build is complete, you can view the Cucumber and Allure reports by clicking on the “Cucumber Reports” and “Allure Report” links in the Jenkins build dashboard.
Note: You may need to adjust the steps based on your actual environment and configuration. Also, you can use Jenkins pipeline to run the test, it provides more flexibility and scalability.
In fact, Cucumber reporting plugin is not necessary.
Question:
Propose some Jenkinsfile for it.
Answer:
Here’s an example of a Jenkinsfile that you can use to run your Cucumber tests, generate the Allure report, and publish it to Jenkins:
pipeline {
agent {
label 'your_label'
}
stages {
stage('Build') {
steps {
sh 'mvn clean test'
}
}
stage('Allure Report') {
steps {
sh 'allure generate -c target/allure-results'
allure([includeProperties: false, jdk: '', properties: [], reportBuildPolicy: 'ALWAYS', results: [[path: 'target/allure-results']]])
}
}
}
}This Jenkinsfile uses the
pipelinesyntax and has two stages:BuildandAllure Report. TheBuildstage runs themvn clean testcommand to run your tests and generate the Cucumber and Allure reports. TheAllure Reportstage generates the Allure report and publishes it to Jenkins.You can also add ‘Publish Cucumber test result report’ build step in the Jenkinsfile to publish cucumber report to Jenkins.
post {
always {
cucumber results: 'target/cucumber.json'
}
}You can also add ‘Publish JUnit test result report’ build step in the Jenkinsfile to publish JUnit report to Jenkins.
post {
always {
junit 'target/surefire-reports/*.xml'
}
}Note: You may need to adjust the path to the report files based on your project structure and you can also add more stages for different test cases. This is just an example and you may need to adjust based on your actual requirement, you can use different libraries such as gradle, maven, sbt etc.
Question:
Where to place above files? What should be the structure of directories?
Answer:
Here’s an example of a directory structure that you can use to organize your project files:
- project_root
- src
- main
- java
- com
- yourcompany
- pageobjects (contains page objects classes)
- steps (contains step definition classes)
- utils (contains utility classes)
- test
- java
- com
- yourcompany
- runners (contains the Cucumber test runner class)
- allure (contains Allure configuration class)
- resources
- features (contains Cucumber feature files)
- target (contains the test results, reports, and allure-results)
- Jenkinsfile
- pom.xml (or build.gradle)The
srcdirectory contains the main source code for the project, including the page objects, step definitions, and utility classes. Thesrc/testdirectory contains the test source code, including the Cucumber test runner and Allure configuration classes, as well as the Cucumber feature files. Thetargetdirectory contains the test results, reports, and allure-results. The Jenkinsfile should be placed in the root directory of the project, and the pom.xml or build.gradle should be also at the root directory of the project.It’s a good practice to keep the project directory structure and file organization clean, consistent, and easy to navigate. This makes it easier to maintain and understand the project. Also, it’s important to keep the test codes separate from the main codes and resources for a better management and maintenance.
Question:
Propose pom.xml
Answer:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.yourcompany</groupId>
<artifactId>yourproject</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<!-- Selenium -->
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
<!-- Cucumber -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java</artifactId>
<version>6.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>6.8.1</version>
<scope>test</scope>
</dependency>
<!-- Allure -->
<dependency>
<groupId>io.qameta.allure</groupId>
<artifactId>allure-cucumber4-jvm</artifactId>
<version>2.13.5</version>
<scope>test</scope>
</dependency>
<!-- JUnit -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<!-- Cucumber reporting -->
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>4.9.0</version>
<executions>
<execution>
<id>execution</id>
<phase>verify</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<projectName>Your Project</projectName>
<outputDirectory>target/cucumber-html-report</outputDirectory>
<cucumberOutput>target/cucumber.json</cucumberOutput>
<skippedFails>true</skippedFails>
<enableFlashCharts>false</enableFlashCharts>
<buildNumber>${BUILD_NUMBER}</buildNumber>
</configuration>
</execution>
</executions>
</plugin>
<!-- Allure -->
<plugin>
<groupId>io.qameta.allure</groupId>
<artifactId>allure-maven</artifactId>
<version>2.13.5</version>
</plugin>
</plugins>
</build>
</project>This pom.xml file configures the dependencies for Selenium, Cucumber, JUnit, Allure and Cucumber reporting plugins. It also configures the plugin to generate cucumber html report and Allure report after the test is run.
Please note that the version of the dependencies may be different based on the latest version.
You can add more dependencies based on your requirements, such as dependency
PUTTING EVERYTHING IN PLACE — HOW MUCH HUMAN INTERVENTION IS STILL REQUIRED?
I attempted to follow the guidance provided. I assumed a basic knowledge of Java, so I did not ask about the names of files, packages, etc. My first choice was Visual Studio Code because it allows for easy editing of the directory structure. The name of the class is enough to determine the filename. After pasting the contents, the package is automatically proposed by the IDE.
The next thing I noticed was that the IDE did not provide a “Quick Fix” option when it showed a lack of constructor. I switched to IntelliJ because it has better integration with Maven and better “Quick Fixes” options.
I encountered an issue with the plugin allure-maven. ChatGPT proposed version 2.13.5, which does not currently exist. The last version is 2.12.0. Sometimes ChatGPT tries to propose something that does not exist in any data. However, finding the last version is not especially difficult, even for beginners.
In the proposed Gherkin scenarios, there was no Feature keyword.
When it comes to the language level, the IDE again helped and proposed changes in pom.xml.
The next problem I encountered was in the AllureConfig.java code. Compiler returned
incompatible types: byte[] cannot be converted to java.lang.StringI tried to ask ChatGPT to fix it, but the provided code had even more issues. However, it also proposed the following code for testing the ability to make screenshots.
@Then("the order confirmation message should include Payment Successful")
public void verifyPaymentSuccessful() {
if (!isPaymentSuccessful) {
AllureConfig.addScreenshot(driver);
Assert.fail("Payment was not successful");
}
}Upon seeing this, the IDE suggested “Move ‘StepDefinitions’ to test root’”. I did so and I added import of org.junit.Assert andcom.yourcompany.allure.AllureConfig.
The next step was fixing quotation marks in the StepDefinitions and feature file. This is not difficult, but the IDE does not provide any hints, so it requires some knowledge about the Cucumber framework.
The entry "allure:target/allure-results" in the CucumberTestRunner caused an error.
The next issue was that the runner did not see any tests, so each execution ended with nothing. Why? The reason for this was that the glue location was described as "com/yourcompany/tests" instead of "com.yourcompany.tests".
Another issue was that the teardown was not properly placed, so the browser was not closing after execution.
The teardown code needs to be moved to the runner, and the webdriver variable should be passed properly.
The next problem could be tricky for beginners. There was no maven-surefire-plugin in the pom.xml in the build section, so Maven could not see the prepared tests. The following code should be added.
<plugin>
<groupid>org.apache.maven.plugins</groupid>
<artifactid>maven-surefire-plugin</artifactid>
<version>2.22.2</version>
<configuration>
<includes>
<include>**/CucumberTestRunner.java</include>
</includes>
<testfailureignore>true</testfailureignore>
</configuration>
</plugin>Also, there was a tricky problem where in the Cucumber scenario, there was no tag @place_order just before the Scenario outline. If someone missed it, they would spend a long time wondering why it wasn't working.
I can say that none of the proposed pieces of code were fully correct.
It is likely that a real attempt to start a testing project with ChatGPT would end with searching StackOverflow and would not be faster than preparing it from scratch.
SUMMARY
The generated code was seriosly misleading. ChatGPT is not yet able to provide correct code and an inexperienced tester is not able to use it without intensive analysis, which may be even more challenging than writing everything from scratch. However, ChatGPT can still be useful for certain tasks such as refactoring or generating specific pieces of code. It is not a replacement for human testers, but rather an additional tool that can aid experienced testers in their work. Beginners should rather start by implementing their tests step-by-step from scratch and learning how specific parts of the code work.
The story was originally created by me, but it may contain parts that were created with AI assistance. My original text has been corrected and partially rephrased by Chat Generative Pre-trained Transformer to improve the language.
