Executing Playwright Test Scripts with SpiraTest

Reporters in Playwright

Playwright Test comes with a few built-in reporters for different needs and ability to provide custom reporters. The easiest way to try out built-in reporters is to pass --reporter command line option.

npx playwright test --reporter=line

For more control, you can specify reporters programmatically in the configuration file.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: 'line',
});

Multiple reporters

You can use multiple reporters at the same time. For example you can use 'list' for nice terminal output and 'json' to get a comprehensive json file with the test results.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['list'],
    ['json', {  outputFile: 'test-results.json' }]
  ],
});

Reporters on CI

You can use different reporters locally and on CI. For example, using concise 'dot' reporter avoids too much output. This is the default on CI.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Concise 'dot' for CI, default 'list' when running locally
  reporter: process.env.CI ? 'dot' : 'list',
});

JUnit reporter

The JUnit Playwright reporter produces a JUnit-style xml report.

Most likely you want to write the report to an xml file. When running with --reporter=junit, use PLAYWRIGHT_JUNIT_OUTPUT_NAME environment variable:

PLAYWRIGHT_JUNIT_OUTPUT_NAME=results.xml npx playwright test --reporter=junit

In configuration file, pass options directly:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [['junit', { outputFile: 'results.xml' }]],
});

JUnit report supports following configuration options and environment variables:

Using the Playwright JUnit Reporter with Spira

Assuming that you have configured Playwright to use the JUnit reporter as described above, the next step is to then run the special Spira xUnit report file importer to read the contents of the file, and report back to Spira:

Configuring the Spira connection

In your test root folder (the folder you have your Playwright tests), create a file named spira.cfg with the following:

[credentials]
# Following are required
url = http://localhost/spira
username = administrator
token = {XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX}}
project_id = 1
# Following are optional:
release_id = 5
test_set_id = 1
create_build = true

# Spira Test case for a specific classname.name
[test_cases]
LIS.Registration.registration1 = 2
LIS.Registration.registration2 = 3
LIS.Registration.registration3 = 4
LIS.Authentication.Login.login1 = 5
LIS.Authentication.Login.login2 = 9
LIS.Authentication.Login.login3 = 8
LIS.Authentication.auth1 = 6
LIS.Authentication.auth2 = 12
LIS.Authentication.auth3 = 13

LIS.Registration.registration4 = 5
LIS.Registration.registration5 = 9
LIS.Registration.registration6 = 8
LIS.Registration.registration7 = 6
LIS.Registration.registration8 = 12

# Spira Test sets for a specific name
# If not, the global value is used instead
[test_sets]
LIS.Registration = 2
LIS.Authentication = 5
LIS.Authentication.Login = 2

For the plugin to work, you must have the following in the credentials group:

• url -- The base url to your Spira installation, without a '/' at the end.
• username -- The username you use to sign into Spira.
• token -- Your API Key / RSS Token. Found in your profile page as the "RSS Token" field, you must have RSS Feeds enabled for this to work.
• project_id -- The ID of the project you would like the test runs to be sent to

In addition, you can set the following optional configuration parameters as well:

• release_id -- Use if you would like to associate the test run with a release.
• test_set_id -- Use if you would like to associate the test run with a default test set.
• create_build -- Set to true if you would like the plugin to create a new Spira build artifact with every run that all the individual test case runs get associated with.

Mapping The Test Cases

This section is required, and is where you map the classname.name of the test case in the JUnit XML report file to the appropriate test case in Spira. For details of which attribute is needed from the XML file, please refer to the sample XML files included at the end of this file.

• classname.name - Used to map the combination of the test case's classname and name to the corresponding test case ID in Spira. The ID in Spira should be the test case id TC:xxx without the TC prefix.

Mapping The Test Sets [Optional]

This section is optional, and is used when you want the different test suites in the XML file to map to different test sets in Spira. If you don't complete this section, all of the test results will be associated with the test_set_id specified in the main configuration section.

In this section you map the name of the test suite in the xUnit XML file to the appropriate test set in Spira. For details of which attribute is needed from the XML file, please refer to the sample XML files included at the end of this file.

• name - Used to map the test set's name to the corresponding test set ID in Spira. The ID in Spira should be the test set id TX:xxx without the TX prefix.

Executing the Tests

Now you are ready to execute your tests and send the results back so Spira. This happens in two steps:

• Execute the unit tests and generate the results.xml JUnit style report file (described above)
• Parse the results.xml report file and send the results to Spira

For the second step, you need to run the Spira results parser module spira_xunit_reader.py to upload the results to Spira. For example, with the results.xml output file generated by the Playwright JUnit reporter:

python spira_xunit_reader.py results.xml spira.cfg

OR

python spira_xunit_reader.py samples\junit-complete.xml spira.cfg

The second parameter is the location and name of the Spira configuration file spira.cfg. One you run the XML parser, you should see a message similar to the following:

Sending test results to Spira at URL 'https://myserver/spiraservice.net'.
Successfully reported 4 test cases to Spira.

If there are any errors or warnings, they will be displayed instead.