azure - Cypress tests fail to run in parallel mode due to mismatched specs between different runs - Stack Overflow

admin管理员组

文章数量:1431994

We're encountering an issue when running Cypress tests in parallel mode. Specifically, when launching different sets of tests with different tags (like smoke and regression), the system reports that the specs (test files) are mismatched and this causes the test run to fail.

Below are the commands we are using:

1. For Smoke tests:

"test:smoke": "cypress run --browser chrome --record --key <key> --parallel --ci-build-id $BUILD_BUILDNUMBER --env grepTags=\"smoke\",grepFilterSpecs=true",

2. For Regression tests:

"test": "cypress run --browser chrome --record --key <key> --parallel --ci-build-id $BUILD_BUILDNUMBER --env grepTags=\"regression\",grepFilterSpecs=true",

The issue

Despite specifying separate test tags, Cypress Cloud claims that the specs do not match between the runs, and this leads to errors. We are unable to run tests in parallel for different test sets.

What we need help with:

• Why is Cypress Cloud comparing our test specs across different runs, even when we’re explicitly using different tags for each set of tests?
• ow can we resolve this issue and successfully run parallel tests for different sets of tests?

Error Log for Regression (but for smoke okay):

In order to run in parallel mode each machine must send identical environment parameters such as:

 - specs
 - osName
 - osVersion
 - browserName
 - browserVersion (major)

This machine sent the following parameters:

{
  "osName": "linux",
  "osVersion": "Ubuntu - ",
  "browserName": "Chrome",
  "browserVersion": "114.0.5735.198",
  "differentSpecs": {
    "added": [
     .....
    ],
    "missing": [
     ....
    ]
  }
}

What we've tried so far:

• Using different Cypress keys for each test group (smoke and regression).
• Assigning unique ci-build-id values for every run.
• Ensuring that all environment configurations and browser versions are consistent across all machines.

We're encountering an issue when running Cypress tests in parallel mode. Specifically, when launching different sets of tests with different tags (like smoke and regression), the system reports that the specs (test files) are mismatched and this causes the test run to fail.

Below are the commands we are using:

1. For Smoke tests:

"test:smoke": "cypress run --browser chrome --record --key <key> --parallel --ci-build-id $BUILD_BUILDNUMBER --env grepTags=\"smoke\",grepFilterSpecs=true",

2. For Regression tests:

"test": "cypress run --browser chrome --record --key <key> --parallel --ci-build-id $BUILD_BUILDNUMBER --env grepTags=\"regression\",grepFilterSpecs=true",

The issue

Despite specifying separate test tags, Cypress Cloud claims that the specs do not match between the runs, and this leads to errors. We are unable to run tests in parallel for different test sets.

What we need help with:

• Why is Cypress Cloud comparing our test specs across different runs, even when we’re explicitly using different tags for each set of tests?
• ow can we resolve this issue and successfully run parallel tests for different sets of tests?

Error Log for Regression (but for smoke okay):

In order to run in parallel mode each machine must send identical environment parameters such as:

 - specs
 - osName
 - osVersion
 - browserName
 - browserVersion (major)

This machine sent the following parameters:

{
  "osName": "linux",
  "osVersion": "Ubuntu - ",
  "browserName": "Chrome",
  "browserVersion": "114.0.5735.198",
  "differentSpecs": {
    "added": [
     .....
    ],
    "missing": [
     ....
    ]
  }
}

What we've tried so far:

• Using different Cypress keys for each test group (smoke and regression).
• Assigning unique ci-build-id values for every run.
• Ensuring that all environment configurations and browser versions are consistent across all machines.

• azure
• parallel-processing
• azure-pipelines
• cypress
• cicd

Share Improve this question Follow asked Nov 19, 2024 at 13:12 Albert AlovAlbert Alov 2311 silver badge22 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 6

The Cypress cloud is configured by project. Although you consider your repository to be one project and running tests by tag to be two aspects of the same project, the Cypress Cloud project is not conceptually the same.

In Cypress Cloud a project is a series of runs against the same grouping of tests.

Further, there is no configuration to record by tag - you must configure two different projects in order to perform two different runs.

So, to obtain consistent statistics you must treat the two tagged runs as separate projects in the Cloud.

The reference for cloud "project" setup is here.

To handle the separate projectId's for each tag, some code is required in the setup file

const projectsByTag = {
  'smoke': 'project-id-for-smoke-tests',           // e.g '4b7344',
  'regression': 'project-id-for-regression-tests',
  'default': 'project-id-when-not-passing-a-tag'
}

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      const tag = config.env.getTags
      config.projectId = projectsByTag[tag] || projectsByTag.default
      return config
    }
  }
})

本文标签：