Introduction

Flaky Score is an AI-driven feature that evaluates the stability of test cases by analyzing their execution history. It indicates the likelihood of a test case behaving unpredictably, helping testers assess the risk associated with its execution.

Flaky test cases can disrupt testing pipelines and undermine confidence in test results. Traditionally, identifying such tests required manual comparison of execution results across multiple runs. With a flaky score, this process is now automated, enabling teams to detect and manage flaky tests more efficiently and reliably.

QA Managers can define and configure settings according to their specific testing processes for calculating the flaky score, ensuring its relevance to their testing methodologies.

For example, the following table shows the execution results of the test cases executed multiple times.

Flaky Score Instance Level Settings

To activate the Flaky Score feature, the super administrator needs to enable it for the QMetry instance.

To enable the flaky score, go to Customizations, and select General Settings and Audit. Next, select AI Configurations. Navigate to the Flaky Score and Success Rate Configuration section. By default, this section is disabled.

Enable the Allow Project Admins to Enable and Configure the Flaky Score for their project settings for Flaky Score calculations.

Open

Flaky Score Project Level Settings

To enable flaky score, users must enable these settings for individual projects in the QTM instance.

To enable project level settings for flaky score, perform the following steps:

1. Go to QMetry Intelligence.

2. Select Flaky Score Settings.

3. Open the Configurations tab.

4. Enable Flaky Score for the Project.

Open

Flaky Score Configuration

The settings on this page enable the admin to define the scope of test executions considered for Flaky Score calculation. AI will calculate the Flaky Score based on the filtered test execution records according to the criteria. The existing Flaky Score will be reset and recalculated whenever the configuration is modified.

Provide the following details for the Flaky Score rate configuration:

• Test Case Executions: Mention the number of the latest test executions for a test case you want to consider while calculating the flaky score. 10 test executions are selected by default. The Test Case Execution value should be from 1 to 1000.

• Test Case Versions: Select the executions for either the “Latest Version” or “All Version” test case versions to consider while calculating the flaky score. The “Latest Version” test case version is selected by default.

  • Latest Version: The test executions of only the latest test case versions will be considered while calculating the flaky score.

  • All Version: The latest test executions of all test case versions will be considered while calculating the flaky score.

• Execution Timeframe (days): Mention the days to define the timeframe in which the test executions were executed in the last specified days. 90 days are selected by default, which indicates the test executions carried out during the previous 90 days will be considered while calculating the flaky score. If left blank, it will be considered as all the days.

• Release: Select Release(s) to consider test executions for those releases. “All” releases are selected by default.

• Cycle: Select Cycle(s) to consider test executions for those cycles. “All” cycles are selected by default.

• Platform: Select Platform(s) to consider test executions for those platforms. “All” platforms are selected by default.

• Build: Select build(s) to consider the test executions pertaining to that particular build(s). “All” builds are selected by default.

• Test Case Execution Status (All execution statuses other than selected as passed and failed will be considered as Skipped. Skipped status will not be considered for calculation).

  • As Passed: The test execution with these statuses will be considered as "Passed" while calculating the Flaky Score. The default value is “Passed”.

  • As Failed: The test executions with these statuses will be considered as "Failed" while calculating the Flaky Score. The default values are “Failed” and “Blocked”.

Click Update to save the Flaky Score Settings.

To reset the settings, click on the Reset button.

To generate the flaky score, click on the Generate button.

The success message pops up.

When you hover over the Generate button, it shows the details of the user who generated Flaky Score last with the timestamp.

Open

Once you generate the Flaky Score on the Flaky Score Settings screen, you can view the Flaky Score on the test case list view, test case details screen, test case link screens, and test execution screens. You can show or hide the Flaky Score column on the test case list view, test case details screen, test case link screen, and test execution screens.

Display of Flaky Score

The lesser the value (closer to 0), the test case has less flakiness. It means the behavior is deterministic.

The larger the value (closer to 1), the test case has more flakiness. It means the behavior is non-deterministic.

The following table interprets the intensity of the flaky score in accordance with its derived calculation and color code.

Filter Test Cases on Flaky Score

The test cases can be filtered on Flaky Score. You can find the Flaky Score under the Advanced Filters.

Open

View Flaky Score on Test Case List View

As per the Flaky Score Settings done in the Configuration, the Flaky Score is calculated and displayed on the test case list view. It allows QA managers and Testers to view the risk probability and test execution history.     

Go to the Test Cases module and make the Flaky Score column visible for the list view.

• Flaky Score: It is calculated based on pass or fail results frequency.

Open

You can see the Flaky Score column with corresponding statistics. You can sort on the column to view test cases with higher flaky scores.

Open

Click on the Flaky Score to view the traceability report for the test case, which shows the test case is associated with which test executions, issues, and execution results. The report helps you analyze the Flaky Score further.

Open

If the Flaky Score Settings are changed, the existing Flaky Score will get reset.

The info icon beside the Flaky Score column displays the details of when the score was last generated and by whom.

Open

View Flaky Score on Test Case Detail Page

The generated Flaky Score is displayed beside the Test Case Key at the top of the screen.

Open

View Flaky Score on Test Case Link Screens

You can show or hide the Flaky Score column on the Link Test Cases screen.

Open

View Flaky Score on Test Execution Screen

As per the Flaky Score Settings done in the Configuration, the Flaky Score is calculated and displayed accordingly. The flaky score shows the tester the probability of risk while executing the test case and provides a way for comparing pre and post-test execution results.

Go to the Test Execution screen.

Test Execution Detail View

Open

If you go further into the Detail View, you can view the Flaky Score count at the top of the page.

Open

Test Execution Default View

You can show or hide the Flaky Score column on the execution screen.

Open

Open

Click on the Flaky Score to drill down to view the test executions and defects associated with the test case.

How to Reduce Flakiness?

Flaky tests can occur due to various reasons. Testers should work with their development teams to find the exact reason for the cause. Here are the 10 common causes of flaky tests:

1. Async wait:

Some tests are written in a way that requires waiting for something else to complete. Many flaky tests use sleep statements for this purpose. However, sleep statements are imprecise, and the test may fail if the waiting time exceeds expectations due to variations in processing time.

2. Concurrency issues:

Flaky tests can result from concurrency issues such as data races, atomicity violations, or deadlocks. These tests often make incorrect assumptions about the ordering of operations performed by different threads. To address this, synchronization blocks can be added or the test can be modified to accommodate a wider range of behaviors.

3. Test order dependency:

Certain tests may pass or fail depending on the order in which preceding tests were executed. A good test should be independent and able to run in any order with other tests. It should be properly isolated and set up its own expected state.

4. Timing issues:

Flaky tests can arise from timing inconsistencies when the test code relies on specific event timings. For example, if a test checks for a particular webpage element after a specific delay, network issues or differences in CPU performance between test runs can lead to intermittent failures.

5. Element locator issues:

Many automation tools use XPath to locate webpage elements, but XPath can be unstable as it is sensitive to changes in the page's DOM. Self-healing AI techniques can address challenging testing scenarios involving dynamic elements, iFrames, and pop-ups. This involves using multiple locator strategies to find an element, and switching to a backup strategy if the primary one fails. Modifications to an element's properties or the addition of similar elements can render the initial XPath invalid, resulting in false positives or negatives.

6. Test code issues:

Poorly written or ambiguous test code can contribute to flaky tests. If the test code lacks clarity regarding the expected application behavior, the test may inconsistently fail or pass. Additionally, complex test code or code relying on external dependencies may be more prone to failure.

7. Test data issues:

Tests that depend on inconsistent test data can become flaky. Corrupted test data or different test runs using the same data can lead to inconsistent results. Tests utilizing random data generators without considering the full range of possible results can also introduce flakiness. It is advisable to control the seed for random data generation and carefully consider all possible values.

8. Test configuration issues:

Inconsistent test configurations between runs can cause flaky tests. Incorrect test parameters or improper test settings setup can result in test failures.

9. Environment issues:

Flaky tests can be attributed to the execution environment. Network connectivity problems, improper handling of I/O operations, hardware differences between test runs, or variations in test environments can introduce nondeterminism, leading to flaky tests.

10. Resource leaks: Tests can become flaky if the application or tests do not adequately acquire and release resources, such as memory or database connections. To avoid such issues, it is recommended to use resource pools and ensure that resources are properly returned to the pool when no longer needed.