Flaky Test Detection in Jest

Flaky tests are a significant pain point in software development – tests that sometimes pass and sometimes fail without any code changes. This challenge asks you to implement a mechanism within a Jest testing environment to detect and report flaky tests, helping teams identify and address these unreliable tests. Detecting flaky tests improves test suite reliability and reduces debugging time.

Problem Description

You need to create a Jest reporter that identifies flaky tests. A flaky test is defined as a test that fails at least once and passes at least once within a single test run. The reporter should:

1. Track Test Results: Monitor the results (pass/fail) of each test within a Jest test run.
2. Identify Flaky Tests: Determine which tests have exhibited both passing and failing results during the run.
3. Report Flaky Tests: Generate a clear report at the end of the test run, listing the names of the flaky tests. The report should be displayed in the console.
4. Handle Test Suites: The reporter should correctly handle tests organized within multiple test suites (describe blocks).
5. Avoid False Positives: Ensure that tests that always pass or always fail are not flagged as flaky.

Expected Behavior:

• The reporter should be integrated into the Jest environment as a custom reporter.
• The reporter should collect test results throughout the test run.
• At the end of the run, the reporter should output a list of flaky tests to the console. If no tests are flaky, it should output a message indicating that no flaky tests were found.
• The reporter should not interfere with the standard Jest output.

Edge Cases to Consider:

• Tests that fail on the first attempt and then pass repeatedly should be considered flaky.
• Tests that pass on the first attempt and then fail repeatedly should be considered flaky.
• Tests that are skipped or pending should not be considered flaky.
• Tests that are retried by Jest's retry mechanism should be correctly tracked.
• Tests within nested describe blocks should be correctly identified.

Examples

Example 1:

Input: A test run with the following results:
  - test1: Pass
  - test2: Fail
  - test3: Pass
  - test4: Fail
  - test5: Pass

Output:
Flaky Tests:
  - test2
  - test4

Explanation: test2 failed and passed, and test4 failed and passed. test1, test3, and test5 only passed.

Example 2:

Input: A test run with the following results:
  - test1: Fail
  - test2: Fail
  - test3: Pass
  - test4: Pass
  - test5: Fail

Output:
No flaky tests found.

Explanation: test1, test2, and test5 only failed. test3 and test4 only passed.

Example 3:

Input: A test run with the following results:
  - describe('suite1') {
    - test1: Pass
    - test2: Fail
    - test3: Pass
  }
  - describe('suite2') {
    - test4: Fail
    - test5: Pass
  }

Output:
Flaky Tests:
  - suite1/test2
  - suite1/test3
  - suite2/test4
  - suite2/test5

Explanation: Tests 2, 3, 4, and 5 exhibited both pass and fail states. The output includes the suite name for clarity.

Constraints

• Language: TypeScript
• Framework: Jest
• Performance: The reporter should not significantly impact the overall test run time. Avoid excessive memory usage.
• Reporter Integration: The reporter must be compatible with standard Jest configuration.
• Output Format: The output should be a simple, readable list of flaky test names in the console.

Notes

• You'll need to create a custom Jest reporter. Refer to the Jest documentation for details on creating custom reporters: https://jestjs.io/docs/creating-a-custom-reporter
• Consider using Jest's TestResult object to access test results.
• Think about how to efficiently track test results without storing excessive data.
• The test names in the output should be descriptive, ideally including the suite name if applicable.
• Focus on accurately identifying flaky tests and providing a clear report. Styling and advanced features are not required.