Je⁄ust Retry
Motivation
E2E test failures due to reasons outside the domain of source code, including:
- Network timeouts
- Flaky endpoints etc.
- Login issues
- Flaky selenium selectors (bad code which is hard to diagnose)
- Whatever other mishaps, etc.
Setup
Create a run-config to pass into jest-retry
Example run
require('jest-retry')({
debug: false,
testDir: 'src/tests',
testFilter: 'table-tests',
// jest-junit is currently the only option
outputTestResults: true,
testResultsOutput: '.',
/* Test retry options */
flakyTestMock: false,
flakyTestMockDir: 'src/mocks',
flakyFailureMessages: ['Network timeout'],
// Matches to test failures `testFilePath`
// These are still marked as fail, but does not contribute to jest exiting with failure status
knownIssues: ['src/test/known-issue.test'],
flakyNumRetries: 2, // 0 for disable retry pattern or Infinity for continue running until getting same results
flakyMarkAll: false,
flakyWaitBeforeRerun: 1000,
// Jest options
setupTestFrameworkScriptFile: 'jest-config.js',
maxWorkers: 1,
noStackTrace: false,
silent: false,
});Test
npm run test runs a short 2 test suite in which
a failure occurs and is forced to pass on the second retry
Test output
☁ jest-retry [master] npm run test
> [email protected] test /code/jest-retry
> ./test/run.js
PASS test/always-pass.test.js
FAIL test/conditional-fail.test.js
● mocks/conditional-fail › conditionally fails
expect(received).toBe(expected) // Object.is equality
Expected value to be:
false
Received:
true
1 | describe('mocks/conditional-fail', async () => {
2 | it('conditionally fails', async () => {
> 3 | if (!process.env.SKIP) expect(true).toBe(false);
4 | });
5 | });
6 |
at Object.it (test/conditional-fail.test.js:3:41)
Test Suites: 1 failed, 1 passed, 2 total
Tests: 1 failed, 1 passed, 2 total
Snapshots: 0 total
Time: 0.28s, estimated 1s
Ran all test suites.
flakyDictionaryCount: {
"Expected value to be": 1
}
Retrying the following test suites: [
"/code/jest-retry/test/conditional-fail.test.js"
]
===============================================
PASS test/conditional-fail.test.js
mocks/conditional-fail
✓ conditionally fails
Test Suites: 1 passed, 1 total
Tests: 1 passed, 1 total
Snapshots: 0 total
Time: 0.099s, estimated 1s
Ran all test suites.
All failures have now passed
Test retries result: true