Custom number of retries

Great, thanks for that inside.

We trust that our automated tests are good enough not to need 3 retries if they fail (we currently have the option disabled), but just in case there is an issue such as latency or network calls taking too long, we still would prefer a max of 1 retry to help with flakiness.

3 retries for each failed test can add lots of runtime in case a testsuite execution has a big number of failures, while only 1 retry would decrease the total runtime down in such a scenario while still showing there is an issue somewhere.

Hypothetical scenario, let's say our testsuite takes 3 hours to execute in best case scenario, so when all tests are green, and each test takes an equal amount of minutes to execute.

But if half of the tests fail due to a legit software issue and not flakiness:

• having the option to only enable 1 single retry per failed test -> 1.5 hours extra only for retries -> 3 hours for normal execution + 1.5 hours retries = total runtime 4.5 hours total runtime.

• currently only option is 3 retries per failed test -> 1.5 hours x 3 times -> 4.5 hours extra only for retries -> 3 hours for normal execution + 4.5h from retries -> 7.5 hours total runtime.

In both bullet points above, the test history would show that there is a real issue with the software, and it's not just bad flakiness, except with one retry we save up on execution time and have full results 3 hours earlier.