Since when a destination fails, all the threads using it will also
fail that moment.
For now not checking which threads are actually using it.
Also lower the time to retry.

Closes: #29891.