Intermittent GitLab CI runner failures: "network already exists"
Since enabling the FF_NETWORK_PER_BUILD
on our Podman CI runners, there have been a number of intermittent errors like this one:
Running with gitlab-runner 16.8.0 (c72a09b6)
on ci-runner-x86-02-main __hc2zXq, system ID: s_39a8ec4bc83a
feature flags: FF_NETWORK_PER_BUILD:true
Preparing the "docker" executor 00:11
Using Docker executor with image debian:latest ...
ERROR: Preparation failed: Error response from daemon: container d4bbbaa38009ad974fa78664c59a1e28536096505fbf5c9dcbf99675343d50c3 does not exist in database: no such container (manager.go:81:1s)
Will be retried in 3s ...
Using Docker executor with image debian:latest ...
ERROR: Preparation failed: Error response from daemon: network name runner-hc2zxq-project-1144-concurrent-0-job-473518-network already used: network already exists (manager.go:67:0s)
Will be retried in 3s ...
Using Docker executor with image debian:latest ...
ERROR: Preparation failed: Error response from daemon: network name runner-hc2zxq-project-1144-concurrent-0-job-473518-network already used: network already exists (manager.go:67:0s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: network name runner-hc2zxq-project-1144-concurrent-0-job-473518-network already used: network already exists (manager.go:67:0s)
The issue has been documented in this GitLab ticket: Podman. preparation failed, sometimes. The gist is that it's been identified as an issue in Podman 4.4 (which we run), and the fix is to upgrade the runners to Podman 4.5, which isn't straightforward because that's not available in Debian stable currently.
Edited by Jérôme Charaoui