time-based snapshots of debian-security are broken: 'Connection failed [IP: 151.101.22.132 80]'

Example:

Mar 25 17:15:00 apt systemd[1]: Starting tails-update-time-based-apt-snapshots-debian-security-3.service - Take time-based snapshot of debian-security...
Mar 25 17:16:09 apt flock[3336014]: aptmethod error receiving 'http://security.debian.org/debian-security/pool/updates/main/n/nginx/libnginx-mod-http-perl_1.18.0-6.1+deb11u4_amd64.deb':
Mar 25 17:16:09 apt flock[3336014]: 'Connection failed [IP: 151.101.22.132 80]'
Mar 25 17:17:11 apt flock[3336014]: aptmethod error receiving 'http://security.debian.org/debian-security/pool/updates/main/n/nginx/nginx-extras_1.18.0-6.1+deb11u4_amd64.deb':
Mar 25 17:17:11 apt flock[3336014]: 'Connection failed [IP: 151.101.22.132 80]'
Mar 25 17:17:42 apt flock[3336002]: "/usr/bin/reprepro" unexpectedly returned exit value 255 at /usr/local/bin/tails-update-time-based-apt-snapshots line 40.
Mar 25 17:17:42 apt systemd[1]: tails-update-time-based-apt-snapshots-debian-security-3.service: Main process exited, code=exited, status=255/EXCEPTION
Mar 25 17:17:42 apt systemd[1]: tails-update-time-based-apt-snapshots-debian-security-3.service: Failed with result 'exit-code'.
Mar 25 17:17:42 apt systemd[1]: Failed to start tails-update-time-based-apt-snapshots-debian-security-3.service - Take time-based snapshot of debian-security.
Mar 25 17:17:42 apt systemd[1]: tails-update-time-based-apt-snapshots-debian-security-3.service: Consumed 2.844s CPU time.

i think this is the same issue as #18195 (closed), that is, connection between our colo and Fastly is flaky.

added Doing Tails labels

assigned to @zen

@micah said this might be related to IPv6. We don't use it in the VM, but despite that i've manually turned it off and it looks like it really makes a difference.

I'll prepare a MR and wait for the next few runs of the relevant jobs to see how it goes.

/cc @boyska

I'll prepare a MR

Here it is: puppet-code!17

FTR, the first approach didn't prevent the issue to happen again.

I now manually configured APT to force IPv4 (assuming reprepro uses apt-get under the hood), and will observe.

root@apt:~# cat /etc/apt/apt.conf.d/99force-ipv4 
Acquire::ForceIPv4 "true";

Can you do some ipv6 tests to see if it functions normally? I wonder if maybe DNS resolution, or ipv6 routing is not working right.

...

On 2025-03-26 14:19:54, zen (@zen) wrote:

zen commented: #18197 (comment 3180040)

@micah said this might be related to IPv6. We don't use it in the VM, but despite that i've manually turned it off and it looks like it really makes a difference.

@micah, ipv6 support is disabled both in the host (this has always been so) and in the VM (since last week), so my understanding is that we can't really do any ipv6 tests. Right?

i can report that none of the measures i tried (disabling support in the kernel via sysctl and adding apt config to force ipv4) fixed the issue. The problem seems intermittent, and seem to only be related with one specific (Fastly) IP, namely: 151.101.22.132

can you share which measures you took on your side to workaround this issue? were them the same i tried or something different?

Thanks!

i can report that none of the measures i tried (disabling support in the kernel via sysctl and adding apt config to force ipv4) fixed the issue. The problem seems intermittent, and seem to only be related with one specific (Fastly) IP, namely: 151.101.22.132

The method I used was adding the apt config to force ipv4.

can you share which measures you took on your side to workaround this issue? were them the same i tried or something different?

It could be that I was fooled into thinking that the intermittent success that I experienced meant it was solved.

we usually point things at the debian package cache, which is local in the colo facility, and that probably hides those failures. Maybe you would like to also use that?

...

On 2025-04-03 16:46:41, zen (@zen) wrote:

we usually point things at the debian package cache, which is local in the colo facility, and that probably hides those failures. Maybe you would like to also use that?

Doesn't the local cache also suffer from the same issue? (Note: our only concern now is for security.debian.org, as there are no official mirrors outside of Fastly.)

If it doesn't i think we could surely use it. Otherwise, we could also setup a cache outside of the colo to workaround that issue.

zen commented on a discussion: #18197 (comment 3185691)

we usually point things at the debian package cache, which is local in the colo facility, and that probably hides those failures. Maybe you would like to also use that?

Doesn't the local cache also suffer from the same issue? (Note: our only concern now is for security.debian.org, as there are no official mirrors outside of Fastly.)

I haven't seen that issue, but maybe?

If it doesn't i think we could surely use it. Otherwise, we could also setup a cache outside of the colo to workaround that issue.

here are the apt proxy lines that work for bookworm/backports and security:

deb http://aptproxy.riseup.net:9999/debian/ bookworm-backports main contrib non-free non-free-firmware
deb http://aptproxy.riseup.net:9999/debian/ bookworm main contrib non-free non-free-firmware
deb http://aptproxy.riseup.net:9999/debian-security bookworm-security main contrib non-free non-free-firmware
deb http://aptproxy.riseup.net:9999/debian/ bookworm-updates main contrib non-free non-free-firmware

...

On 2025-04-07 13:26:01, zen (@zen) wrote:

added Needs Review label and removed Doing label