dist website update fails/aborts on staticiforme
The publish step ( static-update-component cdn.torproject.org && static-update-component dist.torproject.org
) for our dist server is failing with:
richard@staticiforme:~$ static-update-component cdn.torproject.org && static-update-component dist.torproject.org
2022-08-27 18:22:10+0000 /usr/local/bin/static-master-update-component: Acquiring lock on /srv/static.torproject.org/master/cdn.torproject.org.lock...
2022-08-27 18:22:10+0000 /usr/local/bin/static-master-update-component: Got them.
2022-08-27 18:22:10+0000 /usr/local/bin/static-master-update-component: Updating master copy of cdn.torproject.org...
2022-08-27 18:22:10+0000 /usr/local/bin/static-master-update-component: Done. Committing.
2022-08-27 18:22:10+0000 /usr/local/bin/static-master-update-component: Triggering mirror runs...
[2022-08-27 18:22:11] Acquiring lock for /srv/static.torproject.org/master/cdn.torproject.org.lock(3).
[2022-08-27 18:22:11] All locks acquired.
[2022-08-27 18:22:11] Serial is 1661624530.
[2022-08-27 18:22:11] Populating /srv/static.torproject.org/master/cdn.torproject.org-live.new-XQofiX.
[2022-08-27 18:22:11] Renaming /srv/static.torproject.org/master/cdn.torproject.org-live.new-XQofiX to /srv/static.torproject.org/master/cdn.torproject.org-current-push.
[2022-08-27 18:22:11] Calling clients...
[2022-08-27 18:22:11] Stage 1...
[2022-08-27 18:22:11] web-fsn-02.torproject.org >> [MSM] STAGE1-START (2022-08-27 18:22:11+00:00 on web-fsn-02.torproject.org)
[2022-08-27 18:22:11] web-fsn-02.torproject.org >> [MSM] STAGE1-DONE (2022-08-27 18:22:11+00:00 on web-fsn-02.torproject.org)
[2022-08-27 18:22:11] web-fsn-02.torproject.org: waiting
[2022-08-27 18:22:11] cdn-backend-sunet-01.torproject.org >> [MSM] STAGE1-START (2022-08-27 18:22:11+00:00 on cdn-backend-sunet-01.torproject.org)
[2022-08-27 18:22:12] cdn-backend-sunet-01.torproject.org >> [MSM] STAGE1-DONE (2022-08-27 18:22:12+00:00 on cdn-backend-sunet-01.torproject.org)
[2022-08-27 18:22:12] cdn-backend-sunet-01.torproject.org: waiting
[2022-08-27 18:22:13] web-chi-03.torproject.org >> [MSM] STAGE1-START (2022-08-27 18:22:12+00:00 on web-chi-03.torproject.org)
[2022-08-27 18:22:15] web-chi-03.torproject.org >> [MSM] STAGE1-DONE (2022-08-27 18:22:15+00:00 on web-chi-03.torproject.org)
[2022-08-27 18:22:15] web-chi-03.torproject.org: waiting
[2022-08-27 18:22:15] web-chi-04.torproject.org >> [MSM] STAGE1-START (2022-08-27 18:22:13+00:00 on web-chi-04.torproject.org)
[2022-08-27 18:22:16] web-chi-04.torproject.org >> [MSM] STAGE1-DONE (2022-08-27 18:22:16+00:00 on web-chi-04.torproject.org)
[2022-08-27 18:22:16] web-chi-04.torproject.org: waiting
[2022-08-27 18:22:16] web-fsn-01.torproject.org >> [MSM] STAGE1-START (2022-08-27 18:22:11+00:00 on web-fsn-01.torproject.org)
[2022-08-27 18:22:16] web-fsn-01.torproject.org >> [MSM] STAGE1-DONE (2022-08-27 18:22:11+00:00 on web-fsn-01.torproject.org)
[2022-08-27 18:22:16] web-fsn-01.torproject.org: waiting
[2022-08-27 18:22:16] Stage 1 done.
[2022-08-27 18:22:16] Committing...
[2022-08-27 18:22:16] web-fsn-02.torproject.org << go
[2022-08-27 18:22:16] cdn-backend-sunet-01.torproject.org << go
[2022-08-27 18:22:16] web-chi-03.torproject.org << go
[2022-08-27 18:22:16] web-chi-04.torproject.org << go
[2022-08-27 18:22:16] web-fsn-01.torproject.org << go
[2022-08-27 18:22:16] web-fsn-02.torproject.org >> [MSM] STAGE2-DONE
[2022-08-27 18:22:16] web-fsn-02.torproject.org >>
[2022-08-27 18:22:16] web-fsn-02.torproject.org: returned 0
[2022-08-27 18:22:16] cdn-backend-sunet-01.torproject.org >> [MSM] STAGE2-DONE
[2022-08-27 18:22:16] cdn-backend-sunet-01.torproject.org >>
[2022-08-27 18:22:16] cdn-backend-sunet-01.torproject.org: returned 0
[2022-08-27 18:22:16] web-chi-03.torproject.org >> [MSM] STAGE2-DONE
[2022-08-27 18:22:16] web-chi-03.torproject.org >>
[2022-08-27 18:22:16] web-chi-03.torproject.org: returned 0
[2022-08-27 18:22:16] web-chi-04.torproject.org >> [MSM] STAGE2-DONE
[2022-08-27 18:22:16] web-chi-04.torproject.org >>
[2022-08-27 18:22:16] web-chi-04.torproject.org: returned 0
[2022-08-27 18:22:16] web-fsn-01.torproject.org >> [MSM] STAGE2-DONE
[2022-08-27 18:22:16] web-fsn-01.torproject.org >>
[2022-08-27 18:22:16] web-fsn-01.torproject.org: returned 0
[2022-08-27 18:22:16] Moving /srv/static.torproject.org/master/cdn.torproject.org-current-live aside.
[2022-08-27 18:22:16] Renaming /srv/static.torproject.org/master/cdn.torproject.org-current-push to /srv/static.torproject.org/master/cdn.torproject.org-current-live.
[2022-08-27 18:22:16] Cleaning up.
[2022-08-27 18:22:16] Done.
Connection to static-master-fsn.torproject.org closed.
2022-08-27 18:22:17+0000 /usr/local/bin/static-master-update-component: Acquiring lock on /srv/static.torproject.org/master/dist.torproject.org.lock...
2022-08-27 18:22:17+0000 /usr/local/bin/static-master-update-component: Got them.
2022-08-27 18:22:17+0000 /usr/local/bin/static-master-update-component: Updating master copy of dist.torproject.org...
2022-08-27 18:22:17+0000 /usr/local/bin/static-master-update-component: Done. Committing.
2022-08-27 18:22:17+0000 /usr/local/bin/static-master-update-component: Triggering mirror runs...
[2022-08-27 18:22:17] Acquiring lock for /srv/static.torproject.org/master/dist.torproject.org.lock(3).
[2022-08-27 18:22:17] All locks acquired.
[2022-08-27 18:22:17] Serial is 1661624537.
[2022-08-27 18:22:17] Populating /srv/static.torproject.org/master/dist.torproject.org-live.new-08qlti.
[2022-08-27 18:22:17] Removing existing /srv/static.torproject.org/master/dist.torproject.org-current-push.
[2022-08-27 18:22:17] Renaming /srv/static.torproject.org/master/dist.torproject.org-live.new-08qlti to /srv/static.torproject.org/master/dist.torproject.org-current-push.
[2022-08-27 18:22:17] Calling clients...
[2022-08-27 18:22:17] Stage 1...
[2022-08-27 18:22:17] web-fsn-02.torproject.org >> [MSM] STAGE1-START (2022-08-27 18:22:17+00:00 on web-fsn-02.torproject.org)
[2022-08-27 18:22:20] web-fsn-02.torproject.org: failed with returncode 11
[2022-08-27 18:22:20] web-fsn-01.torproject.org >> [MSM] STAGE1-START (2022-08-27 18:22:17+00:00 on web-fsn-01.torproject.org)
[2022-08-27 18:22:20] web-fsn-01.torproject.org: failed with returncode 11
[2022-08-27 18:22:20] web-chi-03.torproject.org >> [MSM] STAGE1-START (2022-08-27 18:22:19+00:00 on web-chi-03.torproject.org)
[2022-08-27 18:22:23] web-chi-03.torproject.org >> [MSM] STAGE1-DONE (2022-08-27 18:22:23+00:00 on web-chi-03.torproject.org)
[2022-08-27 18:22:23] web-chi-03.torproject.org: waiting
[2022-08-27 18:22:23] web-chi-04.torproject.org >> [MSM] STAGE1-START (2022-08-27 18:22:19+00:00 on web-chi-04.torproject.org)
[2022-08-27 18:22:23] web-chi-04.torproject.org >> [MSM] STAGE1-DONE (2022-08-27 18:22:22+00:00 on web-chi-04.torproject.org)
[2022-08-27 18:22:23] web-chi-04.torproject.org: waiting
[2022-08-27 18:22:23] archive-01.torproject.org >> [MSM] STAGE1-START (2022-08-27 18:22:18+00:00 on archive-01.torproject.org)
[2022-08-27 18:22:23] archive-01.torproject.org >> [MSM] STAGE1-DONE (2022-08-27 18:22:18+00:00 on archive-01.torproject.org)
[2022-08-27 18:22:23] archive-01.torproject.org: waiting
[2022-08-27 18:22:23] Stage 1 done.
[2022-08-27 18:22:23] 2 clients failed, aborting...
[2022-08-27 18:22:23] web-chi-03.torproject.org << abort
[2022-08-27 18:22:23] web-chi-04.torproject.org << abort
[2022-08-27 18:22:23] archive-01.torproject.org << abort
[2022-08-27 18:22:23] web-chi-03.torproject.org >> [MSM] STAGE2-ABORT
[2022-08-27 18:22:23] web-chi-03.torproject.org >>
[2022-08-27 18:22:23] web-chi-03.torproject.org: returned 0
[2022-08-27 18:22:23] web-chi-04.torproject.org >> [MSM] STAGE2-ABORT
[2022-08-27 18:22:23] web-chi-04.torproject.org >>
[2022-08-27 18:22:23] web-chi-04.torproject.org: returned 0
[2022-08-27 18:22:23] archive-01.torproject.org >> [MSM] STAGE2-ABORT
[2022-08-27 18:22:23] archive-01.torproject.org >>
[2022-08-27 18:22:23] archive-01.torproject.org: returned 0
[2022-08-27 18:22:23] Aborted.
Connection to static-master-fsn.torproject.org closed.
Edited by morgan