@@ -18,20 +18,21 @@ Specifically, all changes to the service after 2023-10-27 06:23:11 UTC
have been lost. The service itself went offline some time before
2023-10-31 20:33UTC, therefore about 4 days of data was lost. We
normally have point-in-time recovery procedures, but that procedure
failed due to a flaw in the bookworm upgrade procedure, since then
corrected.
failed due to a flaw in the [bookworm upgrade procedure](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm#postgresql-upgrades), since then
If you have registered or made any changes on the service after
October 27th, you will need to redo those changes.
**If you have registered or made any changes on the service after
October 27th, you will need to redo those changes.**
The service was restored on November 8th (2023-11-08 18:29UTC) from a
the October 27th backup. So the service was offline for a little over
8 days, from October 31th to November 8th.
A more detailed discussion and post-mortem can be found in [issue
41388](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388), including a full timeline of the incident, heroic [deleted
file recovery procedures](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/backup#recovering-deleted-files), and possible improvements to our systems
that should keep this specific problem from happening again.
41388](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388), including a [full timeline of the incident](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388/timeline), heroic
[deleted file recovery procedures](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/backup#recovering-deleted-files), and [future improvements](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388/#future-improvements) to
our systems that should keep this specific problem from happening
again.
It is not in our habit to lose data, and we apologize profusely for
