Loading content/issues/2023-11-08-tor-weather-outage.md +16 −13 Original line number Diff line number Diff line Loading @@ -10,25 +10,28 @@ affected: section: issue --- The Tor Weather service broke during [Debian Bookworm upgrade](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252). The database was destroyed and recovered from backups and the service was down for about a week. The Tor Weather service broke during an [upgrade to Debian Bookworm](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252). The database was destroyed (but eventually recovered from backups) and the service was down for about a week. Specifically, all changes to the service after 2023-10-27 06:23:11 UTC have been lost. The service itself went offline some time before 2023-10-31 20:33UTC, which means a little over 4 days of data was lost, since new changes couldn't have been made after the service was offline. 2023-10-31 20:33UTC, therefore about 4 days of data was lost. We normally have point-in-time recovery procedures, but that procedure failed due to a flaw in the bookworm upgrade procedure, since then corrected. So if you have registered or made any changes on the service after If you have registered or made any changes on the service after October 27th, you will need to redo those changes. The service was restored to the October 27th backup on November 8th (2023-11-08 18:29UTC), which means the service was offline for a little over 8 days, from October 31th to November 8th. The service was restored on November 8th (2023-11-08 18:29UTC) from a the October 27th backup. So the service was offline for a little over 8 days, from October 31th to November 8th. A post-mortem can be found in our [bug tracker](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388), including a full timeline of the incident, and planned improvements to our systems to A more detailed discussion and post-mortem can be found in our [bug tracker](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388), including a full timeline of the incident, heroic deleted file recovery, and possible improvements to our systems that should keep this specific problem from happening again. Sorry for the inconvenience. It is not in our habit to lose data, and we apologize profusely for this mishap. Loading
content/issues/2023-11-08-tor-weather-outage.md +16 −13 Original line number Diff line number Diff line Loading @@ -10,25 +10,28 @@ affected: section: issue --- The Tor Weather service broke during [Debian Bookworm upgrade](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252). The database was destroyed and recovered from backups and the service was down for about a week. The Tor Weather service broke during an [upgrade to Debian Bookworm](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252). The database was destroyed (but eventually recovered from backups) and the service was down for about a week. Specifically, all changes to the service after 2023-10-27 06:23:11 UTC have been lost. The service itself went offline some time before 2023-10-31 20:33UTC, which means a little over 4 days of data was lost, since new changes couldn't have been made after the service was offline. 2023-10-31 20:33UTC, therefore about 4 days of data was lost. We normally have point-in-time recovery procedures, but that procedure failed due to a flaw in the bookworm upgrade procedure, since then corrected. So if you have registered or made any changes on the service after If you have registered or made any changes on the service after October 27th, you will need to redo those changes. The service was restored to the October 27th backup on November 8th (2023-11-08 18:29UTC), which means the service was offline for a little over 8 days, from October 31th to November 8th. The service was restored on November 8th (2023-11-08 18:29UTC) from a the October 27th backup. So the service was offline for a little over 8 days, from October 31th to November 8th. A post-mortem can be found in our [bug tracker](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388), including a full timeline of the incident, and planned improvements to our systems to A more detailed discussion and post-mortem can be found in our [bug tracker](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388), including a full timeline of the incident, heroic deleted file recovery, and possible improvements to our systems that should keep this specific problem from happening again. Sorry for the inconvenience. It is not in our habit to lose data, and we apologize profusely for this mishap.
