Verified Commit 0ded775f authored by anarcat's avatar anarcat 💥
Browse files

expand the downtime range from the time of the database failure 

The argument here is that this shows up as "resolved in N hours", and
we should be honest that the outage actually started when we started
losing data.

Another way to put this would mark the upgrade time (2023-10-31) as
the start of the outage, but I think that would be slightly less
honest, as that's just when the service was *visibly* down: all data
in the four days preceeding that are still lsot.

See: team#41388
parent a3733d8f

content/issues/2023-11-08-tor-weather-outage.md

+13 −5
Original line number Diff line number Diff line 
---

title: Tor Weather data loss

date: 2023-11-08 16:00:00 +0000

date: 2023-10-27 06:23:11 +0000

resolved: true

resolvedWhen: 2023-11-08 18:29:00 +0000

# Possible severity levels: down, disrupted, notice
@@ -11,12 +11,20 @@ section: issue 
---



The Tor Weather service broke during Debian Bookworm upgrade. The

database was completely lost because of a critical error in the

upgrade procedure. Service has been restored data was lost.

database was destroyed and recovered from an earlier backup and the

service was down for about a week.



Specifically, all changes to the service after 2023-10-27 06:23:11 UTC

have been lost, so if you have registered or made any changes on the

service after that time, you will need to redo those changes.

have been lost. The service itself went offline some time before

2023-10-31 20:33UTC, which means a little over 4 days of data was

lost.



So if you have registered or made any changes on the service after

November 27th, you will need to redo those changes.



The service was restored to the November 27th backup on November

8th (2023-11-08 18:29UTC), which means the service was offline for a

little over 8 days.



A post-mortem can be found in our [bug tracker](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388), including planned

improvements to our procedures and systems that should keep this from