Onionoo job crash: '<' not supported between instances of 'NoneType' and 'float'

This is ongoing since 23-03-31-20:15:45 according to our logs. Here is the full first line:

/home/weather/tor-weather/logs/tor-weather-23-03-31-20:15:45.log:03/31/2023 08:15:47 PM : JOB : INFO : Onionoo Job Crashed - '<' not supported between instances of 'NoneType' and 'float'

I first thought, this might be related to our fix of #57 (closed) but that got a) deployed earlier and b) does not touch anything related to the Onionoo job.

@sarthikg: I am not sure how severe that problem is but a continuous Onionoo job crash sounds bad and might break Tor Weather functionality as keep failing to import the new statuses? Or are we smarter than just aborting that part?

added Doing label

assigned to @sarthikg

changed the description

And, obviously, tpo/network-health/metrics/monitoring-and-alerting#24 is still a smart idea to do. :)

@gk I think I know why this is happening. It could be because onionoo might be responding with null as the observed bandwidth for a few relays. I think I missed that case. Will add a condition to prevent it from breaking.

But ideally, what should be the right way to handle these scenarios?

But ideally, what should be the right way to handle these scenarios?

I think if we choke at an Onionoo line we should log that as a warning or better an error and then skip that line continuing with the remaining ones. The point is to get as much out of the Onionoo updates as we can.

That said, I am not sure whether things really broke as hard as I assumed given that we got this tor-commits@ spam over the weekend. So, it seems updates did happen.

Another thing that came to mind: it would be nice to make the log a bit more verbose while we are at it. From the line in the description it's not really clear, e.g. which relay is affected here, which could be of help when debugging things.

changed the description

removed Doing label

added Backlog label

Last time I saw this:

tor-weather-23-07-01-08:00:35.log:07/01/2023 08:00:54 AM : JOB : INFO : Onionoo Job Crashed - '<' not supported between instances of 'NoneType' and 'float'

and before 07/01 it was:

tor-weather-23-06-01-08:00:43.log:06/01/2023 08:01:01 AM : JOB : INFO : Onionoo Job Crashed - 
'<' not supported between instances of 'NoneType' and 'float'

and before that:

tor-weather-23-05-03-09:00:45.log:05/03/2023 09:00:56 AM : JOB : INFO : Onionoo Job Crashed - 
'<' not supported between instances of 'NoneType' and 'float'

Weird.

And today:

08/21/2023 01:16:23 PM : JOB : INFO : Onionoo Job Crashed - '<' not supported between instances of 'NoneType' and 'float'

mentioned in merge request !96 (merged)

mentioned in issue #73 (closed)

closed with merge request !96 (merged)