Add bandwidth files archiving to CollecTor

These are referenced by votes, and available via the directory protocol. Unfortunately there is no "current" URL yet, only "next", so we might have to be proactive in downloading these independently of the relaydescs module.

Trac:
Parent Ticket: #21378 (moved)

added component::metrics/collector metrics-roadmap-2019-q2 owner::metrics-team parent::21378 priority::medium resolution::fixed severity::normal status::closed tor-bwauth tor-dirauth type::enhancement labels

The bandwidth file is read by an authority to create its vote at VA-DistSeconds-VoteSeconds: https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n302

That's around HH:50 (and when there's no consensus, HH:20) on the authority's UTC clock.

So you should get the bandwidth file that the authority is about to use for the vote if you start fetching it at HH:49 and HH:19.

After you fetch the vote, you can check bandwidth-file-digest, and re-fetch the bandwidth file if it does not match.

We are trying to build more robust archiving solutions to avoid missing data, and having a window of only a few minutes in which the data is going to be available for certain doesn't help towards that goal. We really should have a "current" URL that keeps a copy of the bandwidth file that was used for at least as long as the consensus is fresh.

Replying to irl:

We are trying to build more robust archiving solutions to avoid missing data, and having a window of only a few minutes in which the data is going to be available for certain doesn't help towards that goal. We really should have a "current" URL that keeps a copy of the bandwidth file that was used for at least as long as the consensus is fresh.

I agree. We have ticket #27047 (moved) for this feature, but it's not on our roadmap right now.

Unfortunately, we've had to minimise non-sponsored work in 2019.

Is there a current sponsor that wants us to do #27047 (moved)? Or should we ask the grants team to find grants for bandwidth authority work?

There is no current sponsor.

Yesterday, I wrote a little script that ran roughly once per minute for over an hour and fetched moria1's and longclaw's "next" bandwidth files. I received a bandwidth file every time, not just between, say, HH:49 and HH:00. More specifically, here's what I got:

Authority	Timestamp	Digest	First received	Last received	Referenced from vote
longclaw	1555868103 (17:35)	lKTscsfb..	..	18:40	18:00
longclaw	1555871704 (18:35)	8KuO5fcL..	18:43	19:40	19:00
longclaw	1555875303 (19:35)	laoWH3KH..	19:41	..	20:00
moria1	1555867341 (17:22)	EAiqle6R..	..	18:45	18:00
moria1	1555871524 (18:32)	5aZPyxPy..	18:46	19:45	19:00
moria1	1555875627 (19:40)	ZTrHiTtI..	19:46	..	20:00

Looking at the 19:00 votes, we could fetch referenced bandwidth files from around 18:45 to around 19:45.

For reference, we're currently fetching votes (and all other relay descriptors) starting at HH:05 and once more at HH:35 in case there was no consensus at HH:00.

How about we simply download "next" bandwidth files at around HH:05 and at around HH:35, knowing that we're really going to receive "previous" bandwidth files? This would be easiest with regard to extending CollecTor's relaydescs module.

Trac:
Reviewer: N/A to irl
Status: new to needs_review

This sounds like an OK approach for now, but it doesn't seem to be particularly robust. We will need to revisit this in the future but I don't think that should hold up implementing this now.

Trac:
Status: needs_review to new
Reviewer: irl to N/A

Please review commit ad1bbc8 in my task-30218 branch.

If you want to try it, be sure to use a metrics-lib-2.6.0-dev.jar built with the #30369 (moved) fix.

This branch worked just fine on my local machine for downloading two hours of bandwidth files. But I have to admit that I haven't tested it as thoroughly as I'd usually do. Let's be sure to give it more testing before we deploy this on the server.

Trac:
Status: new to needs_review

I ran this branch for over a week on my server, and it worked just fine.

When you review the commit above, please also review fixup commit 7d0b9eb in the same branch.

Seems to be working. Code looks good.

Checks and tests pass.

Trac:
Status: needs_review to merge_ready

Thanks for looking! Squashed and merged to master. Opening a new ticket for the release and closing this one. Thanks!

Trac:
Status: merge_ready to closed
Resolution: N/A to fixed

closed

mentioned in issue #30493 (moved)