collector's downloads: avoid httpurl-connection

In November 2012, tor26 was very slow in responding to metrics-db, which made metrics-db exceed its 30-minute schedule. This happened again on March 2, 2013. The workaround was to avoid tor26 and other slow directories and download descriptors only from the other directory authorities. metrics-db should use a timeout similar to how DocTor does.

This should avoid httpurl-connection as suggested in #20323 (moved) for metrics-lib.

This is the parent ticket for three smaller tasks.

changed milestone to %CollecTor 1.2.0

added component::metrics/collector milestone::CollecTor 1.2.0 owner::iwakeh parent::20518 priority::high resolution::fixed severity::normal status::closed type::enhancement labels

added this ticket to the "Areas for Improvement" of CTIP

Trac:
Keywords: N/A deleted, ctip added
Sponsor: N/A to N/A
Reviewer: N/A to N/A
Severity: N/A to Normal

changed metrics-db to collector.

Trac:
Summary: Add timeouts to metrics-db's relay descriptor downloads to Add timeouts to collector's relay descriptor downloads

Added to second release milestone.

Trac:
Milestone: N/A to CollecTor 1.1.0

assigning these issues.

Trac:
Owner: N/A to iwakeh
Status: new to assigned

Trac:
Milestone: CollecTor 1.1.0 to CollecTor 1.2.0

see #20323 (moved)

Moved from #20323 (moved) (comment 3, karsten):

It might be that this issue killed the relaydescs module 1.5 days ago by blocking the downloader indefinitely:

2016-10-30 23:05:00,000 INFO o.t.c.c.CollecTorMain:54 Starting relaydescs module of CollecTor.
2016-10-30 23:05:20,823 INFO o.t.c.r.CachedRelayDescriptorReader:255 Finished importing relay descriptors from local Tor data directories:
cached-consensus: 2016-10-30 23:00:00
cached-descriptors: parsed 0, skipped 24471 server descriptors
cached-descriptors.new: parsed 753, skipped 6024 server descriptors
cached-extrainfo: parsed 0, skipped 24508 extra-info descriptors
cached-extrainfo.new: parsed 745, skipped 1463 extra-info descriptors
v3-status-votes: parsed 8, skipped 0 votes
2016-10-30 23:06:00,001 INFO o.t.c.c.CollecTorMain:54 Starting updateindex module of CollecTor.

There should be a line like this (copied over from the previous run), but there wasn't:

2016-10-30 22:35:19,933 INFO o.t.c.r.RelayDescriptorDownloader:1067 Finished downloading relay descriptors from the directory authorities.

Raising priority to high.

Trac:
Summary: Add timeouts to collector's relay descriptor downloads to collector's relay descriptor downloads: avoid httpurl-connection
Priority: Medium to High
Description: In November 2012, tor26 was very slow in responding to metrics-db, which made metrics-db exceed its 30-minute schedule. This happened again on March 2, 2013. The workaround was to avoid tor26 and other slow directories and download descriptors only from the other directory authorities. metrics-db should use a timeout similar to how DocTor does.

to

In November 2012, tor26 was very slow in responding to metrics-db, which made metrics-db exceed its 30-minute schedule. This happened again on March 2, 2013. The workaround was to avoid tor26 and other slow directories and download descriptors only from the other directory authorities. metrics-db should use a timeout similar to how DocTor does.

This should avoid httpurl-connection as suggested in #20323 (moved) for metrics-lib.

Trac:
Summary: collector's relay descriptor downloads: avoid httpurl-connection to collector's downloads: avoid httpurl-connection
Description: In November 2012, tor26 was very slow in responding to metrics-db, which made metrics-db exceed its 30-minute schedule. This happened again on March 2, 2013. The workaround was to avoid tor26 and other slow directories and download descriptors only from the other directory authorities. metrics-db should use a timeout similar to how DocTor does.

This should avoid httpurl-connection as suggested in #20323 (moved) for metrics-lib.

to

In November 2012, tor26 was very slow in responding to metrics-db, which made metrics-db exceed its 30-minute schedule. This happened again on March 2, 2013. The workaround was to avoid tor26 and other slow directories and download descriptors only from the other directory authorities. metrics-db should use a timeout similar to how DocTor does.

This should avoid httpurl-connection as suggested in #20323 (moved) for metrics-lib.

This is the parent ticket for three smaller tasks.
Keywords: ctip deleted, metrics-help added

Set parent ticket and removed metrics-help tag as this ticket only groups the three subtickets.

Trac:
Parent: N/A to #20518 (moved)
Keywords: metrics-help deleted, N/A added

The download in ExitlistDownloader and RelayDescriptorDownloader should be based on a common code source, which is not part of this task. New ticket?

Trac:
Status: assigned to needs_information

I'd rather simplify the download code to the point where sharing it between modules is not necessary anymore. In any case, that'd be a discussion for a new ticket. Closing. Thanks!

Trac:
Resolution: N/A to fixed
Status: needs_information to closed

closed

mentioned in issue #19828 (moved)

mentioned in issue #19831 (moved)

mentioned in issue #20323 (moved)

mentioned in issue #20518 (moved)