Tor should be more gentle when launching dozens of circuits at once

When starting up Tor as an HS, Tor will create dozens of circs in quick succession (preemptive circs, HSDir circs, intro point circs, CBT circs).

Consider a Tor that starts up with 10 HSes configured(many of those Tors exist); it will easily attempt to make hundreds of circuits within seconds.

It might make sense to have some sort of circuit rate limiting during startup to avoid flooding our guard with so many circuit creations. I notice that Tor will whine about the guard failing circuits during startup very often, and perhaps this initial circuit flood is what's causing it?

We already have MaxClientCircuitsPending but not sure if that's enough. We also used to delay bootup of multiple onion services, but not sure we do this anymore (can't see it in hs_config_service_all()).

changed milestone to %Tor: unspecified

added 034-removed-20180502 034-triage-20180328 component::core tor/tor milestone::Tor: unspecified network-team-roadmap-2020Q1 performance points::16 priority::medium severity::normal sponsor::27-can status::new tor-dos tor-hs type::defect version::tor unspecified labels

Replying to asn:

We already have MaxClientCircuitsPending but not sure if that's enough.

Also, MaxClientCircuitsPending only counts CIRCUIT_PURPOSE_C_GENERAL circuits, and Mike's new feature in #23101 (moved) has stopped using C_GENERAL circuits for posting or fetching hsdescs.

Also also, my upcoming plans in #24902 (moved) will start sending destroys back if you send me too many create cells in too short a time period.

Replying to asn:

Consider a Tor that starts up with 10 HSes configured(many of those Tors exist); it will easily attempt to make hundreds of circuits within seconds.

You can see one of those Tors in action (probably with more than 10 HSes, but then some of them might be client-side onion fetches too) at #24897 (moved):

Jan 20 16:33:30.000 [notice] We'd like to launch a circuit to handle a connection, but we already have 128 general-purpose client circuits pending. Waiting until some finish. [1808313 similar message(s) suppressed in last 600 seconds]

Replying to arma:

Also also, my upcoming plans in #24902 (moved) will start sending destroys back if you send me too many create cells in too short a time period.

Would it be totally crazy for clients to take a look at the dos_cc_circuit_max_count consensus param (or whatever we end up naming it) from #24902 (moved), and try to hold themselves under it when they have some control over their circuit load, like in this case?

Trac:
Cc: N/A to alecmuffett

Replying to arma:

Would it be totally crazy for clients to take a look at the dos_cc_circuit_max_count consensus param (or whatever we end up naming it) from #24902 (moved), and try to hold themselves under it when they have some control over their circuit load, like in this case?

In general, I do think we want a mechanism in a tor client to never go above a circuit creation rate and sounds like a good logic to use the DoS mitigation rate as the upper limit (which can be a moving target over time). That makes client use a rate that they know they won't get blacklisted for that in theory. Although all this makes sense if our DoS mitigation starts being applied with a single TCP connection (DoSCircuitCreationMinConnections 1).

This also has a side effect of effectively having a way to control the circuit creation rate of the entire network (for normal tor clients). Fun power to have (?).

However, that might not play super well with busy hidden service opening rendezvous circuits at a much higher rate than a normal client would do. And also, I wouldn't probably do this differently with HS because it seems to me an easy way to provide a way to neutralize a service, keep opening circuits to reach that limit and the circuit effectively becomes unresponsive because it can't launch more RP circuits. Else, it would queue the requests and go FIFO with those while it respects the circuit rate.

At first, I would go with circuit rate limiting tor clients (excluding HS). Is this something we want/can have in 033?

Trac:
Status: new to needs_information

I think we need to limit clients and HSs, otherwise there isn't much point in doing this patch.

I honestly don't see an easy way for 033 to have this properly for both client and HS as I think they should not follow the same limitation else we'll just make services one step closer to "not fun to use".

I would maybe prefer having a way to queue request while respecting the circuit creation rate but on HS side I see another DoS opportunity there.

For client, this is much more simpler but for HS we are talking about more thinking and engineering in my opinion which is not something I see easily done for 033...

Ok, let's think about the service side in 0.3.4 and later.

This is now very related to #14006 (moved) that is for client we do have MaxClientCircuitsPending so at worst they do a burst of 32 circuits.

For HS, this can go much bigger with 5+ configured service. I suspect that if we can find a solution in #14006 (moved), this is resolved. Not flagging it as duplicate because maybe that circuit limit rate idea is still something we want for an overall client.

Trac:
Parent: N/A to #14006 (moved)

Pushing this to 034 since we lack a plan. Let's work on #14006 (moved) in 033 if we can.

Trac:
Milestone: Tor: 0.3.3.x-final to Tor: 0.3.4.x-final

Trac:
Keywords: tor-dos tor-hs performance deleted, performance, tor-hs, tor-dos, 034-triage-20180328 added

Trac:
Keywords: N/A deleted, 034-removed-20180502 added
Milestone: Tor: 0.3.4.x-final to Tor: unspecified

Trac:
Sponsor: N/A to Sponsor27

We don't have a plan here. Bumping up the points.

Trac:
Points: 3 to 8

Trac:
Sponsor: Sponsor27 to Sponsor27-must

Trac:
Parent: #14006 (moved) to #29999 (moved)

Related closed tickets #15463 (moved) and #24899 (moved).

Trac:
Points: 8 to 16

Add keyword to tickets in network team's roadmap.

Trac:
Keywords: N/A deleted, network-team-roadmap-2019-Q1Q2 added

Lot of word. Now in the stretch side.

Trac:
Sponsor: Sponsor27-must to Sponsor27-can
Status: needs_information to new

Trac:
Parent: #29999 (moved) to N/A

Trac:
Keywords: network-team-roadmap-2019-Q1Q2 deleted, network-team-roadmap-2020Q1 added

changed time estimate to 128h

mentioned in issue #24989 (moved)

moved to tpo/core/tor#24973 (closed)