Alter flag-weight balancing equations

I have a few things I want to do to the flag-weight equations. This ticket will eventually be the parent ticket, but for now it's just a brain-dump:

1. I definitely want to correct the balancing issues discussed in legacy/trac#8240 (moved).
2. I might want to account for changes due to directory guards (legacy/trac#7757 (moved)).
3. I might want to factor in an optional Guard/Middle padding parameter. (legacy/trac#7028 (moved))
4. I might want to factor in an estimate of hidden service traffic somehow.

changed milestone to %Tor: unspecified in legacy/trac

moved from tor#8453 (moved)

added 034-removed-20180328 in Legacy / Trac 034-triage-20180328 in Legacy / Trac SponsorZ in Legacy / Trac component::core tor/tor in Legacy / Trac milestone::Tor: unspecified in Legacy / Trac needs-proposal in Legacy / Trac performance in Legacy / Trac points::6 in Legacy / Trac priority::medium in Legacy / Trac severity::normal in Legacy / Trac status::new in Legacy / Trac tor-client in Legacy / Trac type::enhancement in Legacy / Trac labels

Trac:
Description: I have a few things I want to do to the flag-weight equations. This ticket will eventually be the parent ticket, but for now it's just a brain-dump:

1. I definitely want to correct the balancing issues discussed in legacy/trac#8240 (moved).
2. I might want to account for changes due to directory guards (legacy/trac#7752 (closed)).
3. I might want to factor in an optional Guard/Middle padding parameter. (legacy/trac#7208 (closed))
4. I might want to factor in an estimate of hidden service traffic somehow.

to

I have a few things I want to do to the flag-weight equations. This ticket will eventually be the parent ticket, but for now it's just a brain-dump:

1. I definitely want to correct the balancing issues discussed in legacy/trac#8240 (moved).
2. I might want to account for changes due to directory guards (legacy/trac#7757 (moved)).
3. I might want to factor in an optional Guard/Middle padding parameter. (legacy/trac#7028 (moved))
4. I might want to factor in an estimate of hidden service traffic somehow.

Trac:
Keywords: SponsorZ, performance deleted, performance SponsorZ tor-client added

Also, legacy/trac#8356 (moved) might involve the same sort of data. Might be worth looking at along the way.

See also legacy/trac#2395 (moved).

Trac:
Keywords: performance SponsorZ tor-client deleted, performance SponsorZ tor-client mike-0.2.5 added

Trac:
Keywords: performance SponsorZ tor-client mike-0.2.5 deleted, performance SponsorZ needs-proposal tor-client mike-0.2.5 added

Trac:
Milestone: Tor: 0.2.5.x-final to Tor: 0.2.???

Trac:
Parent: N/A to legacy/trac#9321 (moved)

Trac:
Cc: N/A to isis

Trac:
Parent: legacy/trac#9321 (moved) to N/A

Trac:
Keywords: performance SponsorZ needs-proposal tor-client mike-0.2.5 deleted, performance SponsorZ needs-proposal tor-client mike-0.2.5 028-triage added

Trac:
Sponsor: N/A to N/A
Milestone: Tor: 0.2.??? to Tor: 0.2.8.x-final

Trac:
Points: N/A to large

At the very least, I need to get at least get a proposal for this out in November, so we can consider it in combination with legacy/trac#16861 (moved).

Trac:
Keywords: performance SponsorZ needs-proposal tor-client mike-0.2.5 028-triage deleted, performance SponsorZ needs-proposal tor-client mike-0.2.5 028-triage TorCoreTeam201511 added
Severity: N/A to Normal

Bulk-move uncompleted items to december. :/

Trac:
Keywords: performance SponsorZ needs-proposal tor-client mike-0.2.5 028-triage TorCoreTeam201511 deleted, mike-0.2.5, TorCoreTeam201512, 028-triage, 201511-deferred, performance, tor-client, SponsorZ, needs-proposal added

I'd look at a patch for these if we got one, but AFAIK nobody's looking at them right now, and there's no reason to expect them to get done this month.

Trac:
Keywords: TorCoreTeam201512 deleted, N/A added

It is impossible that we will fix all 226 currently open 028 tickets before 028 releases. Time to move some out. This is my second pass through the "new" and tickets, looking for things to move to 0.2.9.

Trac:
Milestone: Tor: 0.2.8.x-final to Tor: 0.2.9.x-final

Trac:
Sponsor: N/A to SponsorU-can
Reviewer: N/A to N/A

Trac:
Keywords: N/A deleted, tor-sponsorU-orphan added

Trac:
Points: large to 6

Deferring more things -- even ones I love -- to 0.3.0. Please argue if I'm wrong.

Trac:
Milestone: Tor: 0.2.9.x-final to Tor: 0.3.0.x-final
Keywords: N/A deleted, nickm-deferred-20161005 added

Triaged out on December 2016 from 030 to 031.

Trac:
Keywords: N/A deleted, triage-out-030-201612 added
Milestone: Tor: 0.3.0.x-final to Tor: 0.3.1.x-final

Clear sponsorS and sponsorU from open tickets in 0.3.1

Trac:
Sponsor: SponsorU-can to N/A

0.3.1 feature freeze today: these don't seem like they will be all sorted out in time. Let's try to make progress in 0.3.2.

Trac:
Milestone: Tor: 0.3.1.x-final to Tor: 0.3.2.x-final

Trac:
Keywords: mike-0.2.5, 028-triage, 201511-deferred, nickm-deferred-20161005, tor-sponsorU-orphan, triage-out-030-201612 deleted, N/A added

Trac:
Milestone: Tor: 0.3.2.x-final to Tor: 0.3.3.x-final

Trac:
Sponsor: N/A to SponsorV-can

Deferring various "new"-status enhancement tickets to 0.3.4

Trac:
Milestone: Tor: 0.3.3.x-final to Tor: 0.3.4.x-final

Trac:
Keywords: N/A deleted, 034-triage-20180328 added

Per our triage process, these tickets are pending removal from 0.3.4.

Trac:
Keywords: N/A deleted, 034-removed-20180328 added

These tickets, tagged with 034-removed-*, are no longer in-scope for 0.3.4. We can reconsider any of them, if time permits.

Trac:
Milestone: Tor: 0.3.4.x-final to Tor: unspecified

Removing sponsor V as we do not have more time to include this tickets in the sponsor.

Removing sponsor from tickets that we do not have time to fit in the remain of this sponsorship.

Trac:
Sponsor: SponsorV-can to N/A

Trac:
Cc: isis to isis, mikeperry
Owner: mikeperry to N/A
Status: new to assigned

Change tickets that are assigned to nobody to "new".

Trac:
Status: assigned to new

changed time estimate to 48h

mentioned in issue legacy/trac#9321 (moved)

mentioned in issue legacy/trac#24456 (moved)

mentioned in issue legacy/trac#29098 (moved)

moved from legacy/trac#8453 (moved)

added Feature label and removed 1 deleted label

added Client in Tor Needs Proposal in Tor labels and removed Performance in Tor + 1 deleted label

added Proposal Ideas label

removed milestone

Moving to Specifications.

@mikeperry, please close this if it's obsolete?

This is not necessarily obsolete; we should revisit this stuff after congestion control and related sbws upgrades.

Perhaps the time to look at all of this stuff is when we start to implement padding, as tor#29098 is a significant change that will also simplify these equations, and we can look at the other aspects during that, too. (It may be possible to combine some of these additional sources of load into the "overhead" parameters).

There is also tor#24456 (closed), which is potentially useful, and somewhat orthogonal to the flag equations, though.

Hi Mike! Aaron Johnson told me offline that he has a recent paper working through the consensus positional weights, and he has a bugfix for a case we don't hit currently, and also he has some alternative solutions to the equations that satisfy the constraints and also do it in a more defensible way.

So, nothing urgent here I think, but here is a shout-out to Aaron so he can use this ticket when he's ready.

Yes, I recently looked into the equations defining the positional weights (Wgg, Wmg, etc.), and I think they could be derived in a more principled way that yields some improvements. My suggestions don't address any of the issues listed in this ticket, but they would

1. Resolve a failure case (that you already know about https://github.com/torproject/tor/blob/f6600377b495c276cbaf423d9c8907abd4cb3c82/src/test/test_dir.c#L3218)
2. Maximize throughput for another case in which the weights are valid but yield suboptimal throughput
3. Make weights continuous so that that small changes in the network state yields small changes in the weights. Currently, for example, it is possible for a single relay changing its weights by just 1 can cause the D relays (i.e. those with Guard and Exit flags) to jump back and forth between not being used as guards at all to being used a large amount (e.g. 33% of the total D bandwidth).
4. Achieve secondary goals like equalizing the "sensitive" traffic (i.e. guard and exit traffic) that G/E/D classes see.

The writeup is in progress, but I would be happy to engage with you now if you are interested.