Logic for relay gamification

added Next label

assigned to @miko

marked this issue as related to #26 (closed)

added Doing label and removed Next label

changed the description

changed title from Logic for gamification to Logic for relay gamification

Thanks miko! @tpo/network-health

changed the description

marked this issue as related to #38 (closed)

@miko You can find previous gamification logic in

It would be good to get some of the work from there and improve it.

changed the description

mentioned in issue #38 (closed)

changed the description

@gus and @tpo/network-health the spreadsheet for the gamification logic with values is in https://docs.google.com/spreadsheets/d/1WU6ojWd-acpSXSj8JFJo1PMyB68d46ZK/edit?usp=sharing&ouid=113569411319301507551&rtpof=true&sd=true

Feel free to make comments

removed Doing label

added Needs Review label

Some high-level notes:

(neg) Relay nickname violates Code of Conduct

So far, we quickly removed those relays from the network. Is the plan here to change that? Or are the negative points here for nicknames that do not violate our code of conduct badly? Maybe @gus has some opinions here. As far as I see it there should be no negative score for those relay nicknames if they violate our CoC but we should remove them from the network outright.

Autonomous System (AS) rank (lower = better)

I think someone could explain to me how this is supposed to work. First of all, how is the AS rank calculated? Is that by amount of relays at those ASes or by bandwidth or...? And how do we get from there to "Divide rank by 1000 then round up" and 64 ideal relay points?

Operating System is BSD

That likely means "non-Linux", right? (that's not exactly clear in the spreadsheet @gaba linked to either)

Contact info is set and verified

(neg) Contact info is not set or not verified

That morphed to "Email address not set" in the spreadsheet. I assume the spreadsheet is not our source of truth, right? How does that actually relate to the badge for the contact info in #38 (closed)? Does one get all points if an email address is set and in addition to that a badge if the contact info is verified?

A general question for the categories on the spread sheet: what is the idea behind the "Ideal|Sufficient|Poor Relay" columns? Is that some internal metric we have powering the leaderboards or is that shown somewhere? Why do we have three categories and not just, say, "Ideal" and "Poor"?

I have some remarks to specific columns in the spreadsheet but wanted to get those more general points out of the door first (as maybe those remarks are not necessary anymore after the first round of clarifications).

So far, we quickly removed those relays from the network. Is the plan here to change that? Or are the negative points here for nicknames that do not violate our code of conduct badly?

Thank you for bringing this to notice! All strict policies should be kept in tact in my opinion. Additionally it would be tricky to list relays whose names are passable in the CoC but are considered harmful otherwise. Gray areas can harm our transparency.

First of all, how is the AS rank calculated?

This I have no idea. I read the docs to no avail. We can consider removing AS rank if it continues to be vague.

And how do we get from there to "Divide rank by 1000 then round up" and 64 ideal relay points?

I looked at some AS rank values and put them in a calculation that is perceivable to humans and scales better into the points system, that's all. The general idea being that a lower AS rank will turn into more points for any relay. But if AS rank cannot be calculated to begin with, we can remove it.

That likely means "non-Linux", right? (that's not exactly clear in the spreadsheet @gaba linked to either)

Yes, let me change that to non-Linux.

That morphed to "Email address not set" in the spreadsheet. I assume the spreadsheet is not our source of truth, right? How does that actually relate to the badge for the contact info in #38 (closed)? Does one get all points if an email address is set and in addition to that a badge if the contact info is verified?

Spreadsheet is up for all changes we can think of, yes! I remember discussing with Gus which contact info we usually receive from relay operators, and that not all of them are email IDs. So the solution I came up with is to separate the kinds of contact info we get by:

Rewarding POINTS to relays whose operator added ANY contact info
Rewarding a badge to relay OPERATORS who have their emails registered (and verified) with the relays portal. Do let me know if there's flaws/holes in this that I can fix!

These are very good questions @gk , thank you for asking them. Let me reupload the spreadsheet with changes ASAP.

I just highlighted our points of discussion in the Google sheets version (now in description) and added comments. Maybe someone like @hiro can help with how we calculate AS rank (highlighted in yellow) and if AS ranks are visible to the entire network over Metrics or something similar?

I made changes to the Operating System value so it awards 100 points to BSD relays (because BSDs are suitable) and 50 points for non-Linux relays.

For the time being I added a comment to the relay nicknames that violate CoC (highlighted in bright red.) I had a discussion with @gaba about this too, I think we are good to omit it out of the sheet, what do you think @gk, @gaba?

Regarding AS Rank, @gk asked: First of all, how is the AS rank calculated?

This I have no idea. I read the docs to no avail. We can consider removing AS rank if it continues to be vague.

Can you share with us which docs have you read about AS rank? And how exactly you came with the value "lower = better" on the spreadsheet? Thanks!

mentioned in issue #41 (closed)

added Needs Scope label and removed Needs Review label

added Needs Revision label and removed Needs Scope label

changed the description

@gaba asked for feedback here; my only thought is that it might also be good to have per-family stats in addition to per-relay stats. These family stats could include some measure of location diversity, network diversity, and maybe OS diversity.

Interesting idea here! The current mockup (Relays Portal - #37 (closed)) contains a page for Relay Operator profiles. Do you think displaying relay families on any given operator's profiles is a good idea? (Let me know if I'm way off what you're suggesting?)

I also need better understanding on how different Relay Family stats would look from individual relay stats. I don't have any good ideas atm on how to incorporate this into the design of the Relays Portal (#37 (closed)) but I'm working on it this week.

An immediate solution that comes to mind is to introduce a Badge (#38 (closed)) for Relay Operators who run relay families.

Thoughts?

I've not spent much time on the relays portal design. That might be more a thought for @gus. But I could see the merit of introducing a badge for families.

I just highlighted our points of discussion in the Google sheets version (now in description) and added comments. Maybe someone like @hiro can help with how we calculate AS rank (highlighted in yellow) and if AS ranks are visible to the entire network over Metrics or something similar?

FWIW, I do think looking at diversity based on AS is worthwhile and we should try to make rewarding operators not on OVH, Hetzner etc. work. I am just not sure yet I understand how that is supposed to happen and whether the current idea is working or not.

I made changes to the Operating System value so it awards 100 points to BSD relays (because BSDs are suitable) and 50 points for non-Linux relays.

I am not convinced we should make that decision. There are folks out there that are able to run relays on Windows or macOS well and we should not punish them for wanting to contribute with that by giving them "just" 50 points. Maybe they would choose 100 points + BSD instead just to get the points but would not be able to run the relays as well. From a health perspective we should at the moment award the same amount of points to any non-linux relays.

For the time being I added a comment to the relay nicknames that violate CoC (highlighted in bright red.) I had a discussion with @gaba about this too, I think we are good to omit it out of the sheet, what do you think @gk, @gaba?

Yes, I think so, too. Either the nickname is okay according to our rules or not. If not, we should remove the relay instead of giving minus points.

FWIW, I do think looking at diversity based on AS is worthwhile and we should try to make rewarding operators not on OVH, Hetzner etc. work.

Who do you recommend can help me with understanding AS ranks? My formula is: Lower AS rank (good) implies bigger number, so reward points in direct proportion to the rank. Higher AS rank (bad) = smaller number, and therefore, fewer points.

Another question I have for this is - are AS ranks known to the Metrics portal? Will calculation transparency suffer if they're not tracked by Metrics or onionoo?

From a health perspective we should at the moment award the same amount of points to any non-linux relays.

Done! 100 points for ANY non-Linux relay.

we should remove the relay instead of giving minus points.

Great! booting the value Relay nickname violates Code of Conduct off the spreadsheet. They are now empty cells (just for the calculation to work.)

Spreadsheet is up for all changes we can think of, yes! I remember discussing with Gus which contact info we usually receive from relay operators, and that not all of them are email IDs. So the solution I came up with is to separate the kinds of contact info we get by:

Rewarding POINTS to relays whose operator added ANY contact info

Rewarding a badge to relay OPERATORS who have their emails registered (and verified) with the relays portal. Do let me know if there's flaws/holes in this that I can fix!

Works for me.

Looking closer at the spreadsheet here are some thoughts:

We have three rows dealing with AS related points. How is that supposed to work? I can get 100 points for an uncommon AS and then additionally the points for the AS rank? I'd assume we only give points once and not do weird calculations, in particular as that is hard to explain to ourselves and relay operators if they are asking
For the consensus weight: why does one even get points as a poor relay? Additionally, this category is tricky as there is a ramp-up time until relays can push the traffic they are configured, too. It would be unfortunate if relay operators would be greeted with "poor relay"-experience during that time given that the system we have is designed that way and the operators can't do anything about it. (see: https://blog.torproject.org/lifecycle-of-a-new-relay/ phase 1)
I am not sure how much the MyFamily column is useful as written. There is no need to for a relay operator to set a MyFamily option if they are running just one relay anyway. Thus, it does not make much sense that an ideal/sufficient relay should get any points at all if the operators runs only one. If they are running more than one without setting anything then the bad-relay team should take over and get them to fix their settings or reject the relays from the network.

We have three rows dealing with AS related points. How is that supposed to work? I can get 100 points for an uncommon AS and then additionally the points for the AS rank? I'd assume we only give points once and not do weird calculations, in particular as that is hard to explain to ourselves and relay operators if they are asking

The idea is to have a 2-tier reward - meeting a basic requirement first (AS is uncommon) and then further rewarding relays that have a low AS rank somewhat proportionally based on the rank.

But if simplifying it to rewarding 100 points to those on Uncommon ASes sounds more elegant and less confusing, we could definitely do that. I personally would love it if we could simplify this sheet as much as possible to avoid confusion and promote good understanding of any calculations

For the consensus weight: why does one even get points as a poor relay?

I added this value in pretty recently and wasn't sure if poor relays get smaller consensus weights or no consensus weight. I'm guessing no consensus weight it is? Do correct me if I'm wrong. (I borrowed this value from Metrics' Top Relays https://metrics.torproject.org/rs.html#toprelays)

Additionally, this category is tricky as there is a ramp-up time until relays can push the traffic they are configured, too.

We could pause certain values from being considered during special phases like this. This is the simplest solution I can think of.

Maybe we could also pause Uptime calculations around new release announcements so people actually upgrade to the latest Tor version.

Related: I am changing Tor version values so it awards both Alpha and Stable 100 points. It would be weird to award only 50 points to people upgrading to Alpha.

We could award more points to Alpha versions... I remember @gaba pointing out we should focus on rewarding more points to latest releases especially with Arti. Does awarding 200 points to Alpha and 100 points to Stable sound good (or reckless)? Let me know, @gk or @gaba

I am not sure how much the MyFamily column is useful as written. There is no need to for a relay operator to set a MyFamily option if they are running just one relay anyway.

You're right... setting MyFamily might cause a calculation disparity between operators who have relay families versus those who run only one. It COULD be an extra reward for those running relay families but either way you're right - it needs to be removed. There's already badges (#38 (closed)) for folks running relay families (the ones currently named after army hierarchy.) We could introduce another separate badge simply for having 1+ relays. What do you think?

We could award more points to Alpha versions... I remember @gaba pointing out we should focus on rewarding more points to latest releases especially with Arti. Does awarding 200 points to Alpha and 100 points to Stable sound good (or reckless)? Let me know, @gk or @gaba

I don't think we want to encourage every relay operator to run current alpha releases, in particular if they don't know how to deal with unstable releases. Thus, I think both latest releases should get the same points. I am not sure whether a badge would make a difference here. However, I would scope that to relay operator and not to relay. That way the operator could get that badge while not feeling the need to have every relay running the alpha series. (We do have badges per operator and not just per relay, right?)

Thus, I think both latest releases should get the same points.

Sounds good! Keeping the sheet as is.

(We do have badges per operator and not just per relay, right?)

Yes! Just replied to our discussion on #38 (closed)

There's already badges (#38 (closed)) for folks running relay families (the ones currently named after army hierarchy.) We could introduce another separate badge simply for having 1+ relays. What do you think?

What do you mean with the 1+ relay badge compared to the MyFamily badge? I think just having the latter for now sounds reasonable. We could think about special badges for families the size of 5, 10, 20 etc. relays but then we should keep in mind that we don't want to concentrate bandwidth in few hands and that the bandwidth provided is important here, too. That is 5 relays in a family might not be the same as 5 relays in another, pushing way more bandwidth. Thus, maybe those layered family badges (if we get to them) should somehow take bandwidth into account.

What do you mean with the 1+ relay badge compared to the MyFamily badge? I think just having the latter for now sounds reasonable.

Yay!

we don't want to concentrate bandwidth in few hands and that the bandwidth provided is important here, too.

We have points as rewards for better bandwidths. We could add badges at some later point... but if badges aren't scarce they'll lose value immediately.

Thus, maybe those layered family badges (if we get to them) should somehow take bandwidth into account.

Makes sense!

The idea is to have a 2-tier reward - meeting a basic requirement first (AS is uncommon) and then further rewarding relays that have a low AS rank somewhat proportionally based on the rank.

But if simplifying it to rewarding 100 points to those on Uncommon ASes sounds more elegant and less confusing, we could definitely do that. I personally would love it if we could simplify this sheet as much as possible to avoid confusion and promote good understanding of any calculations

Yes. I'd start with just XXX points for uncommon ASes and a penalty for common ones and see how far that gets us.

For the consensus weight: why does one even get points as a poor relay?

I added this value in pretty recently and wasn't sure if poor relays get smaller consensus weights or no consensus weight. I'm guessing no consensus weight it is? Do correct me if I'm wrong. (I borrowed this value from Metrics' Top Relays https://metrics.torproject.org/rs.html#toprelays)

Well, unmeasured relays get 0 in weight and small ones/ones still ramping up can easily get a consensus weight of 1.

Alright, I will set poor relay points to 0 then?

Yes.

Additionally, this category is tricky as there is a ramp-up time until relays can push the traffic they are configured, too.

We could pause certain values from being considered during special phases like this. This is the simplest solution I can think of.

Maybe we could also pause Uptime calculations around new release announcements so people actually upgrade to the latest Tor version

Both is worth thinking about, yes.

@gaba informed me that OS are self-reported. That means people can lie about the OS they're running their relay on... Should we reduce the number of points we reward to non-Linux OS? (Or remove the criterion?)

Or is there a way to test out OS from Tor's side? @gus @gk can you weigh in on this? How do we move forward?

I don't know how much work it is to announce a different OS platform. Maybe it requires changing the tor code and compiling? And then keep doing that for every new tor release? Maybe @gk has more thoughts here.

It's not just a different OS. Tor versions are self-reported as well. Additionally, we have seen a bunch of relay operators faking their observed bandwidth values.

For Tor versions at least, yes, you need some code/configure changes and compiling. I've not checked but I suspect this is the case as well for the OS.

I'd be fine starting with badges for those non-Linux OSes and latest Tor versions despite them being gameable. We could revisit that decision later on if we think things go south.

Or is there a way to test out OS from Tor's side?

Well, there is p0f and Nmap. Don't think it is wise idea to Nmap relays, and it seems to be possible, but too complicated, to use p0f. But putting it here for someone to think about.

mentioned in issue #37 (closed)

closed

Logic for relay gamification

General list of values we can work with (according to Hiro's review with Miko):

For the Gamification mockup components:

Some challenges:

Important:

Some gamification elements that are not completely values-dependent will be turned into badges

Excel sheet for logic with values plugged in:

Child items ...

Activity

YIELD VALUES (very low priority)
Parameter
Speed (minimum)
Speed (measurement)
Advertised Bandwidth
Uptime (up to 1 year)
(neg) Uptime more than 1 year (implies neglected relay running an old Tor version)

MORE IMPORTANT VALUES
Parameter
Contact info is set and verified
(neg) Contact info is not set or not verified
MyFamily
Consensus Weight
Location
Uncommon AS
(neg) Overcrowded AS
Monthly participation