Calculate effectiveness score for individual bridges and then aggregate scores for bridge strategies
This is the metrics-side ticket for the overall project which is documented in tpo/anti-censorship/rdsys#113
Specifically, from the metrics side I want the ability to loop through a bridge's past, computing a daily score for it (that is, a score for each of its past days).
We'll want to experiment with what exactly goes into the score (that is, I expect we will change our minds a bunch of times over the coming years), but maybe there are some approaches here that are quite easy from the metrics perspective and we should start there. For example, how about if we compute a score for each bridge for each day, where today's score is only a function of information from today and yesterday. In terms of specific state we would track a daily number which is "how many days the bridge has been published at this address and running for most of the day and with this distribution strategy and not thought to be blocked anywhere" and then the score for the bridge would be "today's number of users times number of days since first becoming available."
Variations include
-
doing per-country scores, where we only count users reported from that country, and we only consider it blocked if we think it is blocked from that country
-
a simple memoryless snapshot, where we treat the "number of days available" as simply 1.
Then once we have a few versions of the per-bridge scores, the next step is to sum up the bridge scores by distribution strategy, and to average the bridge scores by distribution strategy, both for a global count and also a per-country count, to see what patterns emerge.