Skip to content
GitLab
Menu
Projects
Groups
Snippets
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
The Tor Project
Network Health
sbws
Commits
888fb6e8
Unverified
Commit
888fb6e8
authored
Mar 17, 2021
by
juga
Committed by
Georg Koppen
Mar 17, 2021
Browse files
fix: semi-automatic correction of typos
Closes
#33599
.
parent
f6b40b7f
Pipeline
#3993
passed with stage
in 24 minutes and 43 seconds
Changes
29
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
docs/source/contributing.rst
View file @
888fb6e8
...
...
@@ -111,11 +111,11 @@ New features should add a corresponding documentation in /docs.
An editor compatible with `EditorConfig <https://editorconfig.org/>`_ will
help you to follow the general formatting code style.
Timestamps must be in UTC. It is prefered to use ``datetime`` objects or
Timestamps must be in UTC. It is prefer
r
ed to use ``datetime`` objects or
Unix timestamps. Timestamps read by the user should be always formatted in
`ISO 8601 <https://en.wikipedia.org/wiki/ISO_8601>`_
Functional style is prefered:
Functional style is prefer
r
ed:
- use list comprenhensions lambda, map, reduce
- avoid reasigigning variables, instead create new ones
...
...
@@ -148,7 +148,7 @@ and possibly the bugfix version.
The commit message should contain ``Closes: #bugnumber``.
From version 1.0.2 we started to prefix the summary with the subpackage or
component, though we have not standarized the words to use, eg: ``scanner``,
component, though we have not standar
d
ized the words to use, eg: ``scanner``,
``generate``, ``v3bwfile``, ``relaylist``, ``doc``, ``test``, ``CI``.
From version 1.0.3, we also started to prefix the summary with ``new``,
...
...
@@ -156,7 +156,7 @@ From version 1.0.3, we also started to prefix the summary with ``new``,
sections in the CHANGELOG.
From version 1.1.0 we started to use the words ``new``, ``chg`` and ``fix``,
not in the sense `gitchangelog`_ use them, but to match sematic versioning
not in the sense `gitchangelog`_ use them, but to match sema
n
tic versioning
changes major, minor and patch.
Try to make each commit a logically separate changes.::
...
...
@@ -313,7 +313,7 @@ Example commit message
between, but conventions vary here
.. rubric:: External eferences
.. rubric:: External
r
eferences
.. [DIST] https://git-scm.com/book/en/v2/Distributed-Git-Contributing-to-a-Project
.. [MERG] https://www.atlassian.com/git/tutorials/merging-vs-rebasing
...
...
docs/source/examples/sbws.example.ini
View file @
888fb6e8
...
...
@@ -30,7 +30,7 @@ country = ZZ
max_num_failures
=
3
## The following logging options are set by default.
## There is no need to change them unless other options are prefered.
## There is no need to change them unless other options are prefer
r
ed.
; [logging]
; # Whether or not to log to a rotating file the directory paths.log_dname
; to_file = yes
...
...
docs/source/generator.rst
View file @
888fb6e8
...
...
@@ -23,15 +23,15 @@ Each relay bandwidth measurements are selected in the following way:
#. At least two bandwidth measurements (``Result`` s) MUST have been obtained
within an arbitrary number of seconds (currently one day).
If they are not, the relay MUST NOT be included in the Bandwith File.
#. The measurements than are are older than an arbitrary number of se
n
conds
If they are not, the relay MUST NOT be included in the Bandwi
d
th File.
#. The measurements than are are older than an arbitrary number of seconds
in the past MUST be discarded.
Currently this number is the same as ``data_period`` (5 days) when not
scaling as Torflow and 28 days when scaling as Torflow.
If the number of relays to include in the Bandwidth File are less than
a percententage (currently 60%) than the number of relays in the consensus,
additional Header Lines MUST be added (see XXX) to the Bandwith File and the
additional Header Lines MUST be added (see XXX) to the Bandwi
d
th File and the
relays SHOULD NOT be included.
.. image:: ./images/activity_aggr_file.svg
...
...
docs/source/glossary.rst
View file @
888fb6e8
...
...
@@ -20,7 +20,7 @@ Glossary
generator
Term to refer to the tool that generates the
:term:`bandwidth list file` s. Often used as a syno
m
ym for
:term:`bandwidth list file` s. Often used as a syno
n
ym for
:term:`scanner`.
bandwidth list file
...
...
@@ -44,6 +44,6 @@ Glossary
destination
The term used by ``sbws`` to refer to a Web server where the
:term:`scanner` request files to perform the bandwith measurements.
:term:`scanner` request files to perform the bandwi
d
th measurements.
.. [#] https://metrics.torproject.org/glossary.html
docs/source/how_works.rst
View file @
888fb6e8
...
...
@@ -26,7 +26,7 @@ by a :term:`directory authority` to report relays’ bandwidth in its vote.
.. image:: ./images/dirauths_bwauths.png
Intialization
In
i
tialization
--------------
.. At some point it should be able to get environment variables
...
...
docs/source/implementation.rst
View file @
888fb6e8
...
...
@@ -10,7 +10,7 @@ re-implement the feature:
Possible advantages using other software:
- zero maint
ai
nance
- zero maint
e
nance
- not reinventing the wheel
Possible disadvantages using other software:
...
...
@@ -36,10 +36,10 @@ A first solution would be to obtain the git revision at runtime, but:
the git revision of that other repository.
So next solution was to obtain the git revision at build/install time.
To achive this, an script should be called from the installer or at runtime
To achi
e
ve this, an script should be called from the installer or at runtime
whenever `__version__` needs to be read.
While it could be implemented by us, there're two external tools that achive
While it could be implemented by us, there're two external tools that achi
e
ve
this.
setuptools_scm
...
...
@@ -102,7 +102,7 @@ Changing Bandwidth file monitoring KeyValues
--------------------------------------------
In version 1.1.0 we added KeyValues call ``recent_X_count`` and
``relay_X_count`` which implied to modify se
r
veral parts of the code.
``relay_X_count`` which implied to modify several parts of the code.
We only stored numbers for simpliciy, but then the value of this numbers
accumulate over the time and there is no way to know to which number decrease
...
...
@@ -159,7 +159,7 @@ will be correct after 5 days and they have been wrong for long time.
Additionally ``recent_measurement_failure_count`` will be negative, since it's
calculated as ``recent_measurement_attempt_count`` minus all the results.
While the total number of results in the last 5 days is corr
r
ect, the number of
While the total number of results in the last 5 days is correct, the number of
the attempts won't be until 5 days have pass.
Disadvantages
...
...
docs/source/man_sbws.rst
View file @
888fb6e8
...
...
@@ -75,7 +75,7 @@ In the following list, the first path is the default location when running
**sbws** as a system service.
**$HOME/.sbws.ini** or **/etc/sbws/sbws.ini**
Location where **sbws** searchs for a custom configuration file, when the
Location where **sbws** search
e
s for a custom configuration file, when the
option **--config** is not provided.
**$HOME/.sbws** or **/var/lib/sbws**
...
...
docs/source/roadmap.rst
View file @
888fb6e8
...
...
@@ -6,7 +6,7 @@ Roadmap
Release 1.0.0
--------------
Autum 2018
Autum
n
2018
- Minimal Viable Product (MVP)
Release 1.1.0
...
...
docs/source/testing.rst
View file @
888fb6e8
...
...
@@ -3,7 +3,7 @@
Installing tests dependencies and running tests
==================================================
To run the tests, extra Python depenencies are needed:
To run the tests, extra Python depen
d
encies are needed:
- Flake8_
- tox_
...
...
sbws/core/generate.py
View file @
888fb6e8
...
...
@@ -121,7 +121,7 @@ def gen_parser(sub):
"--min-num"
,
default
=
NUM_MIN_RESULTS
,
type
=
int
,
help
=
"Mini
n
um number of a results to consider them."
,
help
=
"Mini
m
um number of a results to consider them."
,
)
return
p
...
...
sbws/core/scanner.py
View file @
888fb6e8
...
...
@@ -355,7 +355,7 @@ def measure_relay(args, conf, destinations, cb, rl, relay):
return
[
ResultError
(
relay
,
[],
""
,
our_nick
,
msg
=
reason
),
]
# Pick a desti
o
nation
# Pick a destination
dest
=
destinations
.
next
()
# When there're no any functional destinations.
if
not
dest
:
...
...
@@ -686,7 +686,7 @@ def main_loop(
the ``Result`` from the queue and write it to disk, so this doesn't block
the measurement threads.
If there was an exception not ca
tched
by ``measure_relay``, it will call
If there was an exception not ca
ught
by ``measure_relay``, it will call
instead ``result_putter_error``, which logs the error and complete
immediately.
...
...
@@ -774,7 +774,7 @@ def wait_for_results(num_relays_to_measure, pending_results):
While there are relays being measured, just log the progress
and sleep :const:`~sbws.globals.TIMEOUT_MEASUREMENTS` (3mins),
which is aproximately the time it can take to measure a relay in
which is ap
p
roximately the time it can take to measure a relay in
the worst case.
When there has not been any relay measured in ``TIMEOUT_MEASUREMENTS``
...
...
@@ -914,7 +914,7 @@ def run_speedtest(args, conf):
except
KeyboardInterrupt
:
log
.
info
(
"Interrupted by the user."
)
stop_threads
(
signal
.
SIGINT
,
None
)
# Any exception not ca
tched
at this point would make the scanner stall.
# Any exception not ca
ught
at this point would make the scanner stall.
# Log it and exit gracefully.
except
Exception
as
e
:
log
.
critical
(
FILLUP_TICKET_MSG
)
...
...
sbws/core/stats.py
View file @
888fb6e8
...
...
@@ -51,7 +51,7 @@ def _result_type_per_relay(data, result_type):
def
_get_box_plot_values
(
iterable
):
"""Re
u
trn the min, q1, med, q1, and max of the input list or iterable.
"""Ret
u
rn the min, q1, med, q1, and max of the input list or iterable.
This function is NOT perfect, and I think that's fine for basic statistical
needs. Instead of median, it will return low or high median. Same for q1
and q3."""
...
...
sbws/globals.py
View file @
888fb6e8
...
...
@@ -33,7 +33,7 @@ TORRC_STARTING_POINT = {
"SafeLogging"
:
"0"
,
"LogTimeGranularity"
:
"1"
,
"ProtocolWarnings"
:
"1"
,
# To be able to respond
e
to MaxAdvertisedBandwidth as soon as possible.
# To be able to respond to MaxAdvertisedBandwidth as soon as possible.
# If ``FetchDirInfoExtraEarly` is set, but not
# `FetchDirInfoEarly`, Tor will throw this error:
# `FetchDirInfoExtraEarly requires that you also set FetchDirInfoEarly`
...
...
@@ -52,7 +52,7 @@ TORRC_RUNTIME_OPTIONS = {
# so do not let Tor to build them.
"__DisablePredictedCircuits"
:
"1"
,
# The scanner attach the streams to the circuit itself,
# so do not let Tor to attach
e
them.
# so do not let Tor to attach them.
"__LeaveStreamsUnattached"
:
"1"
,
}
# Options that can be set at runtime and can fail with some Tor versions
...
...
sbws/lib/circuitbuilder.py
View file @
888fb6e8
...
...
@@ -96,7 +96,7 @@ class CircuitBuilder:
# In a future refactor, remove this class, since sbws chooses the relays to
# build the circuit, the relays are not just cho
o
sen as random as this class
# build the circuit, the relays are not just chosen as random as this class
# does.
class
GapsCircuitBuilder
(
CircuitBuilder
):
"""Same as ``CircuitBuilder`` but implements build_circuit."""
...
...
sbws/lib/destination.py
View file @
888fb6e8
...
...
@@ -164,7 +164,7 @@ class Destination:
num_attempts_stored
=
NUM_DESTINATION_ATTEMPTS_STORED
,
factor_increment_retry
=
FACTOR_INCREMENT_DESTINATION_RETRY
,
):
"""Initalizes the Web server from which the data is downloaded.
"""Init
i
alizes the Web server from which the data is downloaded.
:param str url: Web server data URL to download.
:param int max_dl: Maximum size of the the data to download.
...
...
@@ -192,8 +192,8 @@ class Destination:
self
.
_max_seconds_between_retries
=
max_seconds_between_retries
# Using a deque (FIFO) to do not grow forever and
# to do not have to remove old attempts.
# Store tuples of timestamp and whether the destination succed or not
# (succed, 1, failed, 0).
# Store tuples of timestamp and whether the destination succe
e
d or not
# (succe
e
d, 1, failed, 0).
# Initialize it as if it never failed.
self
.
_attempts
=
collections
.
deque
(
[
...
...
@@ -268,7 +268,7 @@ class Destination:
And False otherwise.
When the destination is tried again after the consecutive failures,
the time to try again is incremented and reset
ted
as soon as the
the time to try again is incremented and reset as soon as the
destination does not fail.
"""
# NOTE: does a destination fail because several threads are using
...
...
sbws/lib/relaylist.py
View file @
888fb6e8
...
...
@@ -138,7 +138,7 @@ class Relay:
def
consensus_bandwidth_is_unmeasured
(
self
):
# measured appears only votes, unmeasured appears in consensus
# therefore is_unmeasured is needed to know whether the bandwidth
# value in consensus is com
m
ing from bwauth measurements or not.
# value in consensus is coming from bwauth measurements or not.
return
self
.
_from_ns
(
"is_unmeasured"
)
@
property
...
...
sbws/lib/relayprioritizer.py
View file @
888fb6e8
...
...
@@ -100,7 +100,7 @@ class RelayPrioritizer:
those measurements.
:param bool prioritize_result_error: whether prioritize or not
measurements that did not succed.
measurements that did not succe
e
d.
:param bool return_fraction: whether to return only a fraction of the
relays seen in the network or return all.
...
...
@@ -177,7 +177,7 @@ class RelayPrioritizer:
# In a future refactor, a new attribute should not be created,
# then no need to remove it.
del
relay
.
priority
# Increment the number of times a realy was "prioritized" to be
# Increment the number of times a real
l
y was "prioritized" to be
# measured.
relay
.
increment_relay_recent_priority_list
()
yield
relay
sbws/lib/resultdump.py
View file @
888fb6e8
...
...
@@ -130,7 +130,7 @@ def trim_results_ip_changed(
return
new_results_dict
if
on_changed_ipv6
is
True
:
log
.
warning
(
"Reseting bandwidth results when IPv6 changes,"
"Reset
t
ing bandwidth results when IPv6 changes,"
" is not yet implemented."
)
return
result_dict
...
...
@@ -258,7 +258,7 @@ class Result:
"""
Initializes a ``Result.Relay``.
.. note:: in a future refactor the attributes should be d
i
namic
.. note:: in a future refactor the attributes should be d
y
namic
to easy adding/removing them.
They are shared by :class:`~sbws.lib.relaylist.Relay` and
:class:`~sbws.lib.v3bwfile.V3BWLine` and there should not be
...
...
@@ -283,7 +283,8 @@ class Result:
def
__init__
(
self
,
relay
,
circ
,
dest_url
,
scanner_nick
,
t
=
None
):
"""
Initilizes the measurement and the relay with all the relay attributes.
Initializes the measurement and the relay with all the relay
attributes.
"""
self
.
_relay
=
Result
.
Relay
(
relay
.
fingerprint
,
...
...
@@ -351,7 +352,7 @@ class Result:
@
property
def
relay_recent_measurement_attempt
(
self
):
"""Returns the relay recent measurements attemps.
"""Returns the relay recent measurements attemp
t
s.
It is initialized in :class:`~sbws.lib.relaylist.Relay` and
incremented in :func:`~sbws.core.scanner.main_loop`.
...
...
@@ -614,7 +615,7 @@ class ResultErrorSecondRelay(ResultError):
the second relay is an exit without `bad` flag and can exit to port 443.
- If the relay to measure is an exit, the second relay is not an exit.
It is instan
c
iated in :func:`~sbws.core.scanner.measure_relay`.
It is instan
t
iated in :func:`~sbws.core.scanner.measure_relay`.
.. note:: this duplicates code and add more tech-debt,
since it's the same as the other
...
...
@@ -666,7 +667,7 @@ class ResultErrorDestination(ResultError):
"""
Error when there is not a working destination Web Server.
It is instan
c
iated in :func:`~sbws.core.scanner.measure_relay`.
It is instan
t
iated in :func:`~sbws.core.scanner.measure_relay`.
.. note:: this duplicates code and add more tech-debt,
since it's the same as the other
...
...
sbws/lib/v3bwfile.py
View file @
888fb6e8
...
...
@@ -56,7 +56,7 @@ HEADER_KEYS_V1_1_ORDERED = ["version"]
# They can also be pass as an argument to `Header` to overwrite default values,
# what is done in unit tests.
# `latest bandwidth` is special cause it gets its value from timestamp, which
# is not a KeyValue, but it's always pass as an agument.
# is not a KeyValue, but it's always pass as an a
r
gument.
# It could be separaed in other list, but so far there is no need, cause:
# 1. when it's pass to the Header to initialize it, it's just ignored.
# 2. when the file is created, it's took into account.
...
...
@@ -175,7 +175,7 @@ LINE_TERMINATOR = TERMINATOR + LINE_SEP
# KeyValue separator in Bandwidth Lines
BWLINE_KEYVALUES_SEP_V1
=
" "
# not inclding in the files the extra bws for now
# not incl
u
ding in the files the extra bws for now
BWLINE_KEYS_V0
=
[
"node_id"
,
"bw"
]
BWLINE_KEYS_V1_1
=
[
"master_key_ed25519"
,
...
...
@@ -590,7 +590,7 @@ class V3BWHeader(object):
return
len
(
self
.
__str__
().
split
(
LINE_SEP
))
def
add_stats
(
self
,
**
kwargs
):
# Using kwargs because attributes might chage.
# Using kwargs because attributes might cha
n
ge.
[
setattr
(
self
,
k
,
str
(
v
))
for
k
,
v
in
kwargs
.
items
()
...
...
@@ -958,7 +958,7 @@ class V3BWLine(object):
@
staticmethod
def
rtt_from_results
(
results
):
# convert from miliseconds to seconds
# convert from mil
l
iseconds to seconds
rtts
=
[(
round
(
rtt
*
1000
))
for
r
in
results
for
rtt
in
r
.
rtts
]
rtt
=
round
(
median
(
rtts
))
if
rtts
else
None
return
rtt
...
...
sbws/util/fs.py
View file @
888fb6e8
...
...
@@ -11,7 +11,7 @@ Disk space requirements
v3bw files: the maximum space required is ~{mb_bw} MB, after {d_bw} days.
result files: the maximum space required is ~{mb_results} MB, after {d_r} days.
tor directory: the space required is ~{mb_tor} MB.
code and depenencies: the space required is ~{mb_code} MB
code and depen
d
encies: the space required is ~{mb_code} MB
Total disk space required is: ~{mb_total} MB
"""
...
...
@@ -28,7 +28,7 @@ def sbws_required_disk_space(conf):
size_v3bw_file
=
7500
*
220
# default crontab configuration will run genenerate every hour
num_v3bw_files_day
=
24
# ~1000 is the length of a line when the result is successful
l
# ~1000 is the length of a line when the result is successful
# ~4550 is the number of lines of the biggest result file
size_result_file
=
4550
*
1000
num_result_files_day
=
1
...
...
@@ -66,7 +66,7 @@ def sbws_required_disk_space(conf):
text_dict
[
"mb_results"
]
=
round
(
results_max_space_after_delete
/
1000
**
2
)
text_dict
[
"d_r"
]
=
results_delete_after_days
# not counted rotated files and assuming that when it is not rotated the
# size will be aproximately 10MiB
# size will be ap
p
roximately 10MiB
space_log_files
=
0
if
conf
.
getboolean
(
"logging"
,
"to_file"
):
size_log_file
=
conf
.
getint
(
"logging"
,
"to_file_max_bytes"
)
...
...
Prev
1
2
Next
Georg Koppen
@gk
mentioned in merge request
!77 (closed)
·
Mar 17, 2021
mentioned in merge request
!77 (closed)
mentioned in merge request !77
Toggle commit list
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment