Commit eca0d268 authored by Matt Traudt's avatar Matt Traudt
Browse files

Merge branch 'ticket27341'

parents 48715b7c 9239f69e
......@@ -12,6 +12,18 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
- Broken environment variable in default sbws config. To use envvar $FOO, write
$$FOO in the config.
### Changed
- sbws install doc is confusing (#27341)
- Include system and Python dependencies in `INSTALL`.
- Include dependencies for docs and tests in `INSTALL`.
- Point to `DEPLOY` to run sbws.
- Remove obsolete sections in `INSTALL`
- Simplify `DEPLOY`, reuse terms in the `glossary`.
- Remove obsolete ``sbws init`` from `DEPLOY`.
- Point to config documentation.
- Add, unify and reuse terms in `glossary`.
## [0.7.0] - 2018-08-09
**Important changes**:
......
.. _deploy:
Deploying Simple Bandwidth Scanner
=====================================
So you want to run sbws for yourself. You will need
- A machine to measure from, hereafter referred to as |scanner_nick|.
- One or more webservers hosting a large file, hereafter referred to as
|dests|.
Both |scanner_nick| and your |dests| should be on fast, well connected
machines.
- A machine to run the :term:`scanner`.
- One or more :term:`destination` (s) that serve a large file.
|scanner_nick| requirements
-------------------------------
Both :term:`scanner` and your :term:`destination` (s) should be on fast,
well connected machines.
- A fast, well connected machine on the Internet. Ideally its bandwidth should
not be a limiting factor in measurements.
- Linux host OS
- Python 3, virtualenv
- Tor installed with ``tor`` in your ``PATH``
.. _destinations_requirements:
|dests| requirements
-------------------------------
:term:`destination` requirements
------------------------------------
- A fast, well connected machine on the Internet. Ideally its bandwidth should
not be a limiting factor in measurements.
- Some sort of webserver installed and running that supports HEAD and GET
requests (apache and nginx fit this description)
- A Web server installed and running that supports HTTP GET, HEAD and
Range (:rfc:`7233`) requests.
``Apache`` HTTP Server and ``Nginx`` support them.
- Optional support for TLS
- A large file; at the time of writing, at least 1 GiB in size
|scanner_nick| setup
-------------------------------
:term:`scanner` setup
----------------------
Install sbws according to :doc:`/INSTALL`.
Make sure you have initialized sbws. If you haven't already run ``sbws init``,
then enter your virtualenv and run it.
Inside |dotsbws| you will find ``config.ini``. Open it with a text editor. it
should be very simple. Let's give our scanner a nickname. Add the following
lines
::
[scanner]
nickname = D0ntD3@dOpen!nside
(Pick your own nickname. This one just demonstrates that you can use almost any
character)
Congratulations, you've learned how to add a section to your config file and
how to add an option to a section.
Remeber |dests|? We need to add them to ``config.ini``. We're going to assume
you have two you are ready to use and one that isn't quite ready yet.
::
[destinations]
foo = on
bar = on
baz = off
[destinations.foo]
url = http://fooshoomoo.com/sbws.bin
[destinations.bar]
url = https://barstoolsinc.com
[destinations.baz]
url = https://bazistan.com/ask/stan/where/the/file/is.exe
``foo`` demonstrates a typical case.
``bar`` demonstrates a case where you want to use HTTPS and want to assume the
large file for sbws to download is at its default path (probably
``/sbws.bin``).
``baz`` demonstrates a disabled destination that sbws will ignore.
``sbws scanner`` needs :term:`destination` (s) to request files from.
They are not included by default.
To configure destinations, create a configuration file according to
:doc:`man_sbws.ini`
......@@ -3,175 +3,109 @@
Installing Simple Bandwidth Scanner
===================================
(At the time of writing) sbws depends on two Python libraries.
The prefered method to install ``sbws`` is to install it from your system
distribution.
Currently there is not any system distribution package.
In the meantime, follow the following steps.
- Stem_
- Requests_
System requirements
--------------------
Sbws relies on a stem feature that is not planned to be in a tagged release
until stem 1.7.0.
- Tor
- Python 3 (>= 3.5)
- virtualenv_ (while there is not ``stem`` release > 1.6.0, it is
recommended to install the required python dependencies in a virtualenv)
Read all the information for the installation method of your choice before
beginning. Often you will want to be armed with the knowledge of the latest
released version of sbws. Determine that by examining its git tags, or visiting
its `release page`_. In the remainder of this document, we assume the latest
version is 1.5.3, which would be tagged as ``v1.5.3``.
In Debian::
.. _Stem: https://stem.torproject.org/
.. _Requests: http://docs.python-requests.org/
.. _release page: https://github.com/pastly/simple-bw-scanner/releases
Virtualenv - Development
------------------------------------------------------------------------------
Choose a directory to store code in. I might choose ``~/src``.
sudo apt install tor python3 virtualenv
::
cd ~/src
Get stem
~~~~~~~~~~~~~
::
Python dependencies
--------------------
git clone https://git.torproject.org/stem.git
- Stem_ > 1.6.0
- Requests_ (with socks_ support) >= 2.10.0
Get sbws
~~~~~~~~~~~~~~~~~
To install the Python dependencies, create a ``virtualenv`` first
::
git clone https://github.com/pastly/simple-bw-scanner.git
cd simple-bw-scanner
Create and enter virtualenv
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I like to keep mine in my simple-bw-scanner directory and assume that's where
you'll put it.
::
virtualenv -p python3 venv
virtualenv venv -p /usr/bin/python3
source venv/bin/activate
Install stem in virtualenv
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Clone ``sbws``::
Sbws requires on features not yet in a released version of stem (1.7.0 has not
been released as of the time of writing). You can either install from the
master branch, or checkout ``60f034ad8b9c3aa48e7e2ecb0a2e159b6ed5bc71`` or
newer.
git clone https://gitweb.torproject.org/sbws.git
::
Install the python dependencies::
pip install ../stem
cd sbws && pip install --process-dependency-links .
Install sbws and its remaining dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. note:: ``process-dependency-links`` will clone ``stem`` from master and
install it. It's deprecated, but it won't be needed as soon as there is
an ``stem`` release > 1.6.0
Here is where you might want to know what the latest release of sbws is (this
document assumes it is 1.5.3). Skip
the ``git checkout`` if you want to run the bleeding edge tip-of-master version
of sbws.
``sbws`` needs :term:`destination` s to request files from.
Please, see :ref:`deploy` to know how to configure, deploy and run ``sbws``.
::
Installing tests dependencies and running them
------------------------------------------------
git checkout v1.5.3
pip install .
To run the tests, extra Python depenencies are needed:
Run sbws
~~~~~~~~~~
- Flake8_
- tox_
- pytest_
- coverage_
If you would like to use a custom configuration file you can create it in
``~/.sbws.ini`` or provide ``sbws`` with it via ``-c`` ``--config`` option.
To install them from ``sbws`` ::
See the documentation section about configuration files for more information
about how to create a configuration file.
pip install .[dev] && pip install .[test]
Tor run the scanner run
::
To run the tests::
sbws scanner
tox
[OBSOLETE DO NOT FOLLOW] Virtualenv - Production
------------------------------------------------------------------------------
Installing documentation dependendencies and building it
---------------------------------------------------------
Installing
~~~~~~~~~~
To build the documentation, extra Python dependencies are needed:
::
git clone https://github.com/pastly/simple-bw-scanner.git
cd simple-bw-scanner
git checkout v1.5.3
virtualenv -p python3 venv
source venv/bin/activate
pip install --process-dependency-links .
sbws init
.. note::
Because we relay on a ``-dev`` version of stem, we need to fetch it from
git.torproject.org. Thus ``--process-dependency-links`` is necessary.
.. warning::
Run these commands one at a time and check for errors before continuing.
Updating
~~~~~~~~
::
- Sphinx_
- recommonmark_
- Pylint_ (only to update the diagrams)
cd simple-bw-scanner
git pull
# Determine the newest released version. Assuming it is v1.5.3 ...
git checkout v1.5.3
source venv/bin/activate
pip install --process-dependency-links --upgrade-strategy eager --upgrade .
To install them from ``sbws``::
pip install .[doc]
[OBSOLETE DO NOT FOLLOW] Virtualenv - Development
------------------------------------------------------------------------------
To build the documentation as HTML::
These are almost exactly the same. The difference is the pip command: we
install sbws in an editable state so we don't have to re-run pip every time we
make a change.
cd docs/ && make html
::
The generated HTML will be in ``docs/build/``.
git clone https://github.com/pastly/simple-bw-scanner.git
cd simple-bw-scanner
git checkout v1.5.3
virtualenv -p python3 venv-editable
source venv-editable/bin/activate
pip install --process-dependency-links --editable .
sbws init
To build the manual (``man``) pages::
.. note::
cd docs/ && make man
Because we relay on a ``-dev`` version of stem, we need to fetch it from
git.torproject.org. Thus ``--process-dependency-links`` is necessary.
The generated man pages will be in ``docs/man/``.
.. warning::
To build the documentation diagrams::
Run these commands one at a time and check for errors before continuing.
cd docs/ && make umlsvg
Updating
~~~~~~~~
::
The generated diagrams will be in ``docs/build/_images/``.
cd simple-bw-scanner
git pull
# Determine the newest released version. Assuming it is v1.5.3 ...
git checkout v1.5.3
.. todo::
This doesn't update dependencies and needs to be fixed.
.. _virtualenv: https://virtualenv.pypa.io/en/stable/installation/
.. _Stem: https://stem.torproject.org/
.. _socks: http://docs.python-requests.org/en/master/user/advanced/#socks
.. _Requests: http://docs.python-requests.org/
.. _Flake8: http://flake8.pycqa.org/
.. _pytest: https://docs.pytest.org/
.. _tox: https://tox.readthedocs.io
.. _Coverage: https://coverage.readthedocs.io/
.. _Sphinx: http://www.sphinx-doc.org
.. _recommonmark: https://recommonmark.readthedocs.io/
.. _Pylint: https://www.pylint.org/
......@@ -111,14 +111,6 @@ how its code is laid out. Therefore the code may change drastically without a
major version bump as long as the way users interact with it does not change in
a backward incompatible way.
### Build HTML documentation
pip install -e .[doc]
cd docs
make html
The generated HTML will be in `docs/build/`.
## The `.sbws` directory
By default is `~/.sbws`.
......@@ -133,12 +125,3 @@ In this directory you will find
- `v3bw/` This directory stores the v3bw files created with `sbws generate`.
- `state.dat` A file for storing state needed between sbws commands. See its
documentation for more information.
## Running tests
Make sure you have test dependencies installed. From within the top level
repository directory:
pip install -e .[test]
This should install tox and pytest. Then simply run `tox`.
......@@ -4,32 +4,46 @@ Glossary
.. glossary::
directory authority
One of the trusted entities in the Tor network. Calculates a consensus
with the other directory authorities about what relays are running and
much much more.
a special-purpose relay that maintains a list of currently-running
relays and periodically publishes a consensus together with the other
directory authorities. [#]_
bandwidth authority
A :term:`directory authority` that measures and votes about Tor relay
bandwidths. In the context of sbws, it performs measurements with
:term:`sbws scanner` and creates :term:`v3bw files <v3bw file>`
periodically with :term:`sbws generate`
A :term:`directory authority` that runs a :term:`scanner` and a
:term:`generator` or obtain :term:`bandwidth list file` s from a
:term:`generator`.
scanner
Term to refer to the process that measures the relays' bandwidth.
It is also called :term:`generator` when it is the same tool that is
used to generate :term:`bandwidth list file` s.
generator
Term to refer to the tool that generates the
:term:`bandwidth list file` s. Often used as a synomym for
:term:`scanner`.
bandwidth list file
The file generated by :term:`generator` s that is read by the
:term:`directory authority` s and included in their votes.
See `bandwidth-file-spec.txt <https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txt>`_
to know about the file specification.
sbws scanner
The sbws command that a :term:`bandwidth authority` runs continuously
to gather information about how fast Tor relays are. The results from
this process are used by :term:`sbws generate` to generate
:term:`v3bw files <v3bw file>`.
The ``sbws`` command used to run ``sbws`` as a :term:`scanner`.
sbws generate
The sbws command that a :term:`bandwidth authority` runs periodically
to generate a new :term:`v3bw file` from results gathered by
:term:`sbws scanner`.
The ``sbws`` command used to run ``sbws`` as a :term:`generator`.
v3bw file
The file generated by :term:`sbws generate` that a :term:`bandwidth
authority` reads to vote about how fast relays are.
The term used by ``sbws`` to refer to :term:`bandwidth list file`
v1.1.0.
.. literalinclude:: v3bw.txt
:caption: A v3bw file
.. literalinclude:: v3bw.txt
:caption: A v3bw file
.. glossary::
destination
The term used by ``sbws`` to refer to a Web server where the
:term:`scanner` request files to perform the bandwith measurements.
.. [#] https://metrics.torproject.org/glossary.html
......@@ -6,6 +6,13 @@ DESCRIPTION
Tor bandwidth scanner configuration file.
**sbws** (1) ``scanner`` command requires a configuration file with a
"[destinations]" section.
It is the only section that does not have a default value.
It is recommended, but not required to configure "nickname" in the "[scanner]"
section.
SECTIONS
---------
......@@ -54,7 +61,8 @@ destinations
destinations.STR
url = STR
The url to the destination
The URL to the destination. It must include a file path.
It can use both http or https.
tor
datadir = STR
......@@ -147,15 +155,36 @@ logging
Format string to use when logging to syslog.
(Default: %(module)s[%(process)s]: <%(levelname)s> %(message)s)
EXAMPLES
--------
Example ``destinations`` section::
[destinations]
foo = on
bar = on
baz = off
[destinations.foo]
# using HTTP
url = http://example.org/sbws.bin
[destinations.bar]
# using HTTPS
url = https://example.com/data
[destinations.baz]
# this will be ignored
url = https://example.net/ask/stan/where/the/file/is.exe
FILES
-----
$HOME/.sbws.ini
Default location for the sbws user configuration file.
Default ``sbws`` user configuration path.
$HOME/.sbws
Default sbws home, where it stores measurement data files,
bandwidth list files and tor process data.
Any other path to the configuration file can be specified using the
``sbws`` argument ``-c``
SEE ALSO
---------
......
......@@ -40,6 +40,9 @@ Optional arguments
EXAMPLES
--------
sbws scanner
Run the scanner using the configuration file in ~/.sbws.ini
sbws -c ~/.sbwsrc scanner
Run the scanner using the configuration file in ~/.sbwsrc
......@@ -59,6 +62,9 @@ $HOME/.sbws
Default sbws home, where it stores measurement data files,
bandwidth list files and tor process data.
$XDG_RUNTIME_DIR/sbws
Runtime directory for the tor process launched by ``sbws``.
SEE ALSO
---------
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment