Commit 84e5d82c authored by juga's avatar juga
Browse files

destination: record consecutive failures 

Add methods to store consecutive destination failures and retrieve
the destinations that are still functional.
Since destinations can fail because of Tor circuits, it's not count
individual failures but consecutives one.
Also exit with error if there are no functional destinations left.
The maximum number of consecuitve failures is set to 10, but it
may need to be changed depending on the percentage of circuits and
requests that fail.
parent d33ac8f6

DEPLOY.rst

+2 −0
Original line number Diff line number Diff line
@@ -38,6 +38,8 @@ or `<INSTALL.html>`_ (local build or Read the Docs). 


To run the ``scanner`` it is mandatory to create a configuration file with at

least one ``destination``.

It is recommended to set several ``destination``s so that the ``scanner`` can

continue if one fails.



If ``sbws`` is installed from the Debian package, then create a file in

``/etc/sbws/sbws.ini`` like in the following example:

docs/source/examples/sbws.example.ini

+6 −1
Original line number Diff line number Diff line
@@ -4,7 +4,12 @@ 
nickname = sbws_default



[destinations]

# a destination can be disabled changing `on` by `off`

# With several destinations, the scanner can continue even if some of them

# fail, which can be caused by a network problem on their side.

# If all of them fail, the scanner will stop, which

# will happen if there is network problem on the scanner side.



# A destination can be disabled changing `on` by `off`

foo = on



[destinations.foo]

docs/source/man_sbws.ini.rst

+5 −0
Original line number Diff line number Diff line
@@ -58,6 +58,11 @@ paths 
    (Default ~/.sbws/log)



destinations



  It is required to set at least one destination for the scanner to run.

  It is recommended to set several destinations so that the scanner can

  continue if one fails.



  STR = {on, off}

    Name of destination. It is a name for the Web server from where to

    download files in order to measure bandwidths.

sbws/core/scanner.py

+6 −3
Original line number Diff line number Diff line
@@ -244,11 +244,14 @@ def measure_relay(args, conf, destinations, cb, rl, relay): 
        cb.controller, conf.getfloat('general', 'http_timeout'))

    # Pick a destionation

    dest = destinations.next()

    # If there is no any destination at this point, it can not continue.

    if not dest:

        # XXX: this should return a ResultError

        log.debug('Unable to get destination to measure %s %s',

                  relay.nickname, relay.fingerprint)

        return None

        log.critical("There are not any functional destinations.\n"

                     "It is recommended to set several destinations so that "

                     "the scanner can continue if one fails.")

        # This should raise an error so that the caller can close the pool.

        exit(1)

    # Pick a relay to help us measure the given relay. If the given relay is an

    # exit, then pick a non-exit. Otherwise pick an exit.

    helper = None

sbws/globals.py

+5 −0
Original line number Diff line number Diff line
@@ -108,6 +108,11 @@ HTTP_GET_HEADERS = { 
    'Accept-Encoding': 'identity',

}

DESTINATION_VERIFY_CERTIFICATE = True

# This number might need adjusted depending on the percentage of circuits and

# HTTP requests failures.

# While the scanner can not recover from some/all failing destionations,

# set a big number so that it continues trying.

MAXIMUM_NUMBER_DESTINATION_FAILURES = 100





def fail_hard(*a, **kw):