Commit fb477a09 authored by Arturo Filastò's avatar Arturo Filastò
Browse files

Change the timestamp format to use ISO8601.

* Write unittests for timestamp generation functions
* Add documentation on how reports will be stored in the backend
parent 18c2dbd0
Loading
Loading
Loading
Loading
+3 −3
Changes for HACKING: 3 added lines, 3 removed lines.
Original line number Diff line number Diff line
@@ -75,6 +75,9 @@ Code Structure
  Handles running ooni.nettests as well as
  ooni.plugoo.tests.OONITests.

- ooni/otime.py
  Generation of timestamps, time conversions and all the rest

- ooni/kit/
  In here go utilities that can be used by tests.

@@ -102,9 +105,6 @@ Code Structure
  Utilities for working with Tor.
  XXX this code should be removed and merged into txtorcon.

- ooni/utils/otime.py
  Generation of timestamps, time conversions and all the rest

- ooni/utils/txscapy.py
  Tools for making scapy work well with twisted.

+43 −0
Changes for docs/source/reports.rst: 43 added lines, 0 removed lines.
Original line number Diff line number Diff line
Reports
=======

The reports collected by ooniprobe are stored on
https://ooni.torproject.org/reports/`reportFormatVersion`/`CC`/

Where `CC` is the two letter country code as specified by
http://en.wikipedia.org/wiki/ISO_3166-2.

For example the reports for Italy (`CC` is `it`) of the `reportVersion` 0.1 may
be found in:

::
  https://ooni.torproject.org/reports/0.1/IT/


This directory shall contain the various reports for the test using the
following convention:

`ISO8601`-AS`probeASNumber`.yamloo

If a collision is detected then an int (starting with 1) will get appended to
the test.

For example a report that was created on the first of January 2012 at Noon (UTC
time) sharp from MIT (AS3) will be stored here:

::

  https://ooni.torproject.org/reports/0.1/US/2012-01-01T120000Z-AS3.yamloo


Report format version changelog
===============================

In here shall go details about he major changes to the reporting format.

version 0.1
-----------

Initial format version.

+1 −1
Changes for ooni/config.py: 1 added line, 1 removed line.
Original line number Diff line number Diff line
@@ -8,7 +8,7 @@ import yaml

from twisted.internet import reactor, threads, defer

from ooni.utils import otime
from ooni import otime
from ooni.utils import Storage

reports = Storage()
+2 −2
Changes for ooni/kit/__init__.py: 2 added lines, 2 removed lines.
Original line number Diff line number Diff line
__all__ = ['domclass']
from . import domclass
#__all__ = ['domclass']
#from . import domclass
+51 −10
Changes for ooni/otime.py: 51 added lines, 10 removed lines.
Original line number Diff line number Diff line
# -*- encoding: utf-8 -*-
#
# :authors: Arturo Filastò
# :licence: see LICENSE

"""
Here is the location for all time and date related utility functions.
"""
@@ -42,10 +37,56 @@ def utcPrettyDateNow():
def timeToPrettyDate(time_val):
    return time.ctime(time_val)

def timestamp():
    cur_time = datetime.utcnow()
    d_format = "%d_%B_%Y_%H-%M-%S"
    pretty = cur_time.strftime(d_format)
    return pretty
class InvalidTimestampFormat(Exception):
    pass

def fromTimestamp(s):
    """
    Converts a string that is output from the timestamp function back to a
    datetime object

    Args:
        s (str): a ISO8601 formatted string.
            ex. 1912-06-23T101234Z"

    Note: we currently only support parsing strings that are generated from the
        timestamp function and have no intention in supporting the full standard.
    """
    try:
        date_part, time_part = s.split('T')
        hours, minutes, seconds = time_part[:2], time_part[2:4], time_part[4:6]
        year, month, day = date_part.split('-')
    except:
        raise InvalidTimestampFormat(s)

    return datetime(int(year), int(month), int(day), int(hours), int(minutes),
            int(seconds))

def timestamp(t=None):
    """
    The timestamp for ooni reports follows ISO 8601 in
    UTC time format.
    We do not inlcude ':' and include seconds.

    Example:

        if the current date is "10:12:34 AM, June 23 1912" (datetime(1912, 6,
            23, 10, 12, 34))

        the timestamp will be:

           "1912-06-23T101234Z"

    Args:
        t (datetime): a datetime object representing the
            time to be represented (*MUST* be expressed
            in UTC).

        If not specified will default to the current time
        in UTC.
    """
    if not t:
        t = datetime.utcnow()
    ISO8601 = "%Y-%m-%dT%H%M%SZ"
    return t.strftime(ISO8601)
Loading