Hiding client/exit information to address privacy concerns and fixes for...

CHANGE LOG

2/14/10 - version 1.3.2

2/27/10 - version 1.3.3

Hiding client/exit information to address privacy concerns and fixes for numerous issues brought up in irc.

    * added: scrubbing connection details of possible client and exit connections

    * change: providing file descriptions in README, updated known issues and future plans in TODO

    * change: added precision for bandwidth cap and burst if uneven values (requested by mete1989)

    * fix: HiddenService* parameters fetched via a special option (caught by dun, karsten, and grumpy3)

    * fix: workaround for os specific torrc validation bug - unfortunately haven't managed to repro yet so no fix (caught by grumpy3, Tas, and dun)

    * fix: checking for python curses bindings at startup (caught by dun)

    * fix: import error - TorCtl and socket missing from confPanel.py (caught by grumpy3)

    * fix: showing external ip in connection panel rather than local nat address (caught by mete1989)

    * fix: raised minimum width at which graph stats are displayed beside label (caught by dun)

    * fix: wasn't treating "accept *" and "reject *" as catch-all policies

    * fix: wasn't resizing graph panel properly in case of a sighup

2/14/10 - version 1.3.2 (r21646)

Refactoring goodness and bug fixes.

    * change: revised curses utilities to further simplify interfaces

    * change: substantial layout changes (separating into util and init packages) and including a copy of the gpl

    * change: revised curses utilities to further simplify interface implementations

    * change: substantial layout changes (adding util and init packages) and including a copy of the gpl

    * fix: bug with handing of DST for accounting's 'Time to reset' (patch provided by waltman)

    * fix: header and connection panels weren't accounting for having ORListenAddress set (caught by waltman)

    * fix: crashing bug when shrank too much for scrollbars to be drawn

This is started via 'arm' (use the '--help' argument for usage).

-------------------------------------------------------------------------------

FAQ:

> Why is it called 'arm'?

as being in a chroot jail) then it's probably failing due to permission issues.

Arm still runs, just no connection listing or ps stats.

-------------------------------------------------------------------------------

Layout:

./

  arm       - startup script

  ChangeLog - revision history

  LICENSE   - copy of the gpl v3

  README    - um... guess you figured this one out

  TODO      - known issues, future plans, etc

  screenshot_page1.png

  screenshot_page2.png

  init/

    __init__.py

    arm.py    - parses and validates commandline parameters

    prereq.py - checks python version and for required packages

  interface/

    __init__.py

    controller.py          - main display loop, handling input and layout

    headerPanel.py         - top of all pages, providing general information

    connResolver.py        - (daemon thread) periodic netstat lookups

    hostnameResolver.py    - (daemon thread) nonblocking reverse dns lookups

    graphPanel.py          - (page 1) presents graphs for data instances

    bandwidthMonitor.py    - (graph data) tracks tor bandwidth usage

    cpuMemMonitor.py       - (graph data) tracks tor cpu and memory usage

    connCountMonitor.py    - (graph data) tracks number of tor connections

    logPanel.py            - displays tor, arm, and torctl events

    fileDescriptorPopup.py - (popup) displays file descriptors used by tor

    connPanel.py           - (page 2) displays information on tor connections

    descriptorPopup.py     - (popup) displays connection descriptor data

    confPanel.py           - (page 3) displays torrc and performs validation

  util/

    __init__.py

    panel.py   - wrapper for safely working with curses subwindows

    uiTools.py - helper functions for interface

			fallback if an issue's detected.

			notify John Case <case@sdf.lonestar.org>

			caught by Christopher Davis

	* torrc validation bug reported (appears to be os specific)

			Reported instances were with Gentoo, OpenSuse, and OpenBSD. Change should be in:

			interface/confPanel.py lines 104-130

			setting up a Gentoo vm proved to be an absurd pain in the ass so gonna

			try repro in OpenSuse instead.

	* torrc validation doesn't catch if parameters are missing

	* revise multikey sort of connections

			Currently using a pretty ugly hack. Look at:

			http://www.velocityreviews.com/forums/

				t356461-sorting-a-list-of-objects-by-multiple-attributes.html

			and check for performance difference.

	* header panel isn't properly detecting catch-all exit policies

			Missing edge cases

	* avoid hostname lookups of private connections

			Stripped most of them but suspect there might be others (have assertions

			check for this in a debug mode?)

	* exit policy checks aren't handling all inputs

			Still need to handle masks, private keyword, and prepended policy,

			currently erroring on the side of caution.

	* not catching events unexpected by arm

			Future tor and TorCtl revisions could provide new events - these should

			be given the "UNKNOWN" type.

	* regex fails for multiline log entries

	* when logging no events still showing brackets

	* quitting can hang several seconds when there's hostnames left to resolve

			Not sure how to address this - problem is that the calls to 'host' can 

			take a while to time out. Might need another thread to kill the calls?

			Or forcefully terminate thread if it's taking too long (might be noisy)?

- Features / Site

	* rewrite codebase

			Currently the interface is a bit of a rat's nest (especially the

			controller). The goal is to use better modularization to both simplify

			the codebase and make it possible to use smarter caching to improve

			performance (far too much is done in the ui logic). This work is in

			progress, having started with the initialization (/init) and now

			concerning the utilities (/util). Migrating the following to util:

				- os calls (to provide transparent platform independence)

				- torrc validation

				- arm logging (static interface with listener design)

				- wrapper for tor connection, state, and data parsing (abstracting

					TorCtl connection should allow for arm to be resumed if tor restarts)

	* provide performance ARM-DEBUG events

			Help with diagnosing performance bottlenecks. This is pending the

			codebase revisions to figure out the low hanging fruit for caching.

	* condense tor/arm log listing types if they're the same

			Ie, make default "TOR/ARM NOTICE - ERR"

	* graph for arm cpu/mem usage

			Trivial to implement but not sure if this would be helpful.

	* startup option to restrict resource usage or set refresh rate

	* audit tor connections

			Provide warnings if tor misbehaves, checks possibly including:

				- ensuring ExitPolicyRejectPrivate is being obeyed

				- check that ExitPolicy violations don't occure (not possible yet since

					not all relays aren't identified)

				- check that all connections are properly related to a circuit, for

					instance no outbound connections without a corresponding inbound (not

					possible yet due to being unable to correlate connections to circuts)

	* abstract away netstat calls

			In preparation for drop in replacement of lsof or calls to tor's

			GETINFO.

			if set and there's extra room available show 'MaxAdvertisedBandwidth'

	* check family connections to see if they're alive (VERSION cell handshake?)

	* update site's screenshots (pretty out of date...)

	* look into providing UPnP support

			This might be provided by tor itself so wait and see...

- Ideas (low priority)

	* write up a proposal for the control protocol wishlist

	* look into providing UPnP support

	* python 3 compatability

			Currently blocked on TorCtl support.

	* bundle script that dumps relay stats to stdout

			Django has a small terminal coloring module that could be nice for

			formatting. Could possibly include:

				- desc / ns information for our relay

				- ps / netstat stats like load, uptime, and connection counts, etc

			derived from an idea by StrangeCharm

	* provide performance ARM-DEBUG events

			Might help with debugging bottlenecks. This requires that there's more

			refined controls for selecting logged arm runlevel.

	* show qos stats

			Take a look at 'linux-tor-prio.sh' to see if any of the stats are 

			available and interesting.

	* get a test environment for Mac OSX or BSD

			Set up a vm for FreeBSD but found working in it to be... painful (wasted

			five hours and gave up when even asking for a working copy of vim was 

			too much to ask). As for OSX seems that getting a test environment would

			cost quite a bit. Hence mothballing this - someone that actually uses

			these platforms will need to resolve portability issues if they arise.

	* localization

			Abstract strings from code and provide on translation portal. Thus far

			there hasn't been any requests for this.

			submit@bugs.debian.org with subject "RFP: arm" and starting with a line

			"Package: wnpp".

			requested by helmut

	* follow up on control-spec proposal

			Proposal and related information is available at:

			http://www.atagar.com/arm/controlSpecProposal.txt

- Control Protocol Wishlist (low priority)

	* listing of tor's current connections (netstat / lsof replacement)

			Keeping the netstat available would be good for auditing (external view

			of tor and more likely monitored by host based IDS) but tor's listing

			would probably be more effecient, accurate, and could contain additional

			details making it a preferable default.

	* bandwidth usage per connection

			This would need to be rounded and averaged over time to avoid 

			correlation problems. Probably the most interesting stat arm currently

			doesn't have since for most purposes (like security threats) especially

			active connections are of most interest.

	* identification of hop type

			Identification if the first, middle or last hop. When this is available

			I'll hide exit connections by default. Another interesting distinction

			would be when we're serving directory data verses acting as a relay.

	* associate connections to circuits

			Currently listing is connection based rather than circuit, ie it lists:

			previous hop -> localhost

			previous hop -> localhost

			localhost -> next hop

			rather than:

			previous hop -> localhost -> next hop

			previous hop -> localhost -> *unestablished*

			From a debugging and secuirty standpoint this could highlight potential

			issues, for instance relays really shouldn't have any non-client

			connections like:

			*unestablished* -> localhost -> next hop

			and entries like:

			previous hop -> localhost -> *extension failed (error X)*

			might indicate a firewall blocking tor outbound connections. This would

			be especially helpful if paired with server related circuit status

			events (which would note attempted extensions, failures, etc). We could

			also note other circuit based stats like the amount of buffered data.

	* mapping of ip/port to fingerprint

			Currently inferring the mappings but this only has around a 90% success

			rate (not sure why it fails...). Tor has an internal connection

			identifier so what would probably be best is bidirectional translation

			functions with that, ie getting fingerprint would be done via:

			ip/port -> connection id -> fingerprint

			In theory this should be able to tell us if the connection is the first

			or last hop (since in those cases the foreign address doesn't have a

			fingerprint).

	* additional get_info data

			effective relay bandwidth / burst - currently internally mimicing the

				logic of tor (which is RelayBandwidthRate/Burst if set, otherwise 

				BandwidthRate/Burst)

			list of directory authorities recognized by that instance of tor

			total data relayed by tor - this is already kinda tracked for accounting

			file descriptor limit (return value of the getrlimit() function)

			Unfortunatley this doesn't seem to be going anywhere so mothballed for

			now.

#!/bin/sh

python init/versionCheck.py

python init/prereq.py

if [ $? = 0 ]

then

Scripts involved in validating user input, system state, and initializing arm.

"""

__all__ = ["starter", "versionCheck"]

__all__ = ["starter", "prereq"]