Arm release 1.4.1

CHANGE LOG

11/27/10 - version 1.4.0

1/7/11 - version 1.4.1

Platform specific enhancements including BSD compatibility and vastly improved performance on Linux.

    * added: querying the proc contents when able for tor resource and connection samplings to *greatly* reduce arm's resource usage (many thanks to psutil)

    * added: vastly improved BSD compatibility, thanks to patches by Fabian Keil

          o pid resolution via pgrep (all platforms) and sockstat (bsd only)

          o connection resolution via sockstat (all platforms) and procstat (bsd only)

          o autodetecting the path prefix for FreeBSD jails

    * added: displaying summaries of the options on the configuration panel (idea by Sebastian)

    * added: arm cpu usage to the header panel and logs (with an estimate for system call usages)

    * added: testing script for checking connection resolution performance, connection dumps, and the glyph demo

    * added: option to dump arm debug logs (better failsafe option)

    * change: incrementing the uptime field of the header panel each second

    * change: centralizing arm resources in ~/.arm (suggested by Sebastian and also thanks to feedback from rransom)

    * change: using exponential backoff of ps/proc resource resolutions when calls fail or tor isn't running

    * change: reordered resolvers by order of performance

    * change: when tor's man page is unavailable falling back to descriptions provided with arm (often the case with tbb)

    * change: dropping support for graphing of custom ps attributes (feature was never used, kinda pointless, and incompatible with the proc enhancement)

    * fix: providing proper cpu samplings rather than an average over the life of the process

    * fix: expanding relative paths for the authentication cookie (mostly a problem for tbb instances)

    * fix: crashing error when querying hidden service parameters (caught by StrangeCharm, fixed by katmagic and chiiph)

    * fix: initially built diff differed from rebuilt version since it was missing the pycompat (caught by weasel)

    * fix: startup script wasn't handling unusual paths, such as with spaces and dashes (caught by weasel)

    * fix: startup script was forking the final process instead of calling exec (caught by weasel)

    * fix: log entries weren't being bracketed by date dividers when no scroll bars were visible (caught thanks to twur)

    * fix: workaround for config-text providing Log entries regardless of if it matches the default (caught thanks to Trystero)

    * fix: config validation mistakenly thought that Tor stripped spaces from CSV getconf responses (caught by murble)

    * fix: presenting a text input field with python 2.5 would crash (caught by murble)

    * fix: reloading torrc contents when a sighup is issued (caught by StrangeCharm)

    * fix: modifying lsof resolution to work with additional platforms (patch by Fabian Keil)

    * fix: dropping the locale for internal connections (patch by Fabian Keil)

    * fix: labeling connections to our socks port as being client connections (caught by Fabian Keil)

    * fix: skipping internal -> external address translation when the external address is private (caught by Fabian Keil)

    * fix: suppressing superfluous lsof warnings (patch by Hans Schnehl)

    * fix: when the pid was unavailable some resolvers failed to work

    * fix: including udp connection results (needed since exits proxy dns traffic)

    * fix: crashing issue when nickname was undefined but displayed on the connection panel

    * fix: concurrency issue caused the first connection resolution to often not have the pid

    * fix: connection resolution wasn't finding results if tor was running under a different name

    * fix: brought all Linux connection resolvers into parity (established tcp connections only)

    * fix: commands with quoted pipes were being mis-parsed by the sysTools' call function

11/27/10 - version 1.4.0 (r23873)

Introducing a new page for managing tor's configuration, along with several other improvements.

    * added: editor for the tor configuration, providing:

    * fix: making the interface more resilient to being resized while popups are visible

    * fix: log panel wasn't respecting the prepopulate* log level config options

    * fix: off by one error when wrapping lines in the log panel

    * fix (11/30/10, r23882): install script was failing to make the temporary directory for the compressed man page

    * fix (11/30/10, r23882): a torrc validation log entries had a typo

10/6/10 - version 1.3.7 (r23439)

Numerous improvements, most notably being an expanded log panel, installer, and deb/rpm builds.

  ... or including 'ControlPort <PORT>' in your torrc

For full functionality this also needs:

Common *nix commands including: ps, pidof, tail, pwdx, host, ulimit, and a

  method of connection resolution (netstat, ss, or lsof)

To be ran with the same user as tor to avoid permission issues with connection

  resolution and reading the torrc

- To be ran with the same user as tor to avoid permission issues with

  connection resolution and reading the torrc.

- Common *nix commands including:

    * ps

    * a method of connection resolution (any of the following):

      * sockstat

      * netstat

      * ss

      * lsof

      * procstat

    * tail

    * pwdx

    * ulimit

    * pgrep or pidof

    * host (if dns lookups are enabled)

This is started via 'arm' (use the '--help' argument for usage).

as being in a chroot jail) then it's probably failing due to permission issues.

Arm still runs, just no connection listing or ps stats.

> The bandwidth graph showing up

Some terminals, most notably screen sessions on Gentoo, appear to have a bug

where highlighted spaces aren't rendered. A reported workaround is to set:

  TERM="rxvt-unicode"

-------------------------------------------------------------------------------

Layout:

    starter.py        - parses and validates commandline parameters

    prereq.py         - checks python version and for required packages

    version.py        - version and last modified information

    test.py           - method for starting tests and demos

    settings.cfg      - attributes loaded for parsing tor related data

    torConfigDesc.txt - fallback descriptions of Tor's configuration options

    uninstall         - removal script

    interface/

      hostnames.py   - service providing nonblocking reverse dns lookups

      log.py         - aggregator for application events

      panel.py       - wrapper for safely working with curses subwindows

      procTools.py   - queries process & system information from /proc contents

      sysTools.py    - helper for system calls, providing client side caching

      torConfig.py   - functions for working with the torrc and config options

      torTools.py    - TorCtl wrapper, providing caching and derived information

TODO

- Roadmap and completed work for next release (1.4.1)

- Roadmap and completed work for next release (1.4.2)

  [ ] refactor panels

      Currently the interface is a bit of a rat's nest (especially the

      controller). The goal is to use better modularization to both simplify

      progress - /init and /util are done and /interface is partly done. Known

      bugs are being fixed while refactoring.

      [ ] conn panel

      * conn panel

        - expand client connections and note location in circuit (entry-exit)

        - for clients give an option to list all connections, to tell which are

          going through tor and which might be leaking

        - provide bridge / client country / exiting port statistics

          Include bridge related data via GETINFO option (feature request

          by waltman and ioerror).

        - note the common port usage along with the exit statistics

        - show the port used in scrubbed exit connections

        - pick apart applications like iftop and pktstat to see how they get

          per-connection bandwidth usage. Forum thread discussing it:

          https://bbs.archlinux.org/viewtopic.php?pid=715906

      [ ] control port interpreter (interactive prompt)

          Panel and startup option (-t maybe?) for providing raw control port

          access along with usability improvements (piggybacking on the arm

          connection):

          - irc like help (ex "/help GETINFO" could provide a summary of

            getinfo commands, partly using the results from

            "GETINFO info/names")

          - tab completion and up/down for previous commands

          - warn and get confirmation if command would disrupt arm (for

            instance 'SETEVENTS')

          - 'safe' option that restricts to read-only access (start with this)

          - issue sighup reset

      [ ] low hanging fruit from the "client mode use cases" below

        - include an option to show both the internal and external ips for the

          local connection, ie:

          myInternal --> myExternal --> foreign

          idea and initial patch by Fabian Keil

      * expand dump (--debug) information

        - os and python versions

        - arm and tor configs (scrubbing private attributes)

      * classify config options as useful (defaultly shown), standard, and

        deprecated (configured to be hidden by default)

      * check tor source for deprecated options like 'group' (are they

        ignored? idea is thanks to NightMonkey)

  * release prep

    * pylint --indent-string="  " --disable=C,R interface/foo.py | less

    * double check __init__.py and README for changes

    * double check __init__.py and README for added or removed files

    * wait a week, then bump package versions

      * Debian

        Contact: weasel (Peter Palfrader)

        Initial Release: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=603056

        Update Instructions:

          * TBD

      * Gentoo

        Contact: NightMonkey (Jesse Adelman)

        Initial Release: https://bugs.gentoo.org/show_bug.cgi?id=341731

        Update Instructions:

          * go to https://bugs.gentoo.org

          * make a generic bug with "net-misc/arm-X.X.X version bump, please"

      * ArchLinux

        Contact: Spider.007

        Initial Release: http://aur.archlinux.org/packages.php?ID=44172

        Update Instructions:

          * go to aur.archlinux.org

          * select "Out-of-date" for the package

- Roadmap for version 1.4.2

- Roadmap for version 1.4.3

  [ ] refactor panels

      [ ] controller and popup panels

      [ ] attempt to clear controller password from memory

      [ ] controller and popup panels

        - allow arm to resume after restarting tor

            This requires a full move to the torTools controller.

        - provide measurements for startup time, and try to improve bottlenecks

  [ ] menus

      - http://gnosis.cx/publish/programming/charming_python_6.html ?

      - additional options:

        - make update rates configurable via the ui

        - dialog with flag descriptions and other help

        - menu with all torrc options (making them editable/toggleable)

        - improve on performance bottlenecks for startup time and cpu usage

        - intermittent concurrency bugs during shutdown, one possible source:

          https://trac.torproject.org/projects/tor/ticket/2144

  [ ] setup scripts for arm

      [ ] updater (checks for a new tarball and installs it automatically)

        - attempt to verify download signature, providing a warning if unable

            - http://www.linuxjournal.com/article/5737

- Bugs

  * The default resolver isn't configurable.

  * The config write dialog (ie, the one for saving the config) has its a

    misaligned border when it's smaller than the top detail section.

  * The arm header panel doesn't properly reflect when the ip address

    changes. This provides a notice event saying:

    "Our IP Address has changed from X to Y; rebuilding descriptor (source Z)."

  * The cpu usage spikes for scrollable content when the key's held. Try

    coalescing the events.

  * The manpage layout is system dependent, so the scraper needs to be more

    resilient against being confused by whitespace. Another improvement is

    including fallback results if the man page can't be parsed (suggested by

    rransom, issue caught by NightMonkey).

  * Log deduplication is currently an n^2 operation. Hence it can't handle

    large logs (for instance, when at the DEBUG runlevel). Currently we're

    timing out the function if it takes too long, but a more efficient method

    for deduplication would be preferable.

  * when in client mode and tor stops the header panel doesn't say so

  * util are assuming that tor is running under the default command name

      attempt to determine the command name at runtime (if the pid is available

      then ps can do the mapping)

  * util/torTools.py: effective bandwidth rate/burst measurements don't take

      SETCONF into consideration, blocked on:

      https://trac.torproject.org/projects/tor/ticket/1692

    * connection uptimes shouldn't show fractions of a second

    * connections aren't cleared when control port closes

- Packaging

  * OpenWrt - OpenWrt uses the opkg packaging format which could make use of

    arm's current deb packages. Packaging for this platform would help with

    the Torouter project:

    https://trac.torproject.org/projects/tor/wiki/TheOnionRouter/Torouter

  * Mac - Couple of options include macport and dmg...

    * macport (http://guide.macports.org/#development)

      Build-from-source distribution method (like BSD portinstall). This has

      been suggested by several people.

    * dmg (http://en.wikipedia.org/wiki/Apple_Disk_Image)

      Most conventional method of software distribution on mac. This is just

      a container (no updating/removal support), but could contain an icon

      for the dock that starts a terminal with arm. This might include a pkg

      installer.

    * mpkg (http://pypi.python.org/pypi/bdist_mpkg/)

      Plugin for distutils. Like most mac packaging, this can only run on a

      mac. It also requires setuptools:

      http://www.errorhelp.com/search/details/74034/importerror-no-module-named-setuptools

- Future Features

  * client mode use cases

    * not sure what sort of information would be useful in the header (to

    * dialog with bridge statuses (idea by mikeperry)

      https://trac.vidalia-project.net/ticket/570

      https://trac.torproject.org/projects/tor/ticket/2068

  * menus

    * http://gnosis.cx/publish/programming/charming_python_6.html ?

    * additional options:

      * make update rates configurable via the ui

      * dialog with flag descriptions and other help

      * menu with all torrc options (making them editable/toggleable)

  * control port interpreter (interactive prompt)

    Panel and startup option (-t maybe?) for providing raw control port

    access along with usability improvements (piggybacking on the arm

    connection):

    * irc like help (ex "/help GETINFO" could provide a summary of

      getinfo commands, partly using the results from

      "GETINFO info/names")

    * tab completion and up/down for previous commands

    * warn and get confirmation if command would disrupt arm (for

      instance 'SETEVENTS')

    * 'safe' option that restricts to read-only access (start with this)

    * issue sighup reset

  * make use of the new process/* GETINFO options

    They'll be available in the next tor release, as per:

    https://trac.torproject.org/projects/tor/ticket/2291

  * feature parity for arm's config values (armrc entries)

    * editability

    * parse descriptions from the man page? autogeneration of the man page from

    * accounting and alerts for if the bandwidth drops to zero

    * daily/weekly/etc alerts for basic status (log output, bandwidth history,

        etc), borrowing from the consensus tracker for some of the formatting

  * mac installer

    * Couple of options include macport and dmg...

      * macport (http://guide.macports.org/#development)

        Build-from-source distribution method (like BSD portinstall). This has

        been suggested by several people.

      * dmg (http://en.wikipedia.org/wiki/Apple_Disk_Image)

        Most conventional method of software distribution on mac. This is just

        a container (no updating/removal support), but could contain an icon

        for the dock that starts a terminal with arm. This might include a pkg

        installer.

      * mpkg (http://pypi.python.org/pypi/bdist_mpkg/)

        Plugin for distutils. Like most mac packaging, this can only run on a

        mac. It also requires setuptools:

        http://www.errorhelp.com/search/details/74034/importerror-no-module-named-setuptools

  * tab completion for input fields that expect a filesystem path

  * look through vidalia's tickets for more ideas

    https://trac.vidalia-project.net/

  * setup wizard for new relays

    Setting the password and such for torrc generation. Maybe a netinstaller

    that fetches the right package for the plagform, verifies signatures, etc?

    (idea by ioerror)

    Another alternative would be that when arm is started and tor isn't

    running offer to start tor as a client, relay, or bridge. (idea by ioerror)

  * audit what tor does

    * Provide warnings if tor connections misbehaves, for instance:

      * ensuring ExitPolicyRejectPrivate is being obeyed

\fB\-c\fR, \fB\-\-config CONFIG_PATH\fR

user provided configuration file (default is \fB~/.armrc\fR)

.TP

\fB\-d\fR, \fB\-\-debug\fR

writes all arm logs to /home/atagar/.arm/log

.TP

\fB\-b\fR, \fB\-\-blind\fR

disable connection lookups (netstat, lsof, and ss), dropping the parts of the

# startup options

# Startup options

startup.controlPassword

startup.interface.ipAddress 127.0.0.1

startup.interface.port 9051

startup.blindModeEnabled false

startup.events N3

startup.dataDirectory ~/.arm

# Seconds between querying information

queries.ps.rate 5

queries.resourceUsage.rate 5

queries.connections.minRate 5

queries.refreshRate.rate 5

# Read the proc contents directly instead of calling ps, netstat, and other

# resolvers. This provides very sizable performance benefits (around 90%

# faster lookups) but this is only available on Linux.

queries.useProc true

# Renders the interface with color if set and the terminal supports it

features.colorInterface true

features.graph.bw.accounting.rate 10

features.graph.bw.accounting.isTimeLong false

# Parameters for graphing ps stats

# --------------------------------

# primary/secondaryStat

#   any numeric field provided by the ps command

# cachedOnly

#   determines if the graph should query ps or rely on cached results (this

#   lowers the call volume but limits the graph's granularity)

features.graph.ps.primaryStat %cpu

features.graph.ps.secondaryStat rss

features.graph.ps.cachedOnly true

# Thread pool size for hostname resolutions

# Determines the maximum number of concurrent requests. Upping this to around

# thirty or so seems to be problematic, causing intermittently seizing.

# Runlevels at which arm logs its events

log.startTime INFO

log.refreshRate DEBUG

log.highCpuUsage WARN

log.configEntryNotFound NONE

log.configEntryUndefined NOTICE

log.configEntryTypeError NOTICE

log.torCtlPortClosed NOTICE

log.torGetInfo DEBUG

log.torGetInfoCache NONE

log.torGetConf DEBUG

log.torSetConf INFO

log.torEventTypeUnrecognized NOTICE

log.torPrefixPathInvalid NOTICE

log.procCallMade DEBUG

log.sysCallMade DEBUG

log.sysCallCached NONE

log.sysCallFailed INFO

log.sysCallCacheGrowing INFO

log.panelRecreated DEBUG

log.graph.ps.invalidStat WARN

log.graph.ps.abandon WARN

log.graph.bw.prepopulateSuccess NOTICE

log.graph.bw.prepopulateFailure NOTICE

log.logPanel.prepopulateSuccess INFO

log.logPanel.forceDoubleRedraw DEBUG

log.torrc.readFailed WARN

log.torrc.validation.torStateDiffers WARN

log.torrc.validation.unnecessaryTorrcEntries WARN

log.torrc.validation.unnecessaryTorrcEntries NOTICE

log.configDescriptions.readManPageSuccess INFO

log.configDescriptions.readManPageFailed WARN

log.configDescriptions.readManPageFailed NOTICE

log.configDescriptions.unrecognizedCategory NOTICE

log.configDescriptions.internalLoadSuccess NOTICE

log.configDescriptions.internalLoadFailed ERR

log.configDescriptions.persistance.loadSuccess INFO

log.configDescriptions.persistance.loadFailed INFO

log.configDescriptions.persistance.saveSuccess NOTICE

log.configDescriptions.persistance.saveFailed NOTICE

log.connResolverOptions INFO

log.connLookupFailed INFO

log.connLookupFailover NOTICE

log.connLookupAbandon WARN

log.connLookupRateGrowing NONE

log.hostnameCacheTrimmed INFO

log.cursesColorSupport INFO

log.bsdJailFound INFO

log.unknownBsdJailId WARN

log.stats.failedProcResolution DEBUG

log.stats.procResolutionFailover INFO

log.stats.failedPsResolution INFO