Merge branch 'design_revision'

it fits together, what functionality is available for extending Tor,

and gives some notes on how Tor got that way.

Tor remains a work in progress: We've been working on it for more than a

decade, and we've learned a lot about good coding since we first

Tor remains a work in progress: We've been working on it for nearly two

decades, and we've learned a lot about good coding since we first

started.  This means, however, that some of the older pieces of Tor will

have some "code smell" in them that could sure stand a brisk

have some "code smell" in them that could stand a brisk

refactoring.  So when I describe a piece of code, I'll sometimes give a

note on how it got that way, and whether I still think that's a good

idea.

The first drafts of this document were written in the Summer and Fall of

2015, when Tor 0.2.6 was the most recent stable version, and Tor 0.2.7

was under development.  If you're reading this far in the future, some

things may have changed.  Caveat haxxor!

was under development.  There is a revision in progress (as of late

2019), to bring it up to pace with Tor as of version 0.4.2.  If you're

reading this far in the future, some things may have changed.  Caveat

haxxor!

This document is not an overview of the Tor protocol.  For that, see the

design paper and the specifications at https://spec.torproject.org/ .

For more information about Tor's coding standards and some helpful

development tools, see doc/HACKING in the Tor repository.

For more information about writing tests, see doc/HACKING/WritingTests.txt

in the Tor repository.

### The very high level ###

depending on its configuration.

Tor has a few major dependencies, including Libevent (used to tell which

sockets are readable and writable), OpenSSL (used for many encryption

sockets are readable and writable), OpenSSL or NSS (used for many encryption

functions, and to implement the TLS protocol), and zlib (used to

compress and uncompress directory information).

Most of Tor's work today is done in a single event-driven main thread.

Tor also spawns one or more worker threads to handle CPU-intensive

tasks.  (Right now, this only includes circuit encryption.)

tasks.  (Right now, this only includes circuit encryption and the more

expensive compression algorithms.)

On startup, Tor initializes its libraries, reads and responds to its

configuration files, and launches a main event loop.  At first, the only

events that Tor listens for are a few signals (like TERM and HUP), and

one or more listener sockets (for different kinds of incoming

connections).  Tor also configures a timer function to run once per

second to handle periodic events.  As Tor runs over time, other events

will open, and new events will be scheduled.

connections).  Tor also configures several timers to handle periodic

events.  As Tor runs over time, other events will open, and new events

will be scheduled.

The codebase is divided into a few main subdirectories:

The codebase is divided into a few top-level subdirectories, each of

which contains several sub-modules.

   src/common -- utility functions, not necessarily tor-specific.

   * `src/ext` -- Code maintained elsewhere that we include in the Tor

     source distribution.

   src/or -- implements the Tor protocols.

   * src/lib` -- Lower-level utility code, not necessarily tor-specific.

   src/test -- unit and regression tests

   * `src/trunnel` -- Automatically generated code (from the Trunnel

     tool): used to parse and encode binary formats.

   src/ext -- Code maintained elsewhere that we include in the Tor

   source distribution.

   * `src/core` -- Networking code that is implements the central parts of

   the Tor protocol and main loop.

   * `src/feature` -- Aspects of Tor (like directory management, running a

   relay, running a directory authorities, managing a list of nodes,

   running and using onion services) that are built on top of the

   mainloop code.

   * `src/app` -- Highest-level functionality; responsible for setting up

   and configuring the Tor daemon, making sure all the lower-level

   modules start up when required, and so on.

   * `src/tools` -- Binaries other than Tor that we produce.  Currently this

   is tor-resolve, tor-gencert, and the tor_runner.o helper module.

   src/trunnel -- automatically generated code (from the Trunnel)

   tool: used to parse and encode binary formats.

   * `src/test` -- unit tests, regression tests, and a few integration

   tests.

In theory, the above parts of the codebase are sorted from highest-level to

lowest-level, where high-level code is only allowed to invoke lower-level

code, and lower-level code never includes or depends on code of a higher

level.  In practice, this refactoring is incomplete: The modules in `src/lib`

are well-factored, but there are many layer violations ("upward

dependencies") in `src/core` and `src/feature`.  We aim to eliminate those

over time.

### Some key high-level abstractions ###

> **Note**: This section describes the eventual organization of this

> document, which is not yet complete.

We'll begin with an overview of the various utility functions available

in Tor's 'common' directory.  Knowing about these is key to writing

portable, simple code in Tor.

We'll begin with an overview of the facilities provided by the modules

in src/lib.  Knowing about these is key to writing portable, simple code

in Tor.

Then we'll move on to a discussion of how parts of the Tor codebase are

initialized, finalized, configured, and managed.

Then we'll go on and talk about the main data-flow of the Tor network:

how Tor generates and responds to network traffic.  This will occupy a

chapter for the main overview, with other chapters for special topics.

After that, we'll mention the main modules in Tor, and describe the

function of each.

We'll cover the directory subsystem next: how Tor learns about other

relays, and how relays advertise themselves.

Then we'll cover a few specialized modules, such as hidden services,

sandboxing, hibernation, accounting, statistics, guards, path

generation, pluggable transports, and how they integrate with the rest of Tor.

After that, we'll mention the main modules in src/features and describe the

functions of each.

We'll close with a meandering overview of important pending issues in

the Tor codebase, and how they affect the future of the Tor software.

## Utility code in Tor

Most of Tor's utility code is in modules in the src/common subdirectory.

These are divided, broadly, into _compatibility_ functions, _utility_

functions, _containers_, and _cryptography_.  (Someday in the future, it

would be great to split these modules into separate directories.  Also, some

functions are probably put in the wrong modules)

### Compatibility code

These functions live in src/common/compat\*.c; some corresponding macros live

in src/common/compat\*.h.  They serve as wrappers around platform-specific or

compiler-specific logic functionality.

In general, the rest of the Tor code *should not* be calling platform-specific

or otherwise non-portable functions.  Instead, they should call wrappers from

compat.c, which implement a common cross-platform API.  (If you don't know

whether a function is portable, it's usually good enough to see whether it

exists on OSX, Linux, and Windows.)

Other compatibility modules include backtrace.c, which generates stack traces

for crash reporting; sandbox.c, which implements the Linux seccomp2 sandbox;

and procmon.c, which handles monitoring a child process.

Parts of address.c are compatibility code for handling network addressing

issues; other parts are in util.c.

Notable compatibility areas are:

   * mmap support for mapping files into the address space (read-only)

   * Code to work around the intricacies

   * Workaround code for Windows's horrible winsock incompatibilities and

     Linux's intricate socket extensions.

   * Helpful string functions like memmem, memstr, asprintf, strlcpy, and

     strlcat that not all platforms have.

   * Locale-ignoring variants of the ctypes functions.

   * Time-manipulation functions

   * File locking function

   * IPv6 functions for platforms that don't have enough IPv6 support

   * Endianness functions

   * OS functions

   * Threading and locking functions.

=== Utility functions

General-purpose utilities are in util.c; they include higher-level wrappers

around many of the compatibility functions to provide things like

file-at-once access, memory management functions, math, string manipulation,

time manipulation, filesystem manipulation, etc.

(Some functionality, like daemon-launching, would be better off in a

compatibility module.)

In util_format.c, we have code to implement stuff like base-32 and base-64

encoding.

The address.c module interfaces with the system resolver and implements

address parsing and formatting functions.  It converts sockaddrs to and from

a more compact tor_addr_t type.

The di_ops.c module provides constant-time comparison and associative-array

operations, for side-channel avoidance.

The logging subsystem in log.c supports logging to files, to controllers, to

stdout/stderr, or to the system log.

The abstraction in memarea.c is used in cases when a large amount of

temporary objects need to be allocated, and they can all be freed at the same

time.

The torgzip.c module wraps the zlib library to implement compression.

Workqueue.c provides a simple multithreaded work-queue implementation.

### Containers

The container.c module defines these container types, used throughout the Tor

codebase.

There is a dynamic array called **smartlist**, used as our general resizeable

array type.  It supports sorting, searching, common set operations, and so

on.  It has specialized functions for smartlists of strings, and for

heap-based priority queues.

There's a bit-array type.

A set of mapping types to map strings, 160-bit digests, and 256-bit digests

to void \*.  These are what we generally use when we want O(1) lookup.

Additionally, for containers, we use the ht.h and tor_queue.h headers, in

src/ext.  These provide intrusive hashtable and linked-list macros.

###  Cryptography

Once, we tried to keep our cryptography code in a single "crypto.c" file,

with an "aes.c" module containing an AES implementation for use with older

OpenSSLs.

Now, our practice has become to introduce crypto_\*.c modules when adding new

cryptography backend code.  We have modules for Ed25519, Curve25519,

secret-to-key algorithms, and password-based boxed encryption.

Our various TLS compatibility code, wrappers, and hacks are kept in

tortls.c, which is probably too full of Tor-specific kludges.  I'm

hoping we can eliminate most of those kludges when we finally remove

support for older versions of our TLS handshake.

## Library code in Tor.

Most of Tor's utility code is in modules in the `src/lib` subdirectory.  In

general, this code is not necessarily Tor-specific, but is instead possibly

useful for other applications.

This code includes:

  * Compatibility wrappers, to provide a uniform API across different

    platforms.

  * Library wrappers, to provide a tor-like API over different libraries

    that Tor uses for things like compression and cryptography.

  * Containers, to implement some general-purpose data container types.

The modules in `src/lib` are currently well-factored: each one depends

only on lower-level modules.  You can see an up-to-date list of the

modules sorted from lowest to highest level by running

`./scripts/maint/practracker/includes.py --toposort`.

As of this writing, the library modules are (from lowest to highest

level):

   * `lib/cc` -- Macros for managing the C compiler and

     language. Includes macros for improving compatibility and clarity

     across different C compilers.

   * `lib/version` -- Holds the current version of Tor.

   * `lib/testsupport` -- Helpers for making test-only code and test

     mocking support.

   * `lib/defs` -- Lowest-level constants used in many places across the

     code.

   * `lib/subsys` -- Types used for declaring a "subsystem". A subsystem

     is a module with support for initialization, shutdown,

     configuration, and so on.

   * `lib/conf` -- Types and macros used for declaring configuration

     options.

   * `lib/arch` -- Compatibility functions and macros for handling

     differences in CPU architecture.

   * `lib/err` -- Lowest-level error handling code: responsible for

     generating stack traces, handling raw assertion failures, and

     otherwise reporting problems that might not be safe to report

     via the regular logging module.

   * `lib/malloc` -- Wrappers and utilities for memory management.

   * `lib/intmath` -- Utilities for integer mathematics.

   * `lib/fdio` -- Utilities and compatibility code for reading and

      writing data on file descriptors (and on sockets, for platforms

      where a socket is not a kind of fd).

   * `lib/lock` -- Compatibility code for declaring and using locks.

      Lower-level than the rest of the threading code.

   * `lib/ctime` -- Constant-time implementations for data comparison

     and table lookup, used to avoid timing side-channels from standard

     implementations of memcmp() and so on.

   * `lib/string` -- Low-level compatibility wrappers and utility

     functions for string manipulation.

   * `lib/wallclock` -- Compatibility and utility functions for

     inspecting and manipulating the current (UTC) time.

   * `lib/osinfo` -- Functions for inspecting the version and

     capabilities of the operating system.

   * `lib/smartlist_core` -- The bare-bones pieces of our dynamic array

     ("smartlist") implementation. There are higher-level pieces, but

     these ones are used by (and therefore cannot use) the logging code.

   * `lib/log` -- Implements the logging system used by all higher-level

     Tor code.  You can think of this as the logical "midpoint" of the

     library code: much of the higher-level code is higher-level

     _because_ it uses the logging module, and much of the lower-level

     code is specifically written to avoid having to log, because the

     logging module depends on it.

   * `lib/container` -- General purpose containers, including dynamic arrays

     ("smartlists"), hashtables, bit arrays, weak-reference-like "handles",

     bloom filters, and a bit more.

   * `lib/trace` -- A general-purpose API for introducing

     function-tracing functionality into Tor.  Currently not much used.

   * `lib/thread` -- Threading compatibility and utility functionality,

     other than low-level locks (which are in `lib/lock`) and

     workqueue/threadpool code (which belongs in `lib/evloop`).

   * `lib/term` -- Code for terminal manipulation functions (like

     reading a password from the user).

   * `lib/memarea` -- A data structure for a fast "arena" style allocator,

     where the data is freed all at once.  Used for parsing.

   * `lib/encoding` -- Implementations for encoding data in various

     formats, datatypes, and transformations.

   * `lib/dispatch` -- A general-purpose in-process message delivery

     system.  Used by `lib/pubsub` to implement our inter-module

     publish/subscribe system.

   * `lib/sandbox` -- Our Linux seccomp2 sandbox implementation.

   * `lib/pubsub` -- Code and macros to implement our publish/subscribe

     message passing system.

   * `lib/fs` -- Utility and compatibility code for manipulating files,

     filenames, directories, and so on.

   * `lib/confmgt` -- Code to parse, encode, and manipulate our

     configuration files, state files, and so forth.

   * `lib/crypt_ops` -- Cryptographic operations. This module contains

     wrappers around the cryptographic libraries that we support,

     and implementations for some higher-level cryptographic

     constructions that we use.

   * `lib/meminfo` -- Functions for inspecting our memory usage, if the

     malloc implementation exposes that to us.

   * `lib/time` -- Higher level time functions, including fine-gained and

      monotonic timers.

   * `lib/math` -- Floating-point mathematical utilities, including

     compatibility code, and probability distributions.

   * `lib/buf` -- A general purpose queued buffer implementation,

     similar to the BSD kernel's "mbuf" structure.

   * `lib/net` -- Networking code, including address manipulation,

     compatibility wrappers,

   * `lib/compress` -- A compatibility wrapper around several

     compression libraries, currently including zlib, zstd, and lzma.

   * `lib/geoip` -- Utilities to manage geoip (IP to country) lookups

      and formats.

   * `lib/tls` -- Compatibility wrappers around the library (NSS or

     OpenSSL, depending on configuration) that Tor uses to implement the

     TLS link security protocol.

   * `lib/evloop` -- Tools to manage the event loop and related

     functionality, in order to implement asynchronous networking,

     timers, periodic events, and other scheduling tasks.

   * `lib/process` -- Utilities and compatibility code to launch and

     manage subprocesses.

### What belongs in lib?

In general, if you can imagine some program wanting the functionality

you're writing, even if that program had nothing to do with Tor, your

functionality belongs in lib.

If it falls into one of the existing "lib" categories, your

functionality belongs in lib.

If you are using platform-specific `#ifdef`s to manage compatibility

issues among platforms, you should probably consider whether you can

put your code into lib.

## Memory management

### Heap-allocation functions

### Heap-allocation functions: lib/malloc/malloc.h

Tor imposes a few light wrappers over C's native malloc and free

functions, to improve convenience, and to allow wholescale replacement

They are the same as the standard C functions, with the following

exceptions:

   * tor_free(NULL) is a no-op.

   * tor_free() is a macro that takes an lvalue as an argument and sets it to

     NULL after freeing it.  To avoid this behavior, you can use tor_free_()

   * `tor_free(NULL)` is a no-op.

   * `tor_free()` is a macro that takes an lvalue as an argument and sets it to

     NULL after freeing it.  To avoid this behavior, you can use `tor_free_()`

     instead.

   * tor_malloc() and friends fail with an assertion if they are asked to

     allocate a value so large that it is probably an underflow.

   * It is always safe to tor_malloc(0), regardless of whether your libc

   * It is always safe to `tor_malloc(0)`, regardless of whether your libc

     allows it.

   * tor_malloc(), tor_realloc(), and friends are never allowed to fail.

   * `tor_malloc()`, `tor_realloc()`, and friends are never allowed to fail.

     Instead, Tor will die with an assertion.  This means that you never

     need to check their return values.  See the next subsection for

     information on why we think this is a good idea.

We define additional general-purpose memory allocation functions as well:

   * tor_malloc_zero(x) behaves as calloc(1, x), except the it makes clear

   * `tor_malloc_zero(x)` behaves as `calloc(1, x)`, except the it makes clear

     the intent to allocate a single zeroed-out value.

   * tor_reallocarray(x,y) behaves as the OpenBSD reallocarray function.

   * `tor_reallocarray(x,y)` behaves as the OpenBSD reallocarray function.

     Use it for cases when you need to realloc() in a multiplication-safe

     way.

And specific-purpose functions as well:

   * tor_strdup() and tor_strndup() behaves as the underlying libc functions,

     but use tor_malloc() instead of the underlying function.

   * tor_memdup() copies a chunk of memory of a given size.

   * tor_memdup_nulterm() copies a chunk of memory of a given size, then

   * `tor_strdup()` and `tor_strndup()` behaves as the underlying libc

     functions, but use `tor_malloc()` instead of the underlying function.

   * `tor_memdup()` copies a chunk of memory of a given size.

   * `tor_memdup_nulterm()` copies a chunk of memory of a given size, then

     NUL-terminates it just to be safe.

#### Why assert on failure?

#### Why assert on allocation failure?

Why don't we allow tor_malloc() and its allies to return NULL?

Why don't we allow `tor_malloc()` and its allies to return NULL?

First, it's error-prone.  Many programmers forget to check for NULL return

values, and testing for malloc() failures is a major pain.

values, and testing for `malloc()` failures is a major pain.

Second, it's not necessarily a great way to handle OOM conditions. It's

probably better (we think) to have a memory target where we dynamically free

things ahead of time in order to stay under the target.  Trying to respond to

an OOM at the point of tor_malloc() failure, on the other hand, would involve

an OOM at the point of `tor_malloc()` failure, on the other hand, would involve

a rare operation invoked from deep in the call stack.  (Again, that's

error-prone and hard to debug.)

Third, thanks to the rise of Linux and other operating systems that allow

memory to be overcommitted, you can't actually ever rely on getting a NULL

from malloc() when you're out of memory; instead you have to use an approach

from `malloc()` when you're out of memory; instead you have to use an approach

closer to tracking the total memory usage.

#### Conventions for your own allocation functions.

Whenever you create a new type, the convention is to give it a pair of

x_new() and x_free() functions, named after the type.

`x_new()` and `x_free_()` functions, named after the type.

Calling x_free(NULL) should always be a no-op.

Calling `x_free(NULL)` should always be a no-op.

There should additionally be an `x_free()` macro, defined in terms of

`x_free_()`.  This macro should set its lvalue to NULL.  You can define it

using the FREE_AND_NULL macro, as follows:

### Grow-only memory allocation: memarea.c

```

#define x_free(ptr) FREE_AND_NULL(x_t, x_free_, (ptr))

```

### Grow-only memory allocation: lib/memarea

It's often handy to allocate a large number of tiny objects, all of which

need to disappear at the same time.  You can do this in tor using the

and a memarea for the temporary object, the malloc implementation is likelier

to do better.

To create a new memarea, use memarea_new().  To drop all the storage from a

memarea, and invalidate its pointers, use memarea_drop_all().

To create a new memarea, use `memarea_new()`.  To drop all the storage from a

memarea, and invalidate its pointers, use `memarea_drop_all()`.

The allocation functions memarea_alloc(), memarea_alloc_zero(),

memarea_memdup(), memarea_strdup(), and memarea_strndup() are analogous to

the similarly-named malloc() functions.  There is intentionally no

memarea_free() or memarea_realloc().

The allocation functions `memarea_alloc()`, `memarea_alloc_zero()`,

`memarea_memdup()`, `memarea_strdup()`, and `memarea_strndup()` are analogous

to the similarly-named malloc() functions.  There is intentionally no

`memarea_free()` or `memarea_realloc()`.

### Special allocation: lib/malloc/map_anon.h

TODO: WRITEME.