Merge remote-tracking branch 'tor-github/pr/1247'

doc/HACKING/CodeStructure.md

@@ -2,128 +2,121 @@
 TODO: revise this to talk about how things are, rather than how things
 have changed.
 
-TODO: Make this into good markdown.
-
-
-
-For quite a while now, the program "tor" has been built from source
-code in just two directories: src/common and src/or.
+For quite a while now, the program *tor* has been built from source
+code in just two directories: **src/common** and **src/or**.
 
 This has become more-or-less untenable, for a few reasons -- most
 notably of which is that it has led our code to become more
 spaghetti-ish than I can endorse with a clean conscience.
 
 So to fix that, we've gone and done a huge code movement in our git
-master branch, which will land in a release once Tor 0.3.5.1-alpha is
+master branch, which will land in a release once Tor `0.3.5.1-alpha` is
 out.
 
 Here's what we did:
 
-  * src/common has been turned into a set of static libraries.  These
-all live in the "src/lib/*" directories.  The dependencies between
+  * **src/common** has been turned into a set of static libraries.  These
+all live in the **src/lib/*** directories.  The dependencies between
 these libraries should have no cycles.  The libraries are:
 
-    arch -- Headers to handle architectural differences
-    cc -- headers to handle differences among compilers
-    compress -- wraps zlib, zstd, lzma
-    container -- high-level container types
-    crypt_ops -- Cryptographic operations. Planning to split this into
+    - **arch** -- Headers to handle architectural differences
+    - **cc** -- headers to handle differences among compilers
+    - **compress** -- wraps zlib, zstd, lzma
+    - **container** -- high-level container types
+    - **crypt_ops** -- Cryptographic operations. Planning to split this into
 a higher and lower level library
-    ctime -- Operations that need to run in constant-time. (Properly,
+    - **ctime** -- Operations that need to run in constant-time. (Properly,
 data-invariant time)
-    defs -- miscelaneous definitions needed throughout Tor.
-    encoding -- transforming one data type into another, and various
+    - **defs** -- miscelaneous definitions needed throughout Tor.
+    - **encoding** -- transforming one data type into another, and various
 data types into strings.
-    err -- lowest-level error handling, in cases where we can't use
+    - **err** -- lowest-level error handling, in cases where we can't use
 the logs because something that the logging system needs has broken.
-    evloop -- Generic event-loop handling logic
-    fdio -- Low-level IO wrapper functions for file descriptors.
-    fs -- Operations on the filesystem
-    intmath -- low-level integer math and misc bit-twiddling hacks
-    lock -- low-level locking code
-    log -- Tor's logging module.  This library sits roughly halfway up
+    - **evloop** -- Generic event-loop handling logic
+    - **fdio** -- Low-level IO wrapper functions for file descriptors.
+    - **fs** -- Operations on the filesystem
+    - **intmath** -- low-level integer math and misc bit-twiddling hacks
+    - **lock** -- low-level locking code
+    - **log** -- Tor's logging module.  This library sits roughly halfway up
 the library dependency diagram, since everything it depends on has to
 be carefully crafted to *not* log.
-    malloc -- Low-level wrappers for the platform memory allocation functions.
-    math -- Higher-level mathematical functions, and floating-point math
-    memarea -- An arena allocator
-    meminfo -- Functions for querying the current process's memory
+    - **malloc** -- Low-level wrappers for the platform memory allocation functions.
+    - **math** -- Higher-level mathematical functions, and floating-point math
+    - **memarea** -- An arena allocator
+    - **meminfo** -- Functions for querying the current process's memory
 status and resources
-    net -- Networking compatibility and convenience code
-    osinfo -- Querying information about the operating system
-    process -- Launching and querying the status of other processes
-    sandbox -- Backend for the linux seccomp2 sandbox
-    smartlist_core -- The lowest-level of the smartlist_t data type.
+    - **net** -- Networking compatibility and convenience code
+    - **osinfo** -- Querying information about the operating system
+    - **process** -- Launching and querying the status of other processes
+    - **sandbox** -- Backend for the linux seccomp2 sandbox
+    - **smartlist_core** -- The lowest-level of the smartlist_t data type.
 Separated from the rest of the containers library because the logging
 subsystem depends on it.
-    string -- Compatibility and convenience functions for manipulating
+    - **string** -- Compatibility and convenience functions for manipulating
 C strings.
-    term -- Terminal-related functions (currently limited to a getpass
+    - **term** -- Terminal-related functions (currently limited to a getpass
 function).
-    testsupport -- Macros for mocking, unit tests, etc.
-    thread -- Higher-level thread compatibility code
-    time -- Higher-level time management code, including format
+    - **testsupport** -- Macros for mocking, unit tests, etc.
+    - **thread** -- Higher-level thread compatibility code
+    - **time** -- Higher-level time management code, including format
 conversions and monotonic time
-    tls -- Our wrapper around our TLS library
-    trace -- Formerly src/trace -- a generic event tracing API
-    wallclock -- Low-level time code, used by the log module.
+    - **tls** -- Our wrapper around our TLS library
+    - **trace** -- Formerly src/trace -- a generic event tracing API
+    - **wallclock** -- Low-level time code, used by the log module.
 
-  * To ensure that the dependency graph in src/common remains under
-control, there is a tool that you can run called "make
-check-includes".  It verifies that each module in Tor only includes
+  * To ensure that the dependency graph in **src/common** remains under
+control, there is a tool that you can run called `make
+check-includes`.  It verifies that each module in Tor only includes
 the headers that it is permitted to include, using a per-directory
-".may_include" file.
+*.may_include* file.
 
-  * The src/or/or.h header has been split into numerous smaller
+  * The **src/or/or.h** header has been split into numerous smaller
 headers.  Notably, many important structures are now declared in a
-header called foo_st.h, where "foo" is the name of the structure.
+header called *foo_st.h*, where "foo" is the name of the structure.
 
-  * The src/or directory, which had most of Tor's code, had been split
+  * The **src/or** directory, which had most of Tor's code, had been split
 up into several directories.  This is still a work in progress:  This
 code has not itself been refactored, and its dependency graph is still
 a tangled web.  I hope we'll be working on that over the coming
 releases, but it will take a while to do.
 
-    The new top-level source directories are:
-
-     src/core -- Code necessary to actually perform or use onion routing.
-     src/feature -- Code used only by some onion routing
+    - The new top-level source directories are:
+        - **src/core** -- Code necessary to actually perform or use onion routing.
+        - **src/feature** -- Code used only by some onion routing
 configurations, or only for a special purpose.
-     src/app -- Top-level code to run, invoke, and configure the
+        - **src/app** -- Top-level code to run, invoke, and configure the
 lower-level code
 
-   The new second-level source directories are:
-     src/core/crypto -- High-level cryptographic protocols used in Tor
-     src/core/mainloop -- Tor's event loop, connection-handling, and
+    - The new second-level source directories are:
+        - **src/core/crypto** -- High-level cryptographic protocols used in Tor
+        - **src/core/mainloop** -- Tor's event loop, connection-handling, and
 traffic-routing code.
-     src/core/or -- Parts related to handling onion routing itself
-     src/core/proto -- support for encoding and decoding different
+        - **src/core/or** -- Parts related to handling onion routing itself
+        - **src/core/proto** -- support for encoding and decoding different
 wire protocols
-
-     src/feature/api -- Support for making Tor embeddable
-     src/feature/client -- Functionality which only Tor clients need
-     src/feature/control -- Controller implementation
-     src/feature/dirauth -- Directory authority
-     src/feature/dircache -- Directory cache
-     src/feature/dirclient -- Directory client
-     src/feature/dircommon -- Shared code between the other directory modules
-     src/feature/hibernate -- Hibernating when Tor is out of bandwidth
+        - **src/feature/api** -- Support for making Tor embeddable
+        - **src/feature/client** -- Functionality which only Tor clients need
+        - **src/feature/control** -- Controller implementation
+        - **src/feature/dirauth** -- Directory authority
+        - **src/feature/dircache** -- Directory cache
+        - **src/feature/dirclient** -- Directory client
+        - **src/feature/dircommon** -- Shared code between the other directory modules
+        - **src/feature/hibernate** -- Hibernating when Tor is out of bandwidth
 or shutting down
-     src/feature/hs -- v3 onion service implementation
-     src/feature/hs_common -- shared code between both onion service
+        - **src/feature/hs** -- v3 onion service implementation
+        - **src/feature/hs_common** -- shared code between both onion service
 implementations
-     src/feature/nodelist -- storing and accessing the list of relays on
+        - **src/feature/nodelist** -- storing and accessing the list of relays on
 the network.
-     src/feature/relay -- code that only relay servers and exit servers need.
-     src/feature/rend -- v2 onion service implementation
-     src/feature/stats -- statistics and history
-
-     src/app/config -- configuration and state for Tor
-     src/app/main -- Top-level functions to invoke the rest or Tor.
+        - **src/feature/relay** -- code that only relay servers and exit servers need.
+        - **src/feature/rend** -- v2 onion service implementation
+        - **src/feature/stats** -- statistics and history
+        - **src/app/config** -- configuration and state for Tor
+        - **src/app/main** -- Top-level functions to invoke the rest or Tor.
 
-  * The "tor" executable is now built in src/app/tor rather than src/or/tor.
+  * The `tor` executable is now built in **src/app/tor** rather than **src/or/tor**.
 
   * There are more static libraries than before that you need to build
 into your application if you want to embed Tor.  Rather than
-maintaining this list yourself, I recommend that you run "make
-show-libs" to have Tor emit a list of what you need to link.
+maintaining this list yourself, I recommend that you run `make
+show-libs` to have Tor emit a list of what you need to link.

doc/HACKING/EndOfLifeTor.md

@@ -7,7 +7,7 @@ series reaches End of Life.  Note that they are _only_ for entire series that
 have reached their planned EOL: they do not apply to security-related
 deprecations of individual versions.
 
-=== 0. Preliminaries
+### 0. Preliminaries
 
 0. A few months before End of Life:
    Write a deprecation announcement.
@@ -17,7 +17,7 @@ deprecations of individual versions.
    Send the announcement to tor-announce, tor-talk, tor-relays, and the
    packagers.
 
-=== 1. On the day
+### 1. On the day
 
 1. Open tickets to remove the release from:
    - the jenkins builds

doc/HACKING/Fuzzing.md

-= Fuzzing Tor
+# Fuzzing Tor
 
-== The simple version (no fuzzing, only tests)
+## The simple version (no fuzzing, only tests)
 
 Check out fuzzing-corpora, and set TOR_FUZZ_CORPORA to point to the place
 where you checked it out.
@@ -12,7 +12,7 @@ This won't actually fuzz Tor!  It will just run all the fuzz binaries
 on our existing set of testcases for the fuzzer.
 
 
-== Different kinds of fuzzing
+## Different kinds of fuzzing
 
 Right now we support three different kinds of fuzzer.
 
@@ -37,7 +37,7 @@ In all cases, you'll need some starting examples to give the fuzzer when it
 starts out.  There's a set in the "fuzzing-corpora" git repository.  Try
 setting TOR_FUZZ_CORPORA to point to a checkout of that repository
 
-== Writing Tor fuzzers
+## Writing Tor fuzzers
 
 A tor fuzzing harness should have:
 * a fuzz_init() function to set up any necessary global state.
@@ -52,7 +52,7 @@ bug, or accesses memory it shouldn't. This helps fuzzing frameworks detect
 "interesting" cases.
 
 
-== Guided Fuzzing with AFL
+## Guided Fuzzing with AFL
 
 There is no HTTPS, hash, or signature for American Fuzzy Lop's source code, so
 its integrity can't be verified. That said, you really shouldn't fuzz on a
@@ -101,7 +101,7 @@ macOS (OS X) requires slightly more preparation, including:
 * using afl-clang (or afl-clang-fast from the llvm directory)
 * disabling external crash reporting (AFL will guide you through this step)
 
-== Triaging Issues
+## Triaging Issues
 
 Crashes are usually interesting, particularly if using AFL_HARDEN=1 and --enable-expensive-hardening. Sometimes crashes are due to bugs in the harness code.
 
@@ -115,7 +115,7 @@ To see what fuzz-http is doing with a test case, call it like this:
 
 (Logging is disabled while fuzzing to increase fuzzing speed.)
 
-== Reporting Issues
+## Reporting Issues
 
 Please report any issues discovered using the process in Tor's security issue
 policy:

doc/HACKING/ReleasingTor.md

@@ -5,7 +5,7 @@ Putting out a new release
 Here are the steps that the maintainer should take when putting out a
 new Tor release:
 
-=== 0. Preliminaries
+### 0. Preliminaries
 
 1. Get at least two of weasel/arma/Sebastian to put the new
    version number in their approved versions list.  Give them a few
@@ -18,7 +18,7 @@ new Tor release:
    date of a TB that contains it.  See note below in "commit, upload,
    announce".
 
-=== I. Make sure it works
+### I. Make sure it works
 
 1. Make sure that CI passes: have a look at Travis
    (https://travis-ci.org/torproject/tor/branches), Appveyor
@@ -52,7 +52,7 @@ new Tor release:
       memory leaks.)
 
 
-=== II. Write a changelog
+### II. Write a changelog
 
 
 1a. (Alpha release variant)
@@ -139,7 +139,7 @@ new Tor release:
    text of existing entries, though.)
 
 
-=== III. Making the source release.
+### III. Making the source release.
 
 1. In `maint-0.?.x`, bump the version number in `configure.ac` and run
    `make update-versions` to update version numbers in other
@@ -165,7 +165,7 @@ new Tor release:
    If it is not, you'll need to poke Roger, Weasel, and Sebastian again: see
    item 0.1 at the start of this document.
 
-=== IV. Commit, upload, announce
+### IV. Commit, upload, announce
 
 1. Sign the tarball, then sign and push the git tag:
 
@@ -241,7 +241,7 @@ new Tor release:
    For templates to use when announcing, see:
        https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/AnnouncementTemplates
 
-=== V. Aftermath and cleanup
+### V. Aftermath and cleanup
 
 1. If it's a stable release, bump the version number in the
     `maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours`