added some markdown formatting

Coding conventions for Tor

--------------------------

==========================

tl;dr:

    * Run configure with '--enable-gcc-warnings'

    * Run 'make check-spaces' to catch whitespace errors

    * Document your functions

    * Write unit tests

    * Add a file in 'changes' for your branch.

   - Run configure with `--enable-gcc-warnings`

   - Run `make check-spaces` to catch whitespace errors

   - Document your functions

   - Write unit tests

   - Add a file in `changes` for your branch.

Patch checklist

~~~~~~~~~~~~~~~

---------------

If possible, send your patch as one of these (in descending order of

preference)

Did you remember...

   - To build your code while configured with --enable-gcc-warnings?

   - To run "make check-spaces" on your code?

   - To run "make check-docs" to see whether all new options are on

   - To build your code while configured with `--enable-gcc-warnings`?

   - To run `make check-spaces` on your code?

   - To run `make check-docs` to see whether all new options are on

     the manpage?

   - To write unit tests, as possible?

   - To base your code on the appropriate branch?

   - To include a file in the "changes" directory as appropriate?

   - To include a file in the `changes` directory as appropriate?

How we use Git branches

-----------------------

=======================

Each main development series (like 0.2.1, 0.2.2, etc) has its main work

applied to a single branch.  At most one series can be the development series

How we log changes

------------------

==================

When you do a commit that needs a ChangeLog entry, add a new file to

the "changes" toplevel subdirectory.  It should have the format of a

the `changes` toplevel subdirectory.  It should have the format of a

one-entry changelog section from the current ChangeLog file, as in

  o Major bugfixes:

- Major bugfixes:

    - Fix a potential buffer overflow. Fixes bug 99999; bugfix on

      0.3.1.4-beta.

simplifications and refactoring.  Then say what the change does.  If

it's a bugfix, mention what bug it fixes and when the bug was

introduced.  To find out which Git tag the change was introduced in,

you can use "git describe --contains <sha1 of commit>".

you can use `git describe --contains <sha1 of commit>`.

If at all possible, try to create this file in the same commit where you are

making the change.  Please give it a distinctive name that no other branch will

use for the lifetime of your change. To verify the format of the changes file,

you can use "make check-changes".

you can use `make check-changes`.

When we go to make a release, we will concatenate all the entries

in changes to make a draft changelog, and clear the directory. We'll

then edit the draft changelog into a nice readable format.

What needs a changes file?::

   A not-exhaustive list: Anything that might change user-visible

What needs a changes file?

   * A not-exhaustive list: Anything that might change user-visible

   behavior. Anything that changes internals, documentation, or the build

   system enough that somebody could notice.  Big or interesting code

   rewrites.  Anything about which somebody might plausibly wonder "when

   did that happen, and/or why did we do that" 6 months down the line.

Why use changes files instead of Git commit messages?::

   Git commit messages are written for developers, not users, and they

Why use changes files instead of Git commit messages?

   * Git commit messages are written for developers, not users, and they

   are nigh-impossible to revise after the fact.

Why use changes files instead of entries in the ChangeLog?::

   Having every single commit touch the ChangeLog file tended to create

Why use changes files instead of entries in the ChangeLog?

   * Having every single commit touch the ChangeLog file tended to create

   zillions of merge conflicts.

Whitespace and C conformance

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

----------------------------

Invoke "make check-spaces" from time to time, so it can tell you about

Invoke `make check-spaces` from time to time, so it can tell you about

deviations from our C whitespace style.  Generally, we use:

   - Unix-style line endings

   - No more than 79-columns per line.

   - Two spaces per indent.

   - A space between control keywords and their corresponding paren

      "if (x)", "while (x)", and "switch (x)", never "if(x)", "while(x)", or

      "switch(x)".

     `if (x)`, `while (x)`, and `switch (x)`, never `if(x)`, `while(x)`, or

     `switch(x)`.

   - A space between anything and an open brace.

    - No space between a function name and an opening paren. "puts(x)", not

      "puts (x)".

   - No space between a function name and an opening paren. `puts(x)`, not

     `puts (x)`.

   - Function declarations at the start of the line.

We try hard to build without warnings everywhere.  In particular, if you're

using gcc, you should invoke the configure script with the option

"--enable-gcc-warnings".  This will give a bunch of extra warning flags to

`--enable-gcc-warnings`.  This will give a bunch of extra warning flags to

the compiler, and help us find divergences from our preferred C style.

Functions to use; functions not to use

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--------------------------------------

We have some wrapper functions like tor_malloc, tor_free, tor_strdup, and

tor_gettimeofday; use them instead of their generic equivalents.  (They

We have some wrapper functions like `tor_malloc`, `tor_free`, `tor_strdup`, and

`tor_gettimeofday;` use them instead of their generic equivalents.  (They

always succeed or exit.)

You can get a full list of the compatibility functions that Tor provides by

looking through src/common/util*.h and src/common/compat*.h.  You can see the

available containers in src/common/containers*.h.  You should probably

looking through `src/common/util*.h` and `src/common/compat*.h`.  You can see the

available containers in `src/common/containers*.h`.  You should probably

familiarize yourself with these modules before you write too much code, or

else you'll wind up reinventing the wheel.

Use 'INLINE' instead of 'inline' -- it's a vestige of an old hack to make

Use `INLINE` instead of `inline` -- it's a vestige of an old hack to make

sure that we worked on MSVC6.

We don't use strcat or strcpy or sprintf of any of those notoriously broken

old C functions.  Use strlcat, strlcpy, or tor_snprintf/tor_asprintf instead.

We don't use `strcat` or `strcpy` or `sprintf` of any of those notoriously broken

old C functions.  Use `strlcat`, `strlcpy`, or `tor_snprintf/tor_asprintf` instead.

We don't call memcmp() directly.  Use fast_memeq(), fast_memneq(),

tor_memeq(), or tor_memneq() for most purposes.

We don't call `memcmp()` directly.  Use `fast_memeq()`, `fast_memneq()`,

`tor_memeq()`, or `tor_memneq()` for most purposes.

Functions not to write

~~~~~~~~~~~~~~~~~~~~~~

----------------------

Try to never hand-write new code to parse or generate binary

formats. Instead, use trunnel if at all possible.  See  

    https://gitweb.torproject.org/trunnel.git/tree

for more information about trunnel.

For information on adding new trunnel code to Tor, see src/trunnel/README

Calling and naming conventions

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

------------------------------

Whenever possible, functions should return -1 on error and 0 on success.

For multi-word identifiers, use lowercase words combined with

underscores. (e.g., "multi_word_identifier").  Use ALL_CAPS for macros and

underscores. (e.g., `multi_word_identifier`).  Use ALL_CAPS for macros and

constants.

Typenames should end with "_t".

Typenames should end with `_t`.

Function names should be prefixed with a module name or object name.  (In

general, code to manipulate an object should be a module with the same name

as the object, so it's hard to tell which convention is used.)

Functions that do things should have imperative-verb names

(e.g. buffer_clear, buffer_resize); functions that return booleans should

have predicate names (e.g. buffer_is_empty, buffer_needs_resizing).

(e.g. `buffer_clear`, `buffer_resize`); functions that return booleans should

have predicate names (e.g. `buffer_is_empty`, `buffer_needs_resizing`).

If you find that you have four or more possible return code values, it's

probably time to create an enum.  If you find that you are passing three or

takes a bitfield.

What To Optimize

~~~~~~~~~~~~~~~~

----------------

Don't optimize anything if it's not in the critical path.  Right now, the

critical path seems to be AES, logging, and the network itself.  Feel free to

do your own profiling to determine otherwise.

Log conventions

~~~~~~~~~~~~~~~

---------------

https://www.torproject.org/docs/faq#LogLevel

`https://www.torproject.org/docs/faq#LogLevel`

No error or warning messages should be expected during normal OR or OP

operation.

Doxygen comment conventions

^^^^^^^^^^^^^^^^^^^^^^^^^^^

---------------------------

Say what functions do as a series of one or more imperative sentences, as

though you were telling somebody how to be the function.  In other words, DO

Once you've reached this point, here's what you need to know.

  1) Get the source.

  1. Get the source.

     We keep our source under version control in Git.  To get the latest

     version, run  

         git clone https://git.torproject.org/git/tor

     This will give you a checkout of the master branch.  If you're

         git checkout maint-0.2.7

  2) Find your way around the source

  2. Find your way around the source

     Our overall code structure is explained in the "torguts" documents,

     currently at

        git clone https://git.torproject.org/user/nickm/torguts.git

     Find a part of the code that looks interesting to you, and start

     If you see something that doesn't make sense, we love to get

     questions!

  3) Find something cool to hack on.

  3. Find something cool to hack on.

     You may already have a good idea of what you'd like to work on, or

     you might be looking for a way to contribute.

     For your first patch, it is probably NOT a good idea to make

     something huge or invasive.  In particular, you should probably

     avoid:

       * Major changes spread across many parts of the codebase.

       * Major changes to programming practice or coding style.

       * Huge new features or protocol changes.

  4) Meet the developers!

  4. Meet the developers!

     We discuss stuff on the tor-dev mailing list and on the #tor-dev

     IRC channel on OFTC.  We're generally friendly and approachable,

     better.  The time might change in the future, but generally,

     there's no bad time to talk, and ask us about patch ideas.

  5) Do you need to write a design proposal?

  5. Do you need to write a design proposal?

     If your idea is very large, or it will require a change to Tor's

     protocols, there needs to be a written design proposal before it

     You might also like to look around the rest of that directory, to

     see more about open and past proposed changes to Tor's behavior.

  6) Writing your patch

  6. Writing your patch

     As you write your code, you'll probably want it to fit in with the

     standards of the rest of the Tor codebase so it will be easy for us

     components, remember to divide it into a series of Git commits.  A

     series of small changes is much easier to review than one big lump.

  7) Testing your patch

  7. Testing your patch

     We prefer that all new or modified code have unit tests for it to

     ensure that it runs correctly.  Also, all code should actually be

     in Tor.  If you'd like any help writing tests, just ask!  We're

     glad to help out.

  8) Submitting your patch

  8. Submitting your patch

     We review patches through tickets on our bugtracker at

     trac.torproject.org.  You can either upload your patches there, or

     you've done on trac, and then change the status of the ticket to

     needs_review.

  9) Review, Revision, and Merge

  9. Review, Revision, and Merge

     With any luck, somebody will review your patch soon!  If not, you

     can ask on the IRC channel; sometimes we get really busy and take

         so. Or if you disagree with any of the comments, you should

         say so!  And if you won't have time to make some of the

         changes, you should say that too, so that other developers

            will be able to pick up the unfinished portion

         will be able to pick up the unfinished portion.

    Congratulations!  You have now written your first patch, and gotten

    it integrated into mainline Tor.

Useful tools

------------

============

These aren't strictly necessary for hacking on Tor, but they can help track

down bugs.

Jenkins

~~~~~~~

-------

    https://jenkins.torproject.org

Dmalloc

~~~~~~~

-------

The dmalloc library will keep track of memory allocation, so you can find out

if we're leaking memory, doing any double-frees, or so on.

  dmalloc -l ~/dmalloc.log

    dmalloc -l -/dmalloc.log

    (run the commands it tells you)

    ./configure --with-dmalloc

Valgrind

~~~~~~~~

--------

    valgrind --leak-check=yes --error-limit=no --show-reachable=yes src/or/tor

(Note that if you get a zillion openssl warnings, you will also need to

pass --undef-value-errors=no to valgrind, or rebuild your openssl

with -DPURIFY.)

pass `--undef-value-errors=no` to valgrind, or rebuild your openssl

with `-DPURIFY`.)

Coverity

~~~~~~~~

--------

Nick regularly runs the coverity static analyzer on the Tor codebase.

The preprocessor define __COVERITY__ is used to work around instances

The preprocessor define `__COVERITY__` is used to work around instances

where coverity picks up behavior that we wish to permit.

clang Static Analyzer

~~~~~~~~~~~~~~~~~~~~~

---------------------

The clang static analyzer can be run on the Tor codebase using Xcode (WIP)

or a command-line build.

The preprocessor define __clang_analyzer__ is used to work around instances

The preprocessor define `__clang_analyzer__` is used to work around instances

where clang picks up behavior that we wish to permit.

clang Runtime Sanitizers

~~~~~~~~~~~~~~~~~~~~~~~~

------------------------

To build the Tor codebase with the clang Address and Undefined Behavior

sanitizers, see the file contrib/clang/sanitize_blacklist.txt.

sanitizers, see the file `contrib/clang/sanitize_blacklist.txt`.

Preprocessor workarounds for instances where clang picks up behavior that

we wish to permit are also documented in the blacklist file.

Running lcov for unit test coverage

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-----------------------------------

Lcov is a utility that generates pretty HTML reports of test code coverage.

To generate such a report:

-----

    ./configure --enable-coverage

    make

    make coverage-html

    $BROWSER ./coverage_html/index.html

-----

This will run the tor unit test suite `./src/test/test` and generate the HTML

coverage code report under the directory ./coverage_html/. To change the

coverage code report under the directory `./coverage_html/`. To change the

output directory, use `make coverage-html HTML_COVER_DIR=./funky_new_cov_dir`.

Coverage diffs using lcov are not currently implemented, but are being

investigated (as of July 2014).

Running the unit tests

~~~~~~~~~~~~~~~~~~~~~~

----------------------

To quickly run all the tests distributed with Tor:

-----

    make check

-----

To run the fast unit tests only:

-----

    make test

-----

To selectively run just some tests (the following can be combined

arbitrarily):

-----

    ./src/test/test <name_of_test> [<name of test 2>] ...

    ./src/test/test <prefix_of_name_of_test>.. [<prefix_of_name_of_test2>..] ...

    ./src/test/test :<name_of_excluded_test> [:<name_of_excluded_test2]...

-----

To run all tests, including those based on Stem or Chutney:

-----

    make test-full

-----

To run all tests, including those based on Stem or Chutney that require a

working connection to the internet:

-----

    make test-full-online

-----

Running gcov for unit test coverage

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-----------------------------------

-----

    ./configure --enable-coverage

    make

    make check

    # or--- make test-full ? make test-full-online?

    mkdir coverage-output

    ./scripts/test/coverage coverage-output

-----

(On OSX, you'll need to start with "--enable-coverage CC=clang".)

(On OSX, you'll need to start with `--enable-coverage CC=clang`.)

Then, look at the .gcov files in coverage-output.  '-' before a line means

Then, look at the .gcov files in `coverage-output`.  '-' before a line means

that the compiler generated no code for that line.  '######' means that the

line was never reached.  Lines with numbers were called that number of times.

If that doesn't work:

   * Try configuring Tor with --disable-gcc-hardening

   * You might need to run 'make clean' after you run './configure'.

   * Try configuring Tor with `--disable-gcc-hardening`

   * You might need to run `make clean` after you run `./configure`.

If you make changes to Tor and want to get another set of coverage results,

you can run "make reset-gcov" to clear the intermediary gcov output.

you can run `make reset-gcov` to clear the intermediary gcov output.

If you have two different "coverage-output" directories, and you want to see

If you have two different `coverage-output` directories, and you want to see

a meaningful diff between them, you can run:

-----

    ./scripts/test/cov-diff coverage-output1 coverage-output2 | less

-----

In this diff, any lines that were visited at least once will have coverage

"1".  This lets you inspect what you (probably) really want to know: which

untested lines were changed?  Are there any new untested lines?

Running integration tests

~~~~~~~~~~~~~~~~~~~~~~~~~

-------------------------

We have the beginnings of a set of scripts to run integration tests using

Chutney. To try them, set CHUTNEY_PATH to your chutney source directory, and

run "make test-network".

run `make test-network`.

We also have scripts to run integration tests using Stem.  To try them, set

STEM_SOURCE_DIR to your Stem source directory, and run "test-stem".

`STEM_SOURCE_DIR` to your Stem source directory, and run `test-stem`.

Profiling Tor with oprofile

~~~~~~~~~~~~~~~~~~~~~~~~~~~

---------------------------

The oprofile tool runs (on Linux only!) to tell you what functions Tor is

spending its CPU time in, so we can identify performance bottlenecks.

 - Build all the libraries you care about with debugging symbols

   (probably you only care about libssl, maybe zlib and Libevent).

 - Copy this tor to a new directory

 - Copy all the libraries it uses to that dir too (ldd ./tor will

 - Copy all the libraries it uses to that dir too (`ldd ./tor` will

   tell you)

 - Set LD_LIBRARY_PATH to include that dir.  ldd ./tor should now

 - Set LD_LIBRARY_PATH to include that dir.  `ldd ./tor` should now

   show you it's using the libs in that dir

 - Run that tor

 - Reset oprofiles counters/start it

   * "opcontrol --reset; opcontrol --start", if Nick remembers right.

   * `opcontrol --reset; opcontrol --start`, if Nick remembers right.

 - After a while, have it dump the stats on tor and all the libs

   in that dir you created.

   * "opcontrol --dump;"

   * "opreport -l that_dir/*"

   * `opcontrol --dump;`

   * `opreport -l that_dir/*`

 - Profit

Generating and analyzing a callgraph

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

------------------------------------

1. Run ./scripts/maint/generate_callgraph.sh .  This will generate a

1. Run `./scripts/maint/generate_callgraph.sh`.  This will generate a

   bunch of files in a new ./callgraph directory.

2. Run ./scripts/maint/analyze_callgraph.py callgraph/src/*/* .  This

2. Run `./scripts/maint/analyze_callgraph.py callgraph/src/*/*`.  This

   will do a lot of graph operations and then dump out a new

   "callgraph.pkl" file, containing data in Python's "pickle" format.

   `callgraph.pkl` file, containing data in Python's 'pickle' format.

3. Run ./scripts/maint/display_callgraph.py .  It will display:

3. Run `./scripts/maint/display_callgraph.py`.  It will display:

    - the number of functions reachable from each function.

    - all strongly-connnected components in the Tor callgraph

    - the largest bottlenecks in the largest SCC in the Tor callgraph.

through function pointers.

Getting emacs to edit Tor source properly

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-----------------------------------------

Nick likes to put the following snippet in his .emacs file:

-----

    (add-hook 'c-mode-hook

          (lambda ()

            (font-lock-mode 1)

                (set-variable 'c-basic-offset 8)

                (set-variable 'tab-width 8))

            ))))

-----

You'll note that it defaults to showing all trailing whitespace.  The "cond"

You'll note that it defaults to showing all trailing whitespace.  The `cond`

test detects whether the file is one of a few C free software projects that I

often edit, and sets up the indentation level and tab preferences to match

what they want.

If you use emacs for editing Tor and nothing else, you could always just say:

-----

    (add-hook 'c-mode-hook

        (lambda ()

            (font-lock-mode 1)

            (set-variable 'show-trailing-whitespace t)

            (set-variable 'indent-tabs-mode nil)

            (set-variable 'c-basic-offset 2)))

-----

There is probably a better way to do this.  No, we are probably not going

to clutter the files with emacs stuff.

Doxygen

~~~~~~~

-------

We use the 'doxygen' utility to generate documentation from our

source code. Here's how to use it:

  1. Begin every file that should be documented with

         /**

          * \file filename.c

          * \brief Short description of the file.

         * \endcode

         */

  3. Make sure to escape the characters "<", ">", "\", "%" and "#" as "\<",

     "\>", "\\", "\%", and "\#".

  3. Make sure to escape the characters `<`, `>`, `\`, `%` and `#` as `\<`,

     `\>`, `\\`, `\%` and `\#`.

  4. To document structure members, you can use two forms:

        $ doxygen -g

     To generate a file called 'Doxyfile'.  Edit that file and run

     'doxygen' to generate the API documentation.

     to generate a file called `Doxyfile`.  Edit that file and run

     `doxygen` to generate the API documentation.

  6. See the Doxygen manual for more information; this summary just

     scratches the surface.

This directory has helpful information about what you need to know to

hack on Tor!

First, read 'GettingStarted.md' to learn how to get a start in Tor

First, read `GettingStarted.md` to learn how to get a start in Tor

development.

If you've decided to write a patch, 'CodingStandards.txt' will give

If you've decided to write a patch, `CodingStandards.txt` will give

you a bunch of information about how we structure our code.

It's important to get code right!  Reading 'WritingTests.md' will

It's important to get code right!  Reading `WritingTests.md` will

tell you how to write and run tests in the Tor codebase.

There are a bunch of other programs we use to help maintain and

develop the codebase: 'HelpfulTools.md' can tell you how to use them

develop the codebase: `HelpfulTools.md` can tell you how to use them

with Tor.

If it's your job to put out Tor releases, see 'ReleasingTor.md' so

If it's your job to put out Tor releases, see `ReleasingTor.md` so

that you don't miss any steps!

-----------------------

For full information on how Tor is supposed to work, look at the files in

https://gitweb.torproject.org/torspec.git/tree

`https://gitweb.torproject.org/torspec.git/tree`.

For an explanation of how to change Tor's design to work differently, look at

https://gitweb.torproject.org/torspec.git/blob_plain/HEAD:/proposals/001-process.txt

`https://gitweb.torproject.org/torspec.git/blob_plain/HEAD:/proposals/001-process.txt`.

For the latest version of the code, get a copy of git, and

    git clone https://git.torproject.org/git/tor

We talk about Tor on the tor-talk mailing list.  Design proposals and

discussion belong on the tor-dev mailing list.  We hang around on

We talk about Tor on the `tor-talk` mailing list.  Design proposals and

discussion belong on the `tor-dev` mailing list.  We hang around on

irc.oftc.net, with general discussion happening on #tor and development

happening on #tor-dev.

happening on `#tor-dev`.

The other files in this "HACKING" directory may also be useful as you

The other files in this `HACKING` directory may also be useful as you

get started working with Tor.

Happy hacking!

-----------------------

XXXXX also describe

doc/HACKING/WritingTests.md