SHORT SUMMARY OF MERGE AND BUILD PROCESS

0. Prerequisity done once: setup of buildenv (REQUIREMENTS... below)

1. Have a local clone of remote that is clean, one extra copy/clone of that for
   work
2. Merge onto the branches from tpo (see MERGING REPO... below)
3. Update tor.spec.in (changelog, dependencies, etc.)
3. Make tarball and SRPM in working repo to check basic sanity of build: 
    LIBS=-lrt ./autogen.sh && ./configure && make dist-rpm
4. Build all packages with mock with 'makerpm' user (TAILORED BUILD SCRIPTS
   below - build is performed on feddei)
5. Check out log of build if all was ok
   - build scripts will create a subdir in /home/makerpm/TorBuildWorkspace
6. Test one of packages with 'yum localinstall' to make sure basic install works
7. Sign packages
   - RpmSignTor.zsh is a quick signing script, but key IDs are hardcoded there
8. Prepare RPM/YUM metadata - see move_tor_rpms.sh below
9. Upload to a test HTTP server
10. Install and test using the above HTTP server to ensure
    metadata/checksums/signatures work (have a special /etc/yum.d/tor.repo file
    on testing VMs that pull from the test HTTP server)
11. Send packages to weasel

Note: if new rpm is needed before new Tor version comes out, increase
%specver in tor.spec.in. Reset it back to 1 when bumping to new upstream
version.


BUILDING THE RPMS

This readme contains:
 - how to setup the build environment
 - currently used repos (paths)
 - howto merge when new tor-X.Y.X tag is made, create rpm-tor-X.Y.Z tag

There are two build/preparation scripts in ~/TorBuildScripts:

tor_build.sh - builds the packages for various distros (look into it, there are
    some directory location settings, needs a Tor git repo checked out). The
    git tag can be specified either as arg1 on command line, otherwise the
    script will ask (script will pull to update the repo).
    - there are fine-cut versions of the script for 0.2.3.x and 0.2.4.x,
      see TAILORED SCRIPTS below

move_tor_rpms.sh - once the RPMs are built and signed, this script will create
    the metadata structure and sort rpms from source dir to destination dir
    with proper distro/arch subdirs.

Steps:

1. run tor_build.sh
2. try installing some of the packages (yum localinstall tor-X.Y.Z.rpm), test
   basic client functionality with something like:

    curl --socks5-hostname localhost:9050 https://check.torproject.org

3. sign the packages (offline; rpm --resign), reupload
4. use move_tor_rpms.sh to make the structure needed for serving via http
   (takes src dir and dst dir as arguments)


REQUIREMENTS ON THE BUILD ENVIRONMENT (already done on feddei)

  I. Development software

They are described in doc/tor-rpm-creation.txt in Tor git repo. One shortcut
for installing the build tools (this is described in the linked howtos there):

Install group development-tools
in Fedora: 
    yum install @development-tools
in EL: 
    yum groupinstall development-tools

yum install fedora-packager redhat-rpm-config

#this is Tor specific for building the initial RPMs
yum install libevent-devel openssl-devel

Add the user you're using to build to 'mock' group:

usermod -a -G mock makerpm

  II. GPG keys for git tag verification

The user building RPMs must have keys of Tor devs imported via 'gpg --import'
so that tags can be verified. Look for keys/fingerprints here (it may be outdated):

https://www.torproject.org/docs/signing-keys.html.en

It is still recommended though to go over the build output of tor_build.sh just
to check which keys signed the tag and what gpg thought about it.


CAVEATS

  I. EL5, GPG keys and RPM signatures

Using GPG keys and signatures created on newer distros (like EL6 or F16) may
cause EL5 to fail validating signature:

https://bugzilla.redhat.com/show_bug.cgi?id=436812#c10

There are multiple issues:

- you can't use RSA key generated on newer distro, because EL5's rpm/gpg will
  compute incorrect key IDs (V4 key vs V3 key)
- digest algo and signature format must be kept backward compatible to SHA1
  (see the gpg params in the above bug)
- rpm does not handle subkeys (DSA with ElGamal subkey being the only apparent
  exception)
- large GPG DSA keys (>=2048 bit) cannot be used with SHA1 - gnupg will reject
  to make such signature, wants at least 256-bit digest (RFC 4880 thing)
- gnupg digest preferences can override RPM digest preferences, so you might
  end up with SHA256 digest which is not liked by EL5 rpm

For EL5, use 1024-bit DSA key for RPM signing.

Possibly handy macro to put in ~/.rpmmacros (forces the digest type over GPG
preferences):

%__gpg_sign_cmd                 %{__gpg} \
        gpg --batch --digest-algo=sha1 --no-verbose --no-armor \
        --passphrase-fd 3 --no-secmem-warning \
        -u "%{_gpg_name}" -sbo %{__signature_filename} %{__plaintext_filename}

There's also "--force-v3-sigs" parameter that would force V3 signatures, but
experimentation showed that V4 sigs are OK, just V4 RSA keys are not.


NOTES

Remember to increase '%define specver' in tor.spec.in if you're creating a new
RPM while the Tor version is the same (e.g. when fixing a packaging issue). The
specver is reset with new Tor version, so no need to commit it.


TAILORED BUILD SCRIPTS

Currently a copy of the scripts is kept at:

https://gitweb.torproject.org/user/hiviah/rpm-build-scripts.git

It contains some tailored copies for 0.2.3 and 0.2.4 builds (tor_build-0.2.3.sh
and tor_build-0.2.4.sh).

The scripts can be run without parameters and will ask for tag to build.

Note: don't run both scripts at the same time, they share 'mock' configurations
and could remove the resulting rpms they built (can be separated, but you'd need
more space for caches and chroots).


MERGING REPO USED FOR RPM BUILDING FROM ORIGIN (git.tpo/tor.git)

The RPM fork of Tor is currently at:

https://gitweb.torproject.org/user/hiviah/rpm-tor.git

(let's call the remote 'rpm-tor').

Let's assume the remote for official (origin) source is named 'tpo', can be added like:

    git remote add tpo https://git.torproject.org/tor.git

Currently, the following path has the repo branches up to 0.2.6.x:

    /home/makerpm/TorBuildRepos/Tor.git.rpm-release-allbranches

 1) Merging for 0.2.6 release, 0.2.6.N is the new tag the RPM is built for:

    git checkout rpm-release-0.2.6
    git fetch tpo "refs/tags/*:refs/tags/*"
    git pull tpo refs/tags/tor-0.2.6.N
    git tag -v tor-0.2.6.N
    # make changes like editing tor.spec.in
    # commit, build and check the built RPMs, push back to 'rpm-tor'
    # Sign the correct rpm-tor tag if everything looks ok, push it
    git tag -s -u "0x8D62BB1E" -m "Tagging RPM for Tor 0.2.6.N" "rpm-tor-0.2.6.N-alpha"
    git push # check if fast-forwardable (fix previous tag locally if something failed)
    git push --tags
    # Sign RPMs, create repo structure, test, upload

 2) Merging for 0.2.7 release, 0.2.7.N is the new tag the RPM is built for:

    git checkout rpm-release-0.2.7
    git fetch tpo "refs/tags/*:refs/tags/*"
    git pull tpo refs/tags/tor-0.2.7.N
    git tag -v tor-0.2.7.N
    # make changes like editing tor.spec.in
    # commit, build and check the built RPMs, push back to 'rpm-tor'
    # Sign the correct rpm-tor tag if everything looks ok, push it
    git tag -s -u "0x8D62BB1E" -m "Tagging RPM for Tor 0.2.7.N" "rpm-tor-0.2.7.N-alpha"
    git push # check if fast-forwardable (fix previous tag locally if something failed)
    git push --tags
    # Sign RPMs, create repo structure, test, upload

 3) Merging for 0.2.8 release, 0.2.8.N is the new tag the RPM is built for:

    git checkout rpm-release-0.2.8
    git fetch tpo "refs/tags/*:refs/tags/*"
    git pull tpo refs/tags/tor-0.2.8.N
    git tag -v tor-0.2.8.N
    # make changes like editing tor.spec.in
    # commit, build and check the built RPMs, push back to 'rpm-tor'
    # Sign the correct rpm-tor tag if everything looks ok, push it
    git tag -s -u "0x8D62BB1E" -m "Tagging RPM for Tor 0.2.8.N" "rpm-tor-0.2.8.N-alpha"
    git push # check if fast-forwardable (fix previous tag locally if something failed)
    git push --tags
    # Sign RPMs, create repo structure, test, upload


UPLOAD TO DEB.TORPROJECT.ORG FOR DISTRIBUTION

Here we assume rpms are signed, have metadata and have been tested (after
running move_tor_rpms.sh).

 1) scp the rpms with metadata onto palmeri.torproject.org, ssh to palmeri
 2) update stuff in /srv/deb.torproject.org/ftp/torproject.org/rpm
 3) make sure the above dir and its subdirs are group writable:

    find /srv/deb.torproject.org/ftp/torproject.org/rpm -type d -exec chmod g+w '{}' ';'
  
  4) double-check:

    find /srv/deb.torproject.org/ftp/torproject.org/rpm -type d -exec ls -ld '{}' ';'

  5) run (may take a while, even >= 30 min):

    static-update-component deb.torproject.org

  6) check deb.torproject.org from HTTP client if stuff is really there, or
     check directly via yum update


THINGS YET TO BE DONE

See TODO file, I put things that were started but not yet finished in
"unfinished-*" branches in the rpm-tor repo.


VARIOUS NOTES (RECORDED MOSTLY SO THAT I WON'T FORGET LATER)

- The read-write access to tpo git repos is via URI (SSH keys are set up as
  usual in ~/.ssh/config):
  ssh://git@git-rw.torproject.org/user/hiviah/rpm-build-scripts.git
- Web access SVN URI (for modifying RPM instructions) is:
  https://svn.torproject.org/svn/website/trunk
  - make sure access is via TLS tunnel, because credentials are sent in clear
    inside tunnel (i.e. check against typos in https part, check with wireshark
    first when making some meta-repo changes)
  - after making change, someone from tpo needs to be told to rebuild web from
    the SVN (at least Sebastian, weasel and phobos have priviliges to do it)
- when using ssh-agent, don't forget to have IdentitiesOnly option in
  ~/.ssh/config (also usual things like no X Forwarding, no agent forwarding,
  etc.)
- useful git aliases for non-X11 history inspection when merging ([alias] in
  ~/.gitconfig):
    l = log --graph --decorate
    fl = log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%an | %ai)%Creset' --abbrev-commit --date=rfc
- to get the same autoconf substitution as in rpm build later:
    LIBS=-lrt ./autogen.sh && ./configure --prefix='' --bindir=/usr/bin && make dist-rpm

