#          Magical Tor Browser Release Process Incantations
#
#  "May this part of our job one day be replaced by a small shell script"
#

#####################
### Prepare build ###
#####################

#. Tag any relevant component versions.
#  In particular: before tagging the tor-browser tag, the final code from
#  Torbutton needs to get included and in order to avoid unnecessary commit
#  inflation the commit for bumping the git submodule should be a fixup commit
#  to the one with "Bug 25013: Add torbutton as a tor-browser submodule" in
#  its commit message.

#. Update changelog and relevant config files in tor-browser-build.
   cd tor-browser-build
   vim projects/firefox/config
   vim ChangeLog.txt
   vim rbm.conf

#. Tag a build tag in tor-browser-build.
   make signtag-release # or `make signtag-alpha` for an alpha build

#. Push tag and version to tor-browser-build.git. In case of doing a stable
#  release with a maintenance branch use that one instead of `master`.
   torsocks git push origin master:master
   torsocks git push origin --tags


#####################
### Build         ###
#####################

#. Build and generate incremental MAR files.
   git tag -v tbb-$TORBROWSER_VERSION-buildN
   git checkout tbb-$TORBROWSER_VERSION-buildN
   make && make incrementals-release # `make alpha && make incrementals-alpha`

#. Compare the SHA256 sums of the bundles and MAR files with an independent
#  builder.
   sha256sum tor-browser-build/release/unsigned/$TORBROWSER_BUILDDIR/sha256sums-unsigned-build.txt
   sha256sum tor-browser-build/release/unsigned/$TORBROWSER_BUILDDIR/sha256sums-unsigned-build.incrementals.txt

#. If the sums match (download and) upload the bundles to your build dir on
#  people.torproject.org. Fix permissions.
   chmod 755 $TORBROWSER_BUILDDIR
   chmod 644 $TORBROWSER_BUILDDIR/*
   chmod 644 $TORBROWSER_BUILDDIR/.htaccess
   torsocks ssh people.torproject.org "mkdir ~/public_html/builds/${TORBROWSER_BUILDDIR}"
   torsocks rsync -avP $TORBROWSER_BUILDDIR/ people.torproject.org:public_html/builds/$TORBROWSER_BUILDDIR

#. (Optional): Upload your binaries to people using partial rsync over old version
   torsocks ssh people.torproject.org "mv ~/public_html/builds/${TORBROWSER_VERSION}-build1 ~/public_html/builds/$TORBROWSER_BUILDDIR"
   torsocks rsync -avP $TORBROWSER_BUILDDIR/ people.torproject.org:public_html/builds/$TORBROWSER_BUILDDIR

#. Distribute build to tor-qa@lists.torproject.org
   # XXX: Currently manual email with link to candidate build, important changes,
   # and changelog.
   # For stable releases put tails-dev@boum.org into Cc.


#######################
### Android signing ###
#######################

#. The Android signing is done separately from the desktop signing. To
#  do the Android signing you'll need to boot from a Tails usb stick,
#  with persistent storage enabled.

#. Install the openjdk-11-jdk-headless package
   apt install openjdk-11-jdk-headless

#. Make sure you have the android keys, for example in the
#  ~/Persistent/android-keys directory. This directory should include
#  the tba_alpha.p12 and tba_release.p12 files.

#. Clone tor-browser-build, and checkout the tag for the release
   git tag -v tbb-$TORBROWSER_VERSION-buildN
   git checkout tbb-$TORBROWSER_VERSION-buildN

#. Update set-config.tbb-version and set-config.android-signing
   cd tools/signing
   vim set-config.tbb-version
   vim set-config.android-signing

#. Run the android-signing script. This script will download the
#  unsigned apks from the `pkgstage` machine (where `do-all-signing`
#  is run), signed them, and upload back the signed apks to the
#  `pkgstage` machine. It should be done before the hash_signed_bundles.sh
#  step.
   ./android-signing


##########################
### Signing and upload ###
##########################

#. You need a `pkgstage` machine to store the bundles during the signing
#  process. This machine should be:
#   - secure (you will ssh to the signing machines and staticiforme
#     from there)
#   - with good bandwidth (you will download/upload bundles from there)
#
#  All steps from the `Signing and upload` are run from the `pkgstage`
#  machine unless mentioned otherwise.
#
#  The signing scripts are located in the tor-browser-build/tools/signing
#  directory.
#
#  Alternatively, you can run the script `./do-all-signing`, which will
#  do all the signing steps automatically, until the static-update-component
#  commands on staticiforme. Also the script does not currently handle
#  removing the old versions on staticiforme, so you have to do that
#  manually. The script is also not running `./macos-signer-proxy` which
#  needs to be started and stopped manually.

#. Checkout the tor-browser-build.git commit corresponding to the
#  tor browser version you need to sign/publish.
   cd tor-browser-build
   git tag -v tbb-$TORBROWSER_VERSION-buildN
   git checkout tbb-$TORBROWSER_VERSION-buildN

#. Change to the tor-browser-build/tools/signing directory. All following
#  commands are run from this directory.
   cd tor-browser-build/tools/signing

#. Set hosts information.
   vim set-config.hosts

#. Set tor browser version information. You should set tbb_version,
#  tbb_version_build, tbb_version_type. This should match the information
#  in rbm.conf.
   vim set-config.tbb-version

#. Download the tor browser build to the `pkgstage` machine from your
#  build machine (configured in set-config.hosts).
   ./sync-builder-unsigned-to-local-signed

#. Codesign the macOS dmg files.
#
#  Updload the tor browser dmg files to `macos-signer`
   ./sync-macos-local-to-macos-signer

#  Upload updated signing scripts to `macos-signer`. The scripts are
#  located in the `signing-release` or `signing-alpha` directory on
#  `macos-signer`.
   ./sync-scripts-to-macos-signer

#  Run proxy for `macos-signer`. You may need to kill an old proxy
#  process on `macos-signer` if it was still running.
   ./macos-signer-proxy

#  Enable networking on the `macos-signer`.
   macos-signer$ networksetup -setsecurewebproxystate Ethernet on

#  Sign the bundles. On `macos-signer`.
#  (replace signing-release with signing-alpha for an alpha release)
   macos-signer$ ~/signing-release/macos-signer-gatekeeper-signing

#  Notarize the bundles. On `macos-signer`.
#  (replace signing-release with signing-alpha for an alpha release)
   macos-signer$ ~/signing-release/macos-signer-notarization

#  Staple the bundles. On `macos-signer`.
#  (replace signing-release with signing-alpha for an alpha release)
   macos-signer$ ~/signing-release/macos-signer-stapler

#  Disable networking on the `macos-signer`.
   macos-signer$ networksetup -setsocksfirewallproxystate Ethernet off

#  Download the stapled bundles to `pkgstage`.
   ./sync-macos-signer-stapled-to-macos-local-stapled

#. Regenerate macOS DMG files from stapled zip files.
   ./gatekeeper-bundling.sh

#. Regenerate macOS MAR files from code signed dmg files.
   ./dmg2mar

#. Sync bundles to `linux-signer`.
   ./sync-local-to-linux-signer

#. Upload updated signing scripts to `linux-signer`. The scripts are
#  located in the `signing-release` or `signing-alpha` directory on
#  `linux-signer`.
   ./sync-scripts-to-linux-signer

#. Sign the MAR files. On `linux-signer`.
#  (replace signing-release with signing-alpha for an alpha release)
   linux-signer$ ~/signing-release/linux-signer-signmars

#. Authenticode signing of exe files. On `linux-signer`.
   linux-signer$ ~/signing-release/linux-signer-authenticode-signing

#. Authenticode timestamping.
   ./sync-linux-signer-to-local
   ./authenticode-timestamping.sh

#. Create sha256sums-signed-build files
   ./hash_signed_bundles.sh

#. Upload sha256sums-signed-build and updated exe files to `linux-signer`.
   ./sync-local-to-linux-signer

#. Gpg signing.
   ./linux-signer-gpg-sign

#. Fetch signatures on unsigned sha256sums from other builds
   ./download-unsigned-sha256sums-gpg-signatures-from-people-tpo

#. Remove old builds on `staticiforme`
   # IMPORTANT: Remove the oldest version in a series in case there is more
   # than 1 available on dist.torproject.org before proceeding
   # XXX: TORBROWSER_VERSION_OLDEST needs to be set
   staticiforme$ rm -rf /srv/dist-master.torproject.org/htdocs/torbrowser/$TORBROWSER_VERSION_OLDEST
   staticiforme$ static-update-component dist.torproject.org

#. Upload new build to `staticiforme`
   ./sync-local-to-staticiforme

#. Check diskspace available on cdn.tpo
#  We currently have enough disk space to host two alpha and stable
#  releases. However with the size of each Tor Browser release increasing,
#  it may become necessary to increase disk space. The server hosting
#  the files for cdn.tpo is savii.tpo and its disk usage can be monitored
#  by going to https://grafana.torproject.org/d/Z7T7Cfemz/node-exporter-full
#  and selecting the hosts: web-fsn-02.torproject.org, web-cymru-01.torproject.org,
#  web-fsn-01.torproject.org, and cdn-backend-sunet-01.torproject.org

#. Remove the oldest *.mar files from cdn.tpo to save space
   staticiforme$ rm -rf /srv/cdn-master.torproject.org/htdocs/aus1/torbrowser/$TORBROWSER_VERSION_OLDEST
   staticiforme$ static-update-component cdn.torproject.org

#. Sync files to dist.tpo and cdn.tpo mirrored web servers
#  (replace signing-release with signing-alpha for an alpha release)
#  It is a good idea to run the static-update-component inside screen
#  or tmux so that the upload continues if your connection to
#  staticiforme is closed.
   $ ./sync-scripts-to-staticiforme
   staticiforme$ ~/signing-release/staticiforme-prepare-cdn-dist-upload
   staticiforme$ static-update-component cdn.torproject.org && \
                 static-update-component dist.torproject.org

#. Make sure we really built from the proper Mozilla build tag by consulting
   # the respective ESR release branch (for a good overview for ESR78 see
   # https://hg.mozilla.org/releases/mozilla-esr78/graph/). For the platforms following
   # rapid release (only Android, currently), consult the beta repo
   # (https://hg.mozilla.org/releases/mozilla-beta/graph/) or the release repo
   # (https://hg.mozilla.org/releases/mozilla-release/graph/)

#. Update website's torbrowser versions file in the website git
   cd tpo
   torsocks git pull origin
   # Update `win32` in the `torbrowser-stable` section as well if we
   # include a new stable tor version (called the Windows Expert Bundle
   # on the website). See: #14152.
   vim databags/versions.ini
   git commit databags/versions.ini -m "Add new Tor Browser version"
   torsocks git push origin master:master
   cd ..
   # Check build success/failure:
   # https://jenkins.torproject.org/job/lektor-website-tpo-translation/
   # https://jenkins.torproject.org/job/lektor-website-tpo-translation-install/

#. Add new locales to the download page
   # If this release is introducing new locales, add them to the
   # databags/download-alternatives.ini file (for a stable release) or
   # the databags/download-alternatives-alpha.ini file (for an alpha
   # release).
   cd tpo
   torsocks git pull origin
   vim databags/download-alternatives.ini # or databags/download-alternatives-alpha.ini
   git commit databags/download-alternatives.ini -m "Add new Tor Browser locales"
   torsocks git push origin master:master
   cd ..

#. Create blog post from changelog
#  Edit set-config.blog to set you local blog directory.
#  Don't forget to link to Mozilla's security advisories if this is a security
#  update.
  ./create-blog-post

#. Check whether the .exe files got properly signed and timestamped
   # Point OSSLSIGNCODE to your osslsigncode binary
   pushd tor-browser-build/${channel}/signed/$TORBROWSER_VERSION
   OSSLSIGNCODE=/path/to/osslsigncode
   ../../../tools/authenticode_check.sh
   popd

#. Check whether the MAR files got properly signed
   # Point NSSDB to your nssdb containing the mar signing certificate
   # Point SIGNMAR to your signmar binary
   # Point LD_LIBRARY_PATH to your mar-tools directory
   pushd tor-browser-build/${channel}/signed/$TORBROWSER_VERSION
   NSSDB=/path/to/nssdb
   SIGNMAR=/path/to/mar-tools/signmar
   LD_LIBRARY_PATH=/path/to/mar-tools/
   ../../../tools/marsigning_check.sh
   popd

#. Generate and upload update_responses to staticiforme
   ./upload-update_responses-to-staticiforme

#. Enable update. On `staticiforme`.
#  (for alpha release, use deploy_update_responses-alpha.sh instead)
   staticiforme$ ~/deploy_update_responses-release.sh

#. Clean linux-signer and macos-signer
   ./finished-signing-clean-linux-signer
   ./finished-signing-clean-macos-signer

#  Upload APKs to Google Play
   Log into https://play.google.com/apps/publish
   Select correct app (Tor Browser or Tor Browser Alpha)
   Under left-side navigation bar, select "Production" under "Release"
   Select "Create new release" button at upper right-side
   Under "App bundles and APKs" section, "Upload" each signed APK
   After uploading APKs:
     - The "Release Name" should be automatically filled with the Firefox version
     - Next to "Release notes" click "Copy from a previous release"
     - Select the previous release and adjust the blog post url for this release
     - Save, review, and configure rollout percentage at the bottom
       - Use 25% rollout when publishing a scheduled update (without a security driver)
       - Use 100% rollout when publishing an unscheduled update or security-driven release
   Roll out.

   Note, you may receive three warning messages about:
     1) app is larger than it is necessary (android app bundle),
     2) missing deobfuscation file for better crash analysis
     3) missing debug symbol file

   These warnings are expected and should be accepted.

   See below for updating the rollout percentage.

# Release on F-Droid
  Publication on F-Droid via the Guardian Project's repository should be
  automatic. Hans is responsible for maintaining this automation.

# Update rollout percentage
   After 24 hours, check the Release Daskboard:
     - "Release"->"Production" and select the "Release Dashboard" tab
   If the Release Dashboard does not show any concerning trends (significant
   increase in crashes or ANRs (Application Not Responding)), then continue on
   to the next paragraph and increase rollout from 25% to 100%. Otherwise
   consider halting rollout as described in the `RollingBackUpdate` process.

   Select the "Releases" tab on the "Production" page. The current released
   version should indicate 25% rollout: "Available to 25% of users on Google Play"
   On the right-side of the "View release details" button of the release there
   should be a button labeled "Manage rollout" with a down-arrow. Clicking on
   that button should show two options:
     - Update rollout
     - Halt rollout

   Select "Update rollout" and increase to 100% and click "Update". The change
   should be immediately implemented and the "Manage rollout" button disappears.

#. Write an email to tor-announce in case this release is concerned with getting
#  a stable version out. Using the contents of the blog entry should do it.
