Merge branch 'maint-0.4.2' into bug40076_042

c2d5ec5e · Nick Mathewson · c4742b89 · cdb0e6c2 · c2d5ec5e · c2d5ec5e
Commit c2d5ec5e authored Jul 30, 2020 by Nick Mathewson 🎨
--- a/.gitignore
+++ b/.gitignore
@@ -158,14 +158,20 @@ uptime-*.json

 # /src/lib
 /src/lib/libcurve25519_donna.a
+/src/lib/libtor-buf.a
+/src/lib/libtor-buf-testing.a
 /src/lib/libtor-compress.a
 /src/lib/libtor-compress-testing.a
+/src/lib/libtor-confmgt.a
+/src/lib/libtor-confmgt-testing.a
 /src/lib/libtor-container.a
 /src/lib/libtor-container-testing.a
 /src/lib/libtor-crypt-ops.a
 /src/lib/libtor-crypt-ops-testing.a
 /src/lib/libtor-ctime.a
 /src/lib/libtor-ctime-testing.a
+/src/lib/libtor-dispatch.a
+/src/lib/libtor-dispatch-testing.a
 /src/lib/libtor-encoding.a
 /src/lib/libtor-encoding-testing.a
 /src/lib/libtor-evloop.a
@@ -198,6 +204,8 @@ uptime-*.json
 /src/lib/libtor-osinfo-testing.a
 /src/lib/libtor-process.a
 /src/lib/libtor-process-testing.a
+/src/lib/libtor-pubsub.a
+/src/lib/libtor-pubsub-testing.a
 /src/lib/libtor-sandbox.a
 /src/lib/libtor-sandbox-testing.a
 /src/lib/libtor-string.a
@@ -213,6 +221,8 @@ uptime-*.json
 /src/lib/libtor-tls.a
 /src/lib/libtor-tls-testing.a
 /src/lib/libtor-trace.a
+/src/lib/libtor-version.a
+/src/lib/libtor-version-testing.a
 /src/lib/libtor-wallclock.a
 /src/lib/libtor-wallclock-testing.a

@@ -240,20 +250,22 @@ uptime-*.json
 /src/test/test
 /src/test/test-slow
 /src/test/test-bt-cl
-/src/test/test-child
+/src/test/test-process
 /src/test/test-memwipe
 /src/test/test-ntor-cl
 /src/test/test-hs-ntor-cl
+/src/test/test-rng
 /src/test/test-switch-id
 /src/test/test-timers
 /src/test/test_workqueue
 /src/test/test.exe
 /src/test/test-slow.exe
 /src/test/test-bt-cl.exe
-/src/test/test-child.exe
+/src/test/test-process.exe
 /src/test/test-ntor-cl.exe
 /src/test/test-hs-ntor-cl.exe
 /src/test/test-memwipe.exe
+/src/test/test-rng.exe
 /src/test/test-switch-id.exe
 /src/test/test-timers.exe
 /src/test/test_workqueue.exe

--- a/.travis.yml
+++ b/.travis.yml
@@ -29,6 +29,8 @@ env:
    - HARDENING_OPTIONS="--enable-expensive-hardening"
    ## We turn off asciidoc by default, because it's slow
    - ASCIIDOC_OPTIONS="--disable-asciidoc"
+    ## Our default rust version is the minimum supported version
+    - RUST_VERSION="1.31.0"
    ## Turn off tor's sandbox in chutney, until we fix sandbox errors that are
    ## triggered by Ubuntu Xenial and Bionic. See #32722.
    - CHUTNEY_TOR_SANDBOX="0"
@@ -70,7 +72,7 @@ matrix:
    - env: NSS_OPTIONS="--enable-nss" C_DIALECT_OPTIONS="-std=gnu99"

    ## We include a single coverage build with the best options for coverage
-    - env: COVERAGE_OPTIONS="--enable-coverage" HARDENING_OPTIONS=""
+    - env: COVERAGE_OPTIONS="--enable-coverage" HARDENING_OPTIONS="" TOR_TEST_RNG_SEED="636f766572616765"

    ## We clone our stem repo and run `make test-stem`
    - env: TEST_STEM="yes" SKIP_MAKE_CHECK="yes"
@@ -79,7 +81,7 @@ matrix:
    ## concurrent macOS jobs. We're not actively developing Rust, so it is
    ## the lowest priority.
    ## We run rust on macOS, because we have seen macOS rust failures before
-    #- env: RUST_OPTIONS="--enable-rust --enable-cargo-online-mode"
+    #- env: RUST_VERSION="nightly" RUST_OPTIONS="--enable-rust --enable-cargo-online-mode"
    #  compiler: clang
    #  os: osx

@@ -96,7 +98,7 @@ matrix:

    ## Since this job is disabled, there's not much point having an exception
    ## for it
-    #- env: RUST_OPTIONS="--enable-rust --enable-cargo-online-mode"
+    #- env: RUST_VERSION="nightly" RUST_OPTIONS="--enable-rust --enable-cargo-online-mode"
    #  compiler: clang
    #  os: osx

@@ -125,6 +127,7 @@ addons:
      - libscrypt-dev
      - libseccomp-dev
      - libzstd-dev
+      - shellcheck
      ## Conditional build dependencies
      ## Always installed, so we don't need sudo
      - asciidoc
@@ -153,6 +156,7 @@ addons:
      - pkg-config
      ## Optional build dependencies
      - ccache
+      - shellcheck
      ## Conditional build dependencies
      ## Always installed, because manual brew installs are hard to get right
      - asciidoc
@@ -183,8 +187,8 @@ install:
  - if [[ "$ASCIIDOC_OPTIONS" == "" ]] && [[ "$TRAVIS_OS_NAME" == "osx" ]]; then export XML_CATALOG_FILES="/usr/local/etc/xml/catalog"; fi
  ## If we're using Rust, download rustup
  - if [[ "$RUST_OPTIONS" != "" ]]; then curl -Ssf -o rustup.sh https://sh.rustup.rs; fi
-  ## Install the nightly channels of rustc and cargo and setup our toolchain environment
-  - if [[ "$RUST_OPTIONS" != "" ]]; then sh rustup.sh -y --default-toolchain nightly; fi
+  ## Install the stable channels of rustc and cargo and setup our toolchain environment
+  - if [[ "$RUST_OPTIONS" != "" ]]; then sh rustup.sh -y --default-toolchain $RUST_VERSION; fi
  - if [[ "$RUST_OPTIONS" != "" ]]; then source $HOME/.cargo/env; fi
  ## If we're testing rust builds in offline-mode, then set up our vendored dependencies
  - if [[ "$TOR_RUST_DEPENDENCIES" == "true" ]]; then export TOR_RUST_DEPENDENCIES=$PWD/src/ext/rust/crates; fi
@@ -209,6 +213,10 @@ install:
  - if [[ "$CHUTNEY" != "" ]]; then pushd "$CHUTNEY_PATH"; git log -1 ; popd ; fi
  ## If we're running stem, show the stem version and commit
  - if [[ "$TEST_STEM" != "" ]]; then pushd stem; python -c "from stem import stem; print(stem.__version__);"; git log -1; popd; fi
+  ## We don't want Tor tests to depend on default configuration file at
+  ## ~/.torrc. So we put some random bytes in there, to make sure we get build
+  ## failures in case Tor is reading it during CI jobs.
+  - dd ibs=1 count=1024 if=/dev/urandom > ~/.torrc

 script:
  # Skip test_rebind on macOS

--- a/ChangeLog
+++ b/ChangeLog
--- a/LICENSE
+++ b/LICENSE
@@ -9,7 +9,8 @@
        there may be other license terms that you should be aware of.

 ===============================================================================
-Tor is distributed under this license:
+Tor is distributed under the "3-clause BSD" license, a commonly used
+software license that means Tor is both free software and open source:

 Copyright (c) 2001-2004, Roger Dingledine
 Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson

--- a/Makefile.am
+++ b/Makefile.am
@@ -31,9 +31,7 @@ TESTING_TOR_BINARY=$(top_builddir)/src/app/tor$(EXEEXT)
 endif

 if USE_RUST
-## this MUST be $(), otherwise am__DEPENDENCIES will not track it
-rust_ldadd=$(top_builddir)/$(TOR_RUST_LIB_PATH) \
-	$(TOR_RUST_EXTRA_LIBS)
+rust_ldadd=$(top_builddir)/$(TOR_RUST_LIB_PATH)
 else
 rust_ldadd=
 endif
@@ -42,6 +40,10 @@ endif
 TOR_UTIL_LIBS = \
        src/lib/libtor-geoip.a \
 	src/lib/libtor-process.a \
+        src/lib/libtor-buf.a \
+	src/lib/libtor-confmgt.a \
+	src/lib/libtor-pubsub.a \
+	src/lib/libtor-dispatch.a \
 	src/lib/libtor-time.a \
 	src/lib/libtor-fs.a \
 	src/lib/libtor-encoding.a \
@@ -62,6 +64,7 @@ TOR_UTIL_LIBS = \
 	src/lib/libtor-malloc.a \
 	src/lib/libtor-wallclock.a \
 	src/lib/libtor-err.a \
+	src/lib/libtor-version.a \
 	src/lib/libtor-intmath.a \
 	src/lib/libtor-ctime.a

@@ -71,6 +74,10 @@ if UNITTESTS_ENABLED
 TOR_UTIL_TESTING_LIBS = \
        src/lib/libtor-geoip-testing.a \
 	src/lib/libtor-process-testing.a \
+        src/lib/libtor-buf-testing.a \
+	src/lib/libtor-confmgt-testing.a \
+	src/lib/libtor-pubsub-testing.a \
+	src/lib/libtor-dispatch-testing.a \
 	src/lib/libtor-time-testing.a \
 	src/lib/libtor-fs-testing.a \
 	src/lib/libtor-encoding-testing.a \
@@ -91,6 +98,7 @@ TOR_UTIL_TESTING_LIBS = \
 	src/lib/libtor-malloc-testing.a \
 	src/lib/libtor-wallclock-testing.a \
 	src/lib/libtor-err-testing.a \
+	src/lib/libtor-version-testing.a \
 	src/lib/libtor-intmath.a \
 	src/lib/libtor-ctime-testing.a
 endif
@@ -159,7 +167,28 @@ EXTRA_DIST+= \
 	README						\
 	ReleaseNotes					\
 	scripts/maint/checkIncludes.py                  \
-	scripts/maint/checkSpace.pl
+	scripts/maint/checkSpace.pl 			\
+	scripts/maint/checkShellScripts.sh			\
+	scripts/maint/practracker/README                \
+	scripts/maint/practracker/exceptions.txt	\
+	scripts/maint/practracker/includes.py		\
+	scripts/maint/practracker/metrics.py		\
+	scripts/maint/practracker/practracker.py	\
+	scripts/maint/practracker/practracker_tests.py	\
+	scripts/maint/practracker/problem.py		\
+	scripts/maint/practracker/testdata/.may_include	\
+	scripts/maint/practracker/testdata/a.c		\
+	scripts/maint/practracker/testdata/b.c 		\
+	scripts/maint/practracker/testdata/ex0-expected.txt \
+	scripts/maint/practracker/testdata/ex0.txt 	\
+	scripts/maint/practracker/testdata/ex1-expected.txt \
+	scripts/maint/practracker/testdata/ex1.txt 	\
+	scripts/maint/practracker/testdata/ex1-overbroad-expected.txt 	\
+	scripts/maint/practracker/testdata/ex.txt 	\
+	scripts/maint/practracker/testdata/header.h 	\
+	scripts/maint/practracker/testdata/not_c_file	\
+	scripts/maint/practracker/test_practracker.sh   \
+	scripts/maint/practracker/util.py

 ## This tells etags how to find mockable function definitions.
 AM_ETAGSFLAGS=--regex='{c}/MOCK_IMPL([^,]+,\W*\([a-zA-Z0-9_]+\)\W*,/\1/s'
@@ -177,7 +206,7 @@ TEST_CFLAGS=
 TEST_CPPFLAGS=-DTOR_UNIT_TESTS @TOR_MODULES_ALL_ENABLED@
 TEST_NETWORK_FLAGS=--hs-multi-client 1
 endif
-TEST_NETWORK_WARNING_FLAGS=--quiet --only-warnings
+TEST_NETWORK_SHOW_WARNINGS_FOR_LAST_RUN_FLAGS=--quiet --only-warnings

 if LIBFUZZER_ENABLED
 TEST_CFLAGS += -fsanitize-coverage=trace-pc-guard,trace-cmp,trace-div
@@ -213,7 +242,10 @@ doxygen:
 test: all
 	$(top_builddir)/src/test/test

-check-local: check-spaces check-changes check-includes
+shellcheck:
+	$(top_srcdir)/scripts/maint/checkShellScripts.sh
+
+check-local: check-spaces check-changes check-includes shellcheck

 need-chutney-path:
 	@if test ! -d "$$CHUTNEY_PATH"; then \
@@ -233,12 +265,15 @@ test-network: need-chutney-path $(TESTING_TOR_BINARY) src/tools/tor-gencert
 	$(top_srcdir)/src/test/test-network.sh $(TEST_NETWORK_FLAGS)

 # Run all available tests using automake's test-driver
-# only run IPv6 tests if we can ping6 ::1 (localhost)
-# only run IPv6 tests if we can ping ::1 (localhost)
-# some IPv6 tests will fail without an IPv6 DNS server (see #16971 and #17011)
-# only run mixed tests if we have a tor-stable binary
-# Try the syntax for BSD ping6, Linux ping6, and Linux ping -6,
+#   - only run IPv6 tests if we can ping6 or ping -6 ::1 (localhost)
+#     we try the syntax for BSD ping6, Linux ping6, and Linux ping -6,
 #     because they're incompatible
+#   - some IPv6 tests may fail without an IPv6 DNS server
+#     (see #16971 and #17011)
+#   - only run mixed tests if we have a tor-stable binary
+#   - show tor warnings on the console after each network run
+#     (otherwise, warnings go to the logs, and people don't see them unless
+#     there is a network failure)
 test-network-all: need-chutney-path test-driver $(TESTING_TOR_BINARY) src/tools/tor-gencert
 	mkdir -p $(TEST_NETWORK_ALL_LOG_DIR)
 	rm -f $(TEST_NETWORK_ALL_LOG_DIR)/*.log $(TEST_NETWORK_ALL_LOG_DIR)/*.trs
@@ -262,7 +297,7 @@ test-network-all: need-chutney-path test-driver $(TESTING_TOR_BINARY) src/tools/
 	done; \
 	for f in $$flavors; do \
 		$(SHELL) $(top_srcdir)/test-driver --test-name $$f --log-file $(TEST_NETWORK_ALL_LOG_DIR)/$$f.log --trs-file $(TEST_NETWORK_ALL_LOG_DIR)/$$f.trs $(TEST_NETWORK_ALL_DRIVER_FLAGS) $(top_srcdir)/src/test/test-network.sh --flavor $$f $(TEST_NETWORK_FLAGS); \
-		$(top_srcdir)/src/test/test-network.sh $(TEST_NETWORK_WARNING_FLAGS); \
+		$(top_srcdir)/src/test/test-network.sh $(TEST_NETWORK_SHOW_WARNINGS_FOR_LAST_RUN_FLAGS); \
 	done; \
 	echo "Log and result files are available in $(TEST_NETWORK_ALL_LOG_DIR)."; \
 	! grep -q FAIL $(TEST_NETWORK_ALL_LOG_DIR)/*.trs
@@ -317,11 +352,8 @@ coverage-html-full: all
 	lcov --remove "$(HTML_COVER_DIR)/lcov.tmp" --rc lcov_branch_coverage=1 'test/*' 'ext/tinytest*' '/usr/*' --output-file "$(HTML_COVER_DIR)/lcov.info"
 	genhtml --branch-coverage -o "$(HTML_COVER_DIR)" "$(HTML_COVER_DIR)/lcov.info"

-# Avoid strlcpy.c, strlcat.c, aes.c, OpenBSD_malloc_Linux.c, sha256.c,
-# tinytest*.[ch]
-check-spaces:
-if USE_PERL
-	$(PERL) $(top_srcdir)/scripts/maint/checkSpace.pl -C \
+# For scripts: avoid src/ext and src/trunnel.
+OWNED_TOR_C_FILES=\
 		$(top_srcdir)/src/lib/*/*.[ch] \
 		$(top_srcdir)/src/core/*/*.[ch] \
 		$(top_srcdir)/src/feature/*/*.[ch] \
@@ -329,13 +361,26 @@ if USE_PERL
 		$(top_srcdir)/src/test/*.[ch] \
 		$(top_srcdir)/src/test/*/*.[ch] \
 		$(top_srcdir)/src/tools/*.[ch]
+
+check-spaces:
+if USE_PERL
+	$(PERL) $(top_srcdir)/scripts/maint/checkSpace.pl -C \
+		$(OWNED_TOR_C_FILES)
 endif

 check-includes:
 if USEPYTHON
-	$(PYTHON) $(top_srcdir)/scripts/maint/checkIncludes.py
+	$(PYTHON) $(top_srcdir)/scripts/maint/practracker/includes.py $(top_srcdir)
 endif

+check-best-practices:
+if USEPYTHON
+	@$(PYTHON) $(top_srcdir)/scripts/maint/practracker/practracker.py $(top_srcdir) $(TOR_PRACTRACKER_OPTIONS)
+endif
+
+practracker-regen:
+	$(PYTHON) $(top_srcdir)/scripts/maint/practracker/practracker.py --regen $(top_srcdir)
+
 check-docs: all
 	$(PERL) $(top_builddir)/scripts/maint/checkOptionDocs.pl

@@ -412,13 +457,13 @@ endif
 check-changes:
 if USEPYTHON
 	@if test -d "$(top_srcdir)/changes"; then \
-		$(PYTHON) $(top_srcdir)/scripts/maint/lintChanges.py $(top_srcdir)/changes; \
+		PACKAGE_VERSION=$(PACKAGE_VERSION) $(PYTHON) $(top_srcdir)/scripts/maint/lintChanges.py $(top_srcdir)/changes; \
 		fi
 endif

 .PHONY: update-versions
 update-versions:
-	$(PERL) $(top_builddir)/scripts/maint/updateVersions.pl
+	abs_top_srcdir="$(abs_top_srcdir)" $(PYTHON) $(top_srcdir)/scripts/maint/update_versions.py

 .PHONY: callgraph
 callgraph:
@@ -431,6 +476,25 @@ version:
 	   (cd "$(top_srcdir)" && git rev-parse --short=16 HEAD); \
 	fi

+.PHONY: autostyle-ifdefs
+autostyle-ifdefs:
+	$(PYTHON) scripts/maint/annotate_ifdef_directives.py $(OWNED_TOR_C_FILES)
+
+.PHONY: autostyle-ifdefs
+autostyle-operators:
+	$(PERL) scripts/coccinelle/test-operator-cleanup $(OWNED_TOR_C_FILES)
+
+.PHONY: rectify-includes
+rectify-includes:
+	$(PYTHON) scripts/maint/rectify_include_paths.py
+
+.PHONY: update-copyright
+update-copyright:
+	$(PERL) scripts/maint/updateCopyright.pl $(OWNED_TOR_C_FILES)
+
+.PHONY: autostyle
+autostyle: update-versions rustfmt autostyle-ifdefs rectify-includes
+
 mostlyclean-local:
 	rm -f $(top_builddir)/src/*/*.gc{da,no} $(top_builddir)/src/*/*/*.gc{da,no}
 	rm -rf $(HTML_COVER_DIR)

--- a/ReleaseNotes
+++ b/ReleaseNotes
--- a/autogen.sh
+++ b/autogen.sh
 #!/bin/sh

-if [ -x "`which autoreconf 2>/dev/null`" ] ; then
+if command -v autoreconf; then
  opt="-i -f -W all,error"

-  for i in $@; do
+  for i in "$@"; do
    case "$i" in
      -v)
        opt="${opt} -v"
@@ -11,6 +11,7 @@ if [ -x "`which autoreconf 2>/dev/null`" ] ; then
    esac
  done

+  # shellcheck disable=SC2086
  exec autoreconf $opt
 fi


--- a/changes/29241_diagnostic
+++ b/changes/29241_diagnostic
-  o Minor features (NSS, diagnostic):
-    - Try to log an error from NSS (if there is any) and a more useful
-      description of our situation if we are using NSS and a call to
-      SSL_ExportKeyingMaterial() fails.  Diagnostic for ticket 29241.
--- a/changes/bug12399
+++ b/changes/bug12399
-  o Minor bugfixes (logging):
-    - Change log level of message "Hash of session info was not as expected"
-      to LOG_PROTOCOL_WARN. Fixes bug 12399; bugfix on 0.1.1.10-alpha.
--- a/changes/bug13221
+++ b/changes/bug13221
-  o Minor bugfixes (logging):
-    - Correct a misleading error message when IPv4Only or IPv6Only
-      is used but the resolved address can not be interpreted as an
-      address of the specified IP version.  Fixes bug 13221; bugfix
-      on 0.2.3.9-alpha.  Patch from Kris Katterjohn.
--- a/changes/bug22619
+++ b/changes/bug22619
-  o Minor bugfixes (circuit isolation):
-    - Fix a logic error that prevented the SessionGroup sub-option from
-      being accepted. Fixes bug 22619; bugfix on 0.2.7.2-alpha.
--- a/changes/bug23507
+++ b/changes/bug23507
-  o Minor bugfixes (v3 single onion services):
-    - Make v3 single onion services fall back to a 3-hop intro, when there
-      all intro points are unreachable via a 1-hop path. Previously, v3
-      single onion services failed when all intro nodes were unreachable
-      via a 1-hop path. Fixes bug 23507; bugfix on 0.3.2.1-alpha.
--- a/changes/bug23818_v2
+++ b/changes/bug23818_v2
-  o Minor bugfixes (v2 single onion services):
-    - Always retry v2 single onion service intro and rend circuits with a
-      3-hop path. Previously, v2 single onion services used a 3-hop path
-      when rend circuits were retried after a remote or delayed failure,
-      but a 1-hop path for immediate retries. Fixes bug 23818;
-      bugfix on 0.2.9.3-alpha.
--- a/changes/bug23818_v3
+++ b/changes/bug23818_v3
-  o Minor bugfixes (v3 single onion services):
-    - Always retry v3 single onion service intro and rend circuits with a
-      3-hop path. Previously, v3 single onion services used a 3-hop path
-      when rend circuits were retried after a remote or delayed failure,
-      but a 1-hop path for immediate retries. Fixes bug 23818;
-      bugfix on 0.3.2.1-alpha.
--- a/changes/bug24661
+++ b/changes/bug24661
-  o Minor bugfixes (client, guard selection):
-    - When Tor's consensus has expired, but is still reasonably live, use it
-      to select guards. Fixes bug 24661; bugfix on 0.3.0.1-alpha.
--- a/changes/bug27197
+++ b/changes/bug27197
-  o Minor bugfixes (protover, rust):
-    - Reject extra commas in version string. Fixes bug 27197; bugfix on
-      0.3.3.3-alpha.
--- a/changes/bug27199
+++ b/changes/bug27199
-  o Minor bugfixes (rust):
-    - Abort on panic in all build profiles, instead of potentially unwinding
-      into C code. Fixes bug 27199; bugfix on 0.3.3.1-alpha.
--- a/changes/bug27740
+++ b/changes/bug27740
-  o Minor bugfixes (rust):
-    - Return a string that can be safely freed by C code, not one created by
-      the rust allocator, in protover_all_supported(). Fixes bug 27740; bugfix
-      on 0.3.3.1-alpha.
--- a/changes/bug27741
+++ b/changes/bug27741
-  o Minor bugfixes (rust, directory authority):
-    - Fix an API mismatch in the rust implementation of
-      protover_compute_vote(). This bug could have caused crashes on any
-      directory authorities running Tor with Rust (which we do not yet
-      recommend). Fixes bug 27741; bugfix on 0.3.3.6.
--- a/changes/bug27750
+++ b/changes/bug27750
-  o Minor bugfixes (connection, relay):
-    - Avoid a wrong BUG() stacktrace in case a closing connection is being held
-      open because the write side is rate limited but not the read side. Now,
-      the connection read side is simply shutdown instead of kept open until tor
-      is able to flush the connection and then fully close it. Fixes bug 27750;
-      bugfix on 0.3.4.1-alpha.