Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T17:54:04Zhttps://gitlab.torproject.org/legacy/trac/-/issues/25756EARLY_CONSENSUS_NOTICE_SKEW of 60 is too strict for some drifting dirauth clocks2020-06-13T17:54:04ZTracEARLY_CONSENSUS_NOTICE_SKEW of 60 is too strict for some drifting dirauth clocksI keep getting this error on my relay, I set my timezone as Bucharest, Romania. The relay is running ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-119-generic x86_64) and using Tor 0.3.2.10
```
[WARN] Received ns flavor consensus with skewed tim...I keep getting this error on my relay, I set my timezone as Bucharest, Romania. The relay is running ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-119-generic x86_64) and using Tor 0.3.2.10
```
[WARN] Received ns flavor consensus with skewed time (CONSENSUS): It seems that our clock is behind by 1 minutes, 1 seconds, or that theirs is ahead. Tor requires an
accurate clock to work: please check your time, timezone, and date settings.
17:58:59 [WARN] Our clock is 1 minutes, 1 seconds behind the time published in the consensus network status document (2018-04-09 15:00:00 UTC). Tor needs an accurate clock to
work correctly. Please check your time and date settings!
```
**Trac**:
**Username**: DbryrtfbcbhgfTor: 0.3.4.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/27167track "first" OR_CONN2020-06-13T17:44:18ZTaylor Yutrack "first" OR_CONNRight now the first stages of the "first" OR_CONN get reported as `BOOTSTRAP_STATUS_CONN_DIR` and `BOOTSTRAP_STATUS_HANDSHAKE` (the latter is a special bootstrap phase that gets translated into `BOOTSTRAP_STATUS_HANDSHAKE_DIR` or `BOOTST...Right now the first stages of the "first" OR_CONN get reported as `BOOTSTRAP_STATUS_CONN_DIR` and `BOOTSTRAP_STATUS_HANDSHAKE` (the latter is a special bootstrap phase that gets translated into `BOOTSTRAP_STATUS_HANDSHAKE_DIR` or `BOOTSTRAP_STATUS_HANDSHAKE_OR` depending on how much progress was previously reported. The logic in functions that report these events should be moved up to a new abstraction so lower level code has to track less high-level state.
This also eliminates some logic in `control_event_bootstrap()` that tries to figure out whether a given handshake attempt corresponds to a directory connection or an application circuit connection.Tor: 0.4.0.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/22266gather info on jump-to-80% issues2020-06-13T17:43:43ZTaylor Yugather info on jump-to-80% issuesThis ticket is now for gathering additional info and feedback about the category of jump-to-80% bootstrap reporting problems.
background:
If enough existing directory information is available, the bootstrap progress as reported to the ...This ticket is now for gathering additional info and feedback about the category of jump-to-80% bootstrap reporting problems.
background:
If enough existing directory information is available, the bootstrap progress as reported to the logs and the control connection jumps from 0% to 80% almost immediately. This is misleading and causes user frustration, as reported by Linda's study.
When bootstrapping with existing directory information, we should rescale the progress numbers so they advance on something resembling a linear time scale, which is probably closer to what users expect to see.Tor: 0.4.0.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/22265write high-level overview of bootstrap process2020-06-13T17:24:44ZTaylor Yuwrite high-level overview of bootstrap processThere needs to be a better high-level overview of the bootstrap process so developers, users, and UX people can have a better mental model of what's going on, what can go wrong, and how to fix it when it does.
Probably should be a wiki ...There needs to be a better high-level overview of the bootstrap process so developers, users, and UX people can have a better mental model of what's going on, what can go wrong, and how to fix it when it does.
Probably should be a wiki page unless someone has a better idea of where to put it. There is some text in `control-spec.txt` that is a decent starting point, but could use more elaboration of background material.website redesignTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/33339Add script to check ordering of options in manpage2020-06-13T15:51:26ZTaylor YuAdd script to check ordering of options in manpageAdd a script to check the ordering of option names within a manpage section.
This will be an initial version that doesn't restrict section names and doesn't recognize pragma comments that mark intentionally out-of-order option names.
#...Add a script to check the ordering of option names within a manpage section.
This will be an initial version that doesn't restrict section names and doesn't recognize pragma comments that mark intentionally out-of-order option names.
#32621 will contain a more fully-functional script suitable for automation.Tor: 0.4.3.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/33039Memory leaks in handle_control_getconf2020-06-13T15:50:12ZNick MathewsonMemory leaks in handle_control_getconfFound while investigating #33006:
```
=================================================================
==180005==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 32 byte(s) in 1 object(s) allocated from:
#0 0x7fb54c5cdc5...Found while investigating #33006:
```
=================================================================
==180005==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 32 byte(s) in 1 object(s) allocated from:
#0 0x7fb54c5cdc58 in __interceptor_malloc (/lib64/libasan.so.5+0x10dc58)
#1 0x55a4af274f3a in tor_malloc_ src/lib/malloc/malloc.c:45
#2 0x55a4af274fd0 in tor_malloc_zero_ src/lib/malloc/malloc.c:71
#3 0x55a4af2371ca in config_line_append src/lib/encoding/confline.c:40
#4 0x55a4aeffaec7 in control_reply_add_one_kv src/feature/control/control_proto.c:345
#5 0x55a4aefdd2ae in handle_control_getconf src/feature/control/control_cmd.c:307
#6 0x55a4aefe54b2 in handle_single_control_command src/feature/control/control_cmd.c:2376
#7 0x55a4aefe54b2 in handle_control_command src/feature/control/control_cmd.c:2407
#8 0x55a4aefd6e01 in connection_control_process_inbuf src/feature/control/control.c:508
#9 0x55a4aeee0e01 in connection_handle_read_impl src/core/mainloop/connection.c:3737
#10 0x55a4aeee0e01 in connection_handle_read src/core/mainloop/connection.c:3777
#11 0x55a4aeeecf00 in conn_read_callback src/core/mainloop/mainloop.c:892
#12 0x7fb54c32abb6 (/lib64/libevent-2.1.so.6+0x25bb6)
Indirect leak of 26 byte(s) in 1 object(s) allocated from:
#0 0x7fb54c55652d in strdup (/lib64/libasan.so.5+0x9652d)
#1 0x55a4af2751d0 in tor_strdup_ src/lib/malloc/malloc.c:165
#2 0x55a4af2371d5 in config_line_append src/lib/encoding/confline.c:41
#3 0x55a4aeffaec7 in control_reply_add_one_kv src/feature/control/control_proto.c:345
#4 0x55a4aefdd2ae in handle_control_getconf src/feature/control/control_cmd.c:307
#5 0x55a4aefe54b2 in handle_single_control_command src/feature/control/control_cmd.c:2376
#6 0x55a4aefe54b2 in handle_control_command src/feature/control/control_cmd.c:2407
#7 0x55a4aefd6e01 in connection_control_process_inbuf src/feature/control/control.c:508
#8 0x55a4aeee0e01 in connection_handle_read_impl src/core/mainloop/connection.c:3737
#9 0x55a4aeee0e01 in connection_handle_read src/core/mainloop/connection.c:3777
#10 0x55a4aeeecf00 in conn_read_callback src/core/mainloop/mainloop.c:892
#11 0x7fb54c32abb6 (/lib64/libevent-2.1.so.6+0x25bb6)
Indirect leak of 7 byte(s) in 1 object(s) allocated from:
#0 0x7fb54c55652d in strdup (/lib64/libasan.so.5+0x9652d)
#1 0x55a4af2751d0 in tor_strdup_ src/lib/malloc/malloc.c:165
#2 0x55a4af2371f5 in config_line_append src/lib/encoding/confline.c:42
#3 0x55a4aeffaec7 in control_reply_add_one_kv src/feature/control/control_proto.c:345
#4 0x55a4aefdd2ae in handle_control_getconf src/feature/control/control_cmd.c:307
#5 0x55a4aefe54b2 in handle_single_control_command src/feature/control/control_cmd.c:2376
#6 0x55a4aefe54b2 in handle_control_command src/feature/control/control_cmd.c:2407
#7 0x55a4aefd6e01 in connection_control_process_inbuf src/feature/control/control.c:508
#8 0x55a4aeee0e01 in connection_handle_read_impl src/core/mainloop/connection.c:3737
#9 0x55a4aeee0e01 in connection_handle_read src/core/mainloop/connection.c:3777
#10 0x55a4aeeecf00 in conn_read_callback src/core/mainloop/mainloop.c:892
#11 0x7fb54c32abb6 (/lib64/libevent-2.1.so.6+0x25bb6)
SUMMARY: AddressSanitizer: 65 byte(s) leaked in 3 allocation(s).
Exit code was 1
success (15.21s)
```Tor: 0.4.3.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/32708manpage: alphabetize General Options2020-06-13T15:49:01ZTaylor Yumanpage: alphabetize General OptionsThis ticket is for Swati's alphabetizing change in #4310. Making this a child ticket so we can create other child tickets for working on alphabetizing the other sections.This ticket is for Swati's alphabetizing change in #4310. Making this a child ticket so we can create other child tickets for working on alphabetizing the other sections.Tor: 0.4.3.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/32277tor manpage: reword command-line options and description; improve formatting2020-06-13T15:47:28ZTaylor Yutor manpage: reword command-line options and description; improve formattingApply most of the wording improvements from #31943 to COMMAND-LINE OPTIONS and DESCRIPTION. Also improve formatting in these sections to make option names appear with fixed-width font in the HTML version.
Did not apply tense changes fr...Apply most of the wording improvements from #31943 to COMMAND-LINE OPTIONS and DESCRIPTION. Also improve formatting in these sections to make option names appear with fixed-width font in the HTML version.
Did not apply tense changes from imperative to descriptive present tense. Changed a few existing inconsistent uses of present tense to imperative.Tor: 0.4.3.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/32206write high-level outline of target modular tor architecture2020-06-13T15:47:12ZTaylor Yuwrite high-level outline of target modular tor architectureTor: 0.4.3.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/32176tor manpage: reword FILES and back matter2020-06-13T15:47:03ZTaylor Yutor manpage: reword FILES and back matterIncorporate most of the rewording changes from #31943 for the FILES, SEE ALSO, and BUGS sections of the tor.1.txt manpage.
Includes some formatting changes to make file names display with fixed-width font in the HTML version.
Also dele...Incorporate most of the rewording changes from #31943 for the FILES, SEE ALSO, and BUGS sections of the tor.1.txt manpage.
Includes some formatting changes to make file names display with fixed-width font in the HTML version.
Also delete a redundant entry for DataDirectory/approved-routers.Tor: 0.4.3.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/31245Coverity: multiple defects in btrack tests2020-06-13T15:43:51ZTaylor YuCoverity: multiple defects in btrack testsFix the following CIDs in the bootstrap tracker tests: 1446052, 1446050, 1446049, 1446047. These are inconsequential uninitialized structure members in tests.Fix the following CIDs in the bootstrap tracker tests: 1446052, 1446050, 1446049, 1446047. These are inconsequential uninitialized structure members in tests.Tor: unspecifiedTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/30984Make a key-value line abstraction to output control replies2020-06-13T15:42:59ZTaylor YuMake a key-value line abstraction to output control repliesA few controller commands still use `connection_buf_add()` or similar low-level functions after constructing a list of reply lines. Almost all of these are key-value pairs. Create a new abstraction to output these, including by automatic...A few controller commands still use `connection_buf_add()` or similar low-level functions after constructing a list of reply lines. Almost all of these are key-value pairs. Create a new abstraction to output these, including by automatically including the correct separator character between the numeric code and the rest of the line.Tor: 0.4.3.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/23756tor's .gitlab-ci.yml is doing mirroring? why?2020-06-13T15:42:36ZIsis Lovecrufttor's .gitlab-ci.yml is doing mirroring? why?Currently in master we have the following stanza in our .gitlab-ci.yml (from #22891):
```
update: ...Currently in master we have the following stanza in our .gitlab-ci.yml (from #22891):
```
update:
script:
- "apt-get install -y --fix-missing git openssh-client"
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <("$DEPLOY_KEY")
# For Docker builds disable host key checking. Be aware that by adding that
# you are suspectible to man-in-the-middle attacks.
# WARNING: Use this only with the Docker executor, if you use it with shell
# you will overwrite your user's SSH config.
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
# In order to properly check the server's host key, assuming you created the
# SSH_SERVER_HOSTKEYS variable previously, uncomment the following two lines
# instead.
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo "$SSH_SERVER_HOSTKEYS" > ~/.ssh/known_hosts'
- echo "merging from torgit"
- git config --global user.email "labadmin@oniongit.eu"
- git config --global user.name "gitadmin"
- "mkdir tor"
- "cd tor"
- git clone --bare https://git.torproject.org/tor.git
- git push --mirror git@oniongit.eu:network/tor.git
```
Why are we doing this? Can we put a cronjob on the oniongit.eu server instead? It's pretty weird and frankly unexpected that my personal fork of tor at https://gitlab.com/isis/tor is cloning the official tor repo and then trying to mirror it to oniongit.eu. It also has a bunch of other problems:
* The `ssh-add` line [is broken, causing CI to fail because it sits there forever waiting for a passphrase](https://gitlab.com/isis/tor/-/jobs/34990901).
I was originally going to patch the `ssh-add` line to instead be `[[ -n "${DEPLOY_KEY}" -a -r "$DEPLOY_KEY" ]] && ssh-add "$DEPLOY_KEY" <<<""` but if I fix that, then all the rest of this script would run, so I'm rather glad it's failing on a more innocuous command.
* Even if the `ssh-add` line weren't broken, this whole thing fails unless it's being run from a fork on oniongit.eu.
* Why is it disabling SSH hostkey checking?!
* Why is it making the `~/.ssh` directory twice?
* Why is it assuming that environment variables are set? e.g. `$FOO` versus `${FOO}` or better `test -n ${FOO}`
* Why is it unconditionally setting (global!) git config options? (I assume to disable the warning that git spits out when you don't have `$GIT_{AUTHOR,COMMITTER}_{NAME,EMAIL}` set, but why would a CI config set them globally instead of just setting the correct environment variables?)
* Why are the mirror URLs hardcoded?
* Why is the git username and email hardcoded?
* Why is any of this even running when I push to https://gitlab.com/isis/tor?
* Why is any of this even running when I push anywhere?
* Why is it unconditionally starting an ssh-agent?
* Why is using the existence of a ([deprecated!](https://superuser.com/questions/1021834/what-are-dockerenv-and-dockerinit#1021925)) `/.dockerenv` file to determine if we're in a docker container?
* Why is it assuming we're in the _correct_ docker container, when lots of things, especially lots of CI systems, use docker?
I'm sorry if this is all necessary and I'm just not understanding the setup, but it's all just extremely unexpected behaviour from what is supposed to be a CI config file. Further, it's not even doing the same testing as our .travis.yml, but I'll make another ticket for that issue.Tor: unspecifiedTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/30007refactor control.c output to be more abstract2020-06-13T15:42:36ZTaylor Yurefactor control.c output to be more abstractLots of code explicitly assembles control protocol responses, including the numeric response codes, the line separators, etc.
We should abstract this better to simplify code and for greater consistency in the protocol.Lots of code explicitly assembles control protocol responses, including the numeric response codes, the line separators, etc.
We should abstract this better to simplify code and for greater consistency in the protocol.Tor: 0.4.1.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/30889Eliminate some uses of lower-level control protocol output functions2020-06-13T15:42:36ZTaylor YuEliminate some uses of lower-level control protocol output functionsContinuing the work of #30007, refactor more control.c stuff to stop using lower-level control protocol output functions like `connection_write_str_to_buf()`, `connection_printf_to_buf()`, and `connection_buf_add()`.Continuing the work of #30007, refactor more control.c stuff to stop using lower-level control protocol output functions like `connection_write_str_to_buf()`, `connection_printf_to_buf()`, and `connection_buf_add()`.Tor: 0.4.2.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/29976rework bootstrap reporting to use pubsub2020-06-13T15:40:03ZTaylor Yurework bootstrap reporting to use pubsubTor: 0.4.2.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/28925distinguish PT vs proxy for real in bootstrap tracker2020-06-13T15:39:42ZTaylor Yudistinguish PT vs proxy for real in bootstrap trackerThe bootstrap tracker work in #27167 adds distinctions (in the form of additional bootstrap phases) between connecting directly to a relay vs connecting through a proxy. It also tries to distinguish proxies from PTs.
The changes to do ...The bootstrap tracker work in #27167 adds distinctions (in the form of additional bootstrap phases) between connecting directly to a relay vs connecting through a proxy. It also tries to distinguish proxies from PTs.
The changes to do the distinguishing between PT vs proxy don't work, because `conn->proxy_type` gets set to the protocol type of the underlying protocol that tor uses to talk to the PT locally.Tor: 0.4.0.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/29789practracker.py codec exception in some locales2020-06-13T15:39:25ZTaylor Yupractracker.py codec exception in some localespractracker.py, implemented in #29221, seems to have a locale dependency when python3 is being used. If the locale isn't a UTF-8 locale, UTF-8 characters in sources can result in an exception:
```
$ LANG=en_US.US-ASCII make check-best-...practracker.py, implemented in #29221, seems to have a locale dependency when python3 is being used. If the locale isn't a UTF-8 locale, UTF-8 characters in sources can result in an exception:
```
$ LANG=en_US.US-ASCII make check-best-practices PYTHON=python
python ../scripts/maint/practracker/practracker.py ..
mirkwood:build-norust tlyu$ LANG=en_US.US-ASCII make check-best-practices
python3 ../scripts/maint/practracker/practracker.py ..
Traceback (most recent call last):
File "../scripts/maint/practracker/practracker.py", line 151, in <module>
main()
File "../scripts/maint/practracker/practracker.py", line 134, in main
found_new_issues = consider_all_metrics(files_list)
File "../scripts/maint/practracker/practracker.py", line 89, in consider_all_metrics
found_new_issues |= consider_metrics_for_file(fname, f)
File "../scripts/maint/practracker/practracker.py", line 104, in consider_metrics_for_file
found_new_issues |= consider_file_size(fname, f)
File "../scripts/maint/practracker/practracker.py", line 51, in consider_file_size
file_size = metrics.get_file_len(f)
File "/Users/tlyu/src/tor/scripts/maint/practracker/metrics.py", line 11, in get_file_len
for i, l in enumerate(f):
File "/Users/tlyu/src/brew/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128)
make: *** [check-best-practices] Error 1
```
I'm also seeing this on gitlab.com CI, but I don't know offhand what its locale environment variables are.
We might want to use the `encoding=` keyword parameter to `open()`, but I think that would no longer be python2 compatible.Tor: 0.4.1.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/29210Distribute control.c functionality across various modules2020-06-13T15:37:19ZNick MathewsonDistribute control.c functionality across various modulesThe control.c module accesses, and is accessed by, many other modules: it represents a failure of modularity and layering.
Instead, we should give control.c only the responsibility of marshalling and unmarshalling messages for the contr...The control.c module accesses, and is accessed by, many other modules: it represents a failure of modularity and layering.
Instead, we should give control.c only the responsibility of marshalling and unmarshalling messages for the controller, and give individual modules the responsibility for publishing events and handling commands.Tor: 0.4.3.x-finalTaylor YuTaylor Yuhttps://gitlab.torproject.org/legacy/trac/-/issues/28929fix typo/mispaste in BOOTSTRAP_STATUS_AP_CONN_PROXY summary text2020-06-13T15:35:58ZTaylor Yufix typo/mispaste in BOOTSTRAP_STATUS_AP_CONN_PROXY summary textChanges for #27167 had a typo so some text was missing from the summary.Changes for #27167 had a typo so some text was missing from the summary.Tor: 0.4.0.x-finalTaylor YuTaylor Yu