After log sanitization, proxy-go didn't restart, broker isn't logging

added component::circumvention/snowflake owner::cohosh priority::medium resolution::fixed severity::normal status::closed type::defect labels

Looking into this... it looks like since the update for #30125 (moved) there are no new logs being created for the broker or proxy-go instances.

Trac:
Owner: N/A to cohosh
Status: new to assigned

I've rolled back the versions of both proxy-go and broker to commit 88f282c with no luck. I think something about the log rotations in https://trac.torproject.org/projects/tor/ticket/30125#comment:8 are causing problems.

Okay not sure if this was causing the weird behaviour but the snowflake broker wasn't logging. The following additional commands fixed this:

$ sv d snowflake-broker/log
$ sv start snowflake-broker/log
ok: run: snowflake-broker/log: (pid 17524) 0s

Trying this for the proxy-go instances now (there's one running)

Errors from runit appear in the argv of the main runsvdir process, which you can see with ps axwww:

runsvdir -P /etc/service log: alone-restartless.log.d: file does not exist svlogd: fatal: no functional log directories. svlogd: pausing: unable to rename current: /home/snowflake-proxy/snowflake-proxy-standalone-23h.log.d: file does not exist svlogd: warning: unable to open log directory: /home/snowflake-proxy/snowflake-proxy-standalone-restartless.log.d: file does not exist svlogd: fatal: no functional log directories. 

/home/snowflake-proxy/snowflake-proxy-standalone-restartless.log.d is actually missing, but I'm not sure about the errors about "current" not existing.

Looking at the status of the processes gives me an error I don't know how to interpret:

# sv status snowflake-proxy-standalone-17h/log
run: snowflake-proxy-standalone-17h/log: (pid 1372) 4940035s, got TERM

But a kill and a restart seems to have gotten that one working again:

# sv kill snowflake-proxy-standalone-17h/log
# sv restart snowflake-proxy-standalone-17h/log
ok: run: snowflake-proxy-standalone-17h/log: (pid 17809) 0s
# sv status snowflake-proxy-standalone-17h/log
run: snowflake-proxy-standalone-17h/log: (pid 17809) 156s

I forgot that the svlogd log processes have to be stopped and started separately when I wrote the instructions in comment:4:ticket:30125.

Replying to dcf:

Looking at the status of the processes gives me an error I don't know how to interpret: {{{

sv status snowflake-proxy-standalone-17h/log

run: snowflake-proxy-standalone-17h/log: (pid 1372) 4940035s, got TERM }}} But a kill and a restart seems to have gotten that one working again: {{{

sv kill snowflake-proxy-standalone-17h/log

sv restart snowflake-proxy-standalone-17h/log

ok: run: snowflake-proxy-standalone-17h/log: (pid 17809) 0s

sv status snowflake-proxy-standalone-17h/log

run: snowflake-proxy-standalone-17h/log: (pid 17809) 156s }}}

I forgot that the svlogd log processes have to be stopped and started separately when I wrote the instructions in comment:4:ticket:30125.

Awesome, thanks! Setting this as fixed.

Trac:
Resolution: N/A to fixed
Status: assigned to closed

I restarted the instances at these times:

• snowflake-proxy-standalone-17h 2019-04-16 22:23
• snowflake-proxy-standalone-23h 2019-04-16 22:58
• snowflake-proxy-standalone-29h 2019-04-16 22:58

Additionally, I created a log directory for snowflake-proxy-standalone-restartless and started it at 2019-04-16 23:14.

Trac:
Summary: Snowflake is not working again to After log sanitization, proxy-go didn't restart, broker isn't logging

Replying to dcf:

I restarted the instances at these times:

• snowflake-proxy-standalone-17h 2019-04-16 22:23
• snowflake-proxy-standalone-23h 2019-04-16 22:58
• snowflake-proxy-standalone-29h 2019-04-16 22:58

Additionally, I created a log directory for snowflake-proxy-standalone-restartless and started it at 2019-04-16 23:14.

On second thought, the existing snowflake-proxy-restartless was already using the standalone broker. I'm not sure where snowflake-proxy-standalone-restartless came from, but I removed it from /etc/service so it doesn't run.

That's my bad, I started the snowflake-proxy-standalone-restartless but we can use snowflake-proxy-restartless for now.

I will probably make another ticket for renaming the proxy-go instances at some point since we shouldn't need to have them restart anymore.

closed

moved to tpo/anti-censorship/pluggable-transports/snowflake#30205 (closed)