polyanthum's (bridgedb's) apache needs some tuning?
The first hint, our monit on anti-censorship-alerts just sent us three alerts, of the form:
Description: failed protocol test [HTTP] at [bridges.torproject.org]:443/ [TCP/IP TLS] -- Connection timed out
The second hint, I tried to telnet to bridges.torproject.org 443 from my house and it worked but it took like 5 or 10 seconds to make the connection.
The third hint, I looked at the apache processes on polyanthum and there are 30+ of them. That actually doesn't seem like that many. I think they all just proxypass into various internal components like bridgedb?
The fourth hint, /var/log/apache2/error.log has lines like
[Sat Sep 24 04:26:10.566973 2022] [mpm_worker:error] [pid 249378:tid 139951308307776] AH00288: scoreboard is full, not at MaxRequestWorkers
[Sat Sep 24 04:26:15.571575 2022] [mpm_worker:error] [pid 249378:tid 139951308307776] AH00287: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting
[Sat Sep 24 04:26:20.576705 2022] [mpm_worker:error] [pid 249378:tid 139951308307776] AH00288: scoreboard is full, not at MaxRequestWorkers
Conclusion: can we figure out what cap is going on inside polyanthum's apache, and let it handle more proxypasses in parallel?
Bridgedb is popular this week and it looks like we're turning away people without needing to.
Thanks!
(Cc'ing @meskio, @cohosh, @shelikhoo, @dcf so they can follow along.)