Probetest(NAT Type Testing Assist Server) Load Balancing
Probetest(NAT Type Testing Assist Server) Load Balancing
The probetest server is presumably overloaded as a result of the recent increase of proxy. The unknown NAT type has increased in an unexpected way. We are currently seeking a way to load balance this server to increase its capacity. Steps identified: (On a secondary server)
- Setup ACME with nginx.
- Setup clustered probetest
- Test deployment on the standalone proxy
- Test deployment on the browser proxy (On primary server)
- Replicate secondary server setup
IRC logs:
<shelikhoo> cohosh, meskio: probetest is crashed again?
<+cohosh> shelikhoo: not sure, the metrics look stable from the last few days but it's possible
<+cohosh> we have a pretty high number of unknown NATs overall
<shelikhoo> I have tried in my environment and I see a HTTP response from NAT Type probetest
<shelikhoo> so it is probably working
<+cohosh> what made you ask?
<shelikhoo> [anti-censorship-alerts] [FIRING:1] Snowflake probetest failures
<shelikhoo> There is such an alert...
<+cohosh> ah i think i missed that
<+cohosh> i wonder if it's just overloaded
<shelikhoo> it would be possible to load balance it, as there is no state in that
<meskio> looking at the 7 days graph it looks like the unknown nat proxies has grown from ~10% to ~50% in the last 7 days
<meskio> but the other kind of nats hasn't gone down
<meskio> maybe probetest is in it's limit and is not handling well the increase of snowflakes or there is something weird there
<meskio> actually going back a bit longer in the graph there is a huge increase of proxies since the begining of the month
<meskio> something like x10 the number of proxies
<meskio> we might need to load balance probetest
<shelikhoo> Yes...
<+cohosh> yeah that sounds like a good idea
<meskio> shelikhoo: should we open an issue about that and ping dcf? do you want to do it? I guess you have a better idea of the details of what is needed than me
<+cohosh> i don't think we need to ping dcf here?
<meskio> I thought probetest was run by dcf
<shelikhoo> Yes. I will open a ticket for this.
<+cohosh> it's run on the same machine as the snowflake broker
<+cohosh> hmm so yeah just a heads up on a potential change to reverse proxy settings is good