TPA team issueshttps://gitlab.torproject.org/tpo/tpa/team/-/issues2020-06-27T14:16:59Zhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32383retire build-arm-* raspi boxes2020-06-27T14:16:59Zanarcatretire build-arm-* raspi boxesthere are three boxes in our infra that are just too slow to provide the service they were designed for. they are the `build-arm-0[123].torproject.org` boxes and should be retired.there are three boxes in our infra that are just too slow to provide the service they were designed for. they are the `build-arm-0[123].torproject.org` boxes and should be retired.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32419Add ahf to git team LDAP groups and provide access to the GitHub admin account2020-06-27T14:16:59ZirlAdd ahf to git team LDAP groups and provide access to the GitHub admin account```
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
For trac.torproject.org, 2019-11-07, afternoon:
Please add ahf to the LDAP groups: gitweb and gitolite.
Please also give ahf access to the external-services-git passwords in
tor-pass...```
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
For trac.torproject.org, 2019-11-07, afternoon:
Please add ahf to the LDAP groups: gitweb and gitolite.
Please also give ahf access to the external-services-git passwords in
tor-passwords.git by re-encrypting that file.
I would believe that the fingerprint for ahf's GPG key is:
1C2A7A3D5A8548B4ADEFD52AF9BC2FE22B08CE8F
though I have not verified this myself, enough people I trust to verify it that
I have verified myself have.
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEE/ps4MHN0fw3t6AudF4mIfdjVvF0FAl3ERM0ACgkQF4mIfdjV
vF1Evw/+MVT1gbP3Pk7K9xUhQxq4z6sWfK5EEuL/9ehdkJ940n67hfIZPoxItcgz
0WhkesqQ/SGcGwIhjzH8WvNBxiRywemM/A0NvF4dk61487b2p5gJXkCky3fVaGLP
d1mc92d+4+PoI+6Fd3/giFbMJX9ZYEWT3nAqyz9eZBsXK/OIJkH12gug9IgqUVpX
G7IySc5qQFTWALhHl5c4ype2TJefYkVgUpYER1R46+DS3Qb5o+u2lgTCzpfpZY6x
PqIWtjZdBeCDgE57bdnkU1bXjIz5CgVqV8+xb0nYsfJKKCJWBjdLY7T9Pk7iqsUX
iz0ekj/ObvAPCpV/dooA5rqyn0NMG9w/0nJmuuBtNPhbJzJgQ3csInB5VHtHbcCb
irbrDZ0vETQWFgvZIFMIDtj2vXQA72Q9oD0tLsfJcDk8wtJSmQHjlyUrpUM0UYI2
WBvFdKFSiq49IcDJcQywF65seBhPq+F3tj3IN2Xf163uLNkpNcGn1SNCboEXq9Su
0Ge7ymrGTtS4dIn9x0RSsUiXXeNxXVuIxCPZ8sGfMXCE16HA2oaRp3T6p3J29CmZ
aTY6VoOQCIgI2qfBm9vnNp7iAPbGZEVFBp6YYoXdQvAfPJEWBbU2Z7OQwptYKtBP
h1HzUwQ5KuGhnX7T9FGdV/XOOFGqYgpEkiTyhQSCPiWuFP6tJGw=
=KKdn
-----END PGP SIGNATURE-----
```anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32421Add ahf to git team mail aliases2020-06-27T14:16:58ZirlAdd ahf to git team mail aliasesThere is at least one mail alias for the git team, and then I think we use + addresses for different services. If there are aliases that have at least me and hiro on them, please add ahf to that list.There is at least one mail alias for the git team, and then I think we use + addresses for different services. If there are aliases that have at least me and hiro on them, please add ahf to that list.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32424Make digitalocean@ alias2020-06-27T14:16:58ZAlexander Færøyahf@torproject.orgMake digitalocean@ aliasHello,
David Goulet have gotten us some monthly credit on DigitalOcean to run a CI builder or two for our Gitlab setup.
I would like to create an account there, but I'd like it to be with an alias such that other people can get access ...Hello,
David Goulet have gotten us some monthly credit on DigitalOcean to run a CI builder or two for our Gitlab setup.
I would like to create an account there, but I'd like it to be with an alias such that other people can get access to the system too, in case there is some issue.
I would like to have: ahf, hiro, and gaba added to the alias, if possible please :-)
(another name than digitalocean@ is fine for me as well)anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32457Give cohosh and phw access to gettor-012020-06-27T14:16:58ZCecylia BocovichGive cohosh and phw access to gettor-01For debugging, updates, and maintenance it would be good for a few of us to have access to the machine that gettor is deployed on.For debugging, updates, and maintenance it would be good for a few of us to have access to the machine that gettor is deployed on.HiroHirohttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32532Install ZNC on Chives, make pastly admin it2020-06-27T14:16:57ZpastlyInstall ZNC on Chives, make pastly admin itI think I want to migrate the TPO people who use my bouncer off my server and onto TPO infra. If possible.
Initial discussion with anarcat suggested that chives.tpo would be the box. Okay cool.
**Q1**: Can it get a valid TLS certificat...I think I want to migrate the TPO people who use my bouncer off my server and onto TPO infra. If possible.
Initial discussion with anarcat suggested that chives.tpo would be the box. Okay cool.
**Q1**: Can it get a valid TLS certificate? Both for the web interface (**edit** for account management, NOT CHAT) and also for protecting the IRC traffic.
**Q2**: Can Tor get installed on the box? Right now I also have an onion service pointing to my ZNC and it'd be cool to keep that.
If desired, I can talk more about how I have accomplished Q1 with Let's Encrypt, nginx, and a cron job. Q2 is just because it's easy and cool. No big deal.pastlypastlyhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32553Group details for exit scanner2020-06-27T14:16:57ZirlGroup details for exit scannerCan you let me know what existing LDAP groups exist relating to the exit scanner (TorDNSEL) service on chiwui.torproject.org, and who is in them?Can you let me know what existing LDAP groups exist relating to the exit scanner (TorDNSEL) service on chiwui.torproject.org, and who is in them?anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32679Create VM to run monitoring software for anti-censorship team2020-06-27T14:16:55ZPhilipp Winterphw@torproject.orgCreate VM to run monitoring software for anti-censorship teamSo far, the anti-censorship team's infrastructure is monitored by a sysmon instance that gman999 generously runs for us. Every five minutes, sysmon establishes TCP connections to a number of machines and if any of these checks fails twic...So far, the anti-censorship team's infrastructure is monitored by a sysmon instance that gman999 generously runs for us. Every five minutes, sysmon establishes TCP connections to a number of machines and if any of these checks fails twice, we get an email alert.
The problem is that we cannot directly edit its configuration file, so we email gman999 whenever it needs an update. I would like to avoid this friction. Besides, sysmon is very simple and cannot handle, say, HTTP redirects.
I think it would be best for the anti-censorship team to run its own monitoring service, on a dedicated VM. We can then add monitoring targets ourselves and don't need to block on others.
I have been experimenting with a service called [monit](https://mmonit.com/monit/). It's free software and lightweight, yet flexible enough to fulfill our needs. I think it would be helpful to run monit on a dedicated VM. Does this make sense?https://gitlab.torproject.org/tpo/tpa/team/-/issues/32681Please refresh PGP key2020-06-27T14:16:55ZMatthew FinkelPlease refresh PGP keyPlease refresh my pgp key: `0xCB8FC772D1AA1D30`.
I'll attach it, as well.
Thanks.Please refresh my pgp key: `0xCB8FC772D1AA1D30`.
I'll attach it, as well.
Thanks.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32735Please remove my access to staticiforme and ability to push Tor Browser relea...2020-06-27T14:16:54ZGeorg KoppenPlease remove my access to staticiforme and ability to push Tor Browser releases and add sysrqb instead```
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
I currently have access to staticiforme and am able to get Tor Browser releases
out. That's not needed anymore as I am not involved anymore in pushing
Tor Browser releases live. Instea...```
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
I currently have access to staticiforme and am able to get Tor Browser releases
out. That's not needed anymore as I am not involved anymore in pushing
Tor Browser releases live. Instead, please add sysrqb to the list of
people who are able to access staticiforme and get Tor Browser releases out (I
think the latter involves being part of the tb-release group but I am not exactly
sure).
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEE1LbTp/1VD4Q+DIDnfkOLWrkXMD4FAl3yHpQACgkQfkOLWrkX
MD5zlA/+Jbf/TTKqn8G9HEWyEqaPWfJFfnk01uj1pn1abXqw1Va6NywOtKneba8/
F7XEZEIuPIB2kz44Ru64Z7OKUtTEUbOsNXBPMtMld438xIPx2Xh4QXEymajzPo1W
gzRiKL0BZY1+wmF0BlwoS5QzgtEfNiO7HJGELexnjfm299T89lYp3LHp5joS1vFv
QZllzC4OrHp1CVYnL9/o7vPij4ZEKS0MNHYQ2LgYwpbyjCcluCmiMmLhtdwH7eZD
lX6CEzu59fhyHGiQ9Nn3CT3A1zQglUBVtbJ7kwEgJQyvotHVPRqp2jhzJklLL+zV
b0nMM1lqeoQAEIx3SI24BOQwnBfEkopprN7qNI/86Fjfavi5hJNadpHbWVBLrJn8
bub7Gvg/2qLnus6oAWV7VWEbohfcq2p8s5lUCVVGatJyMTN/3/oBzjJK5CRW4Tyr
9b5cIKM+9HWGm+ZYK1J8CxP2RyR+jeQG/pqB0v0UP70ixhCbMKQGMGmEUQacUAlN
BsDB8tdW5k8VWZPjWk1YRkWCHGDK/K80yG+tpq6UrmLc46KNfcfmXe1ds1UnPXfL
EPSBJxCQLKMcftG7T1w+73B38flxNdx5mB9aVx5DV8+/2EKzfcaTRW8WRgZ5Nmjj
gs/2BJJbBuW9InS5CG8JNc7QjwBy2MQX6Eq5IASQrkjWp19CPq0=
=YS46
-----END PGP SIGNATURE-----
```anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32762major networking issues on moly, affects: majus, fallax, web-cymru-01, build-...2020-06-27T14:16:53Zemmapeelmajor networking issues on moly, affects: majus, fallax, web-cymru-01, build-x86-05, build-x86-06hello:
something happened in https://db.torproject.org/machines.cgi?host=majus the translation server.
last night the last update happened, around 2019-12-15 23:55:01
and since then the translations are not getting updated.
i have ...hello:
something happened in https://db.torproject.org/machines.cgi?host=majus the translation server.
last night the last update happened, around 2019-12-15 23:55:01
and since then the translations are not getting updated.
i have entered on the server and i can ping to google.com, but i could not git clone https://github.com/transifex/transifex-client.git repository-git
and when i run the script at /srv/translation.torproject.org/tools/update_translations it just stays there waiting and cannot connect.
is it possible that some network functions cannot be performed?
the scripts and such are run under the translations user
please have a look as this is a pretty important repo for updates in translations and for the translators to review their work as well.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32763Please provision an onionoo-backend-02 and add to the varnish configuration2020-06-27T14:16:53ZirlPlease provision an onionoo-backend-02 and add to the varnish configurationThis should be pretty identical to onionoo-backend-01, ideally arranged such that hardware failure won't kill both instances at once, but with a different hostname.This should be pretty identical to onionoo-backend-01, ideally arranged such that hardware failure won't kill both instances at once, but with a different hostname.https://gitlab.torproject.org/tpo/tpa/team/-/issues/32782Add google search DNS verification TXT record for ooni.torproject.org2020-06-27T14:16:53ZArturo FilastòAdd google search DNS verification TXT record for ooni.torproject.orgFollowing https://trac.torproject.org/projects/tor/ticket/31718 we would like to start setting up 301 redirects from ooni.torproject.org to ooni.org, so that we can improve our SEO a bit (see: https://github.com/ooni/ooni.org/issues/314)...Following https://trac.torproject.org/projects/tor/ticket/31718 we would like to start setting up 301 redirects from ooni.torproject.org to ooni.org, so that we can improve our SEO a bit (see: https://github.com/ooni/ooni.org/issues/314).
In order to get the google search data for ooni.torproject.org we need to prove ownership of the domain name via DNS.
This requires adding the following TXT record for the ooni.torproject.org domain:
```
google-site-verification=8q628kou8EXLTU-ggciHl7E4tYvwLqiapH2kdJoO3vk
```anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32787Delete tb-crashes role2020-06-27T14:16:53ZMatthew FinkelDelete tb-crashes roleThe tb-crashes role was created in legacy/trac#22923 for creating a pair of (mini-breakpad) servers where crash reports would be sent and viewed. That server was decommissioned in legacy/trac#31095. Although legacy/trac#23624 is still op...The tb-crashes role was created in legacy/trac#22923 for creating a pair of (mini-breakpad) servers where crash reports would be sent and viewed. That server was decommissioned in legacy/trac#31095. Although legacy/trac#23624 is still open, and this would be a really nice improvement, I don't see us getting back to this in the near future.
I think we should delete the tb-crashes role for now, and we can recreate it in the future if/when we get back to this.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32800Creating some space to host Tor Browser nightly updates2020-06-27T14:16:53ZboklmCreating some space to host Tor Browser nightly updatesWe need a space to upload and make available the Tor Browser nightly updates, for legacy/trac#18867.
In the past we had https://nightlies.tbb.torproject.org/, created in legacy/trac#24921. However it seems it doesn't exist anymore.
We ...We need a space to upload and make available the Tor Browser nightly updates, for legacy/trac#18867.
In the past we had https://nightlies.tbb.torproject.org/, created in legacy/trac#24921. However it seems it doesn't exist anymore.
We could reuse the same name, https://nightlies.tbb.torproject.org/. An .onion address would be nice too.
Space needed will be ~10G in the beginning (starting with only 5 locales, but we'll maybe want to increase that number in the future).https://gitlab.torproject.org/tpo/tpa/team/-/issues/32801major outage: kvm4 down, affected: eugeni (mail, lists), alberti (ldap), paul...2020-06-27T14:16:53Zanarcatmajor outage: kvm4 down, affected: eugeni (mail, lists), alberti (ldap), pauli (puppet), rouyi (jenkins), etcDuring a security reboot today, kvm4.torproject.org did not return. All virtual machines on this host are down and unavailable.
According to the Nextcloud spreadsheet (since LDAP is down), that includes:
| host | service ...During a security reboot today, kvm4.torproject.org did not return. All virtual machines on this host are down and unavailable.
According to the Nextcloud spreadsheet (since LDAP is down), that includes:
| host | service | impact | mitigation |
|----------------|-------------------------|--------|------------|
| alberti | LDAP, db.tpo | critical, no passwd change | read-only copies everywhere |
| build-x86-09 | buildbox | redundant | N/A |
| eugeni | incoming mail, lists | critical, total outage | peek at `tor-puppet/modules/postfix/files/virtual` and email people directly |
| meronense | metrics.tpo | critical, total outage | ? |
| neriniflorum | DNS | redundant, higher TTFB? | possible to remove from rotation |
| oo-hetzner-03 | onionoo | redundant | ? |
| pauli | puppet | major, no config management | use `cumin`, local git copies |
| rouyi | jenkins | critical, total outage | ? |
| web-hetzner-01 | web mirror | redundant, no effect? | removed from rotation automatically |
| weissi | build box | no windows builds | N/A |
| woronowii | build box | no windows builds | N/A |
I'll note that it seems both windows build boxes are on the same machine so even if jenkins *would* be able to dispatch builds, we wouldn't be able to do those...
A ticket was filed with Hetzner to try and rescue the server.
Our disaster recover plan so far is to wait for that rescue to succeed, which might take up to 24h but hopefully less.
If that fails, I would suggest the following plan:
1. recover eugeni, pauli, alberti from backups on gnt-fsn or elsewhere (we need those three to build new machines)
2. build a new ganeti cluster (because we can't recover all of this on gnt-fsn)
3. restore remaining machines on the new cluster
4. decommission kvm4 officially
This could take a few days of work. :(HiroHirohttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32802retire kvm4, 8 VMs to migrate2020-06-27T14:16:53Zanarcatretire kvm4, 8 VMs to migratekvm4 is getting fairly old. it's been setup in 2015 and is showing sign of old age. for example today it freaked us all out by not returning after a reboot right before the holidays (legacy/trac#32801). considering how critical that serv...kvm4 is getting fairly old. it's been setup in 2015 and is showing sign of old age. for example today it freaked us all out by not returning after a reboot right before the holidays (legacy/trac#32801). considering how critical that server is (email, puppet, ldap, jenkins, dns, web mirror, all the windows buildboxes!) we should start considering a decomissionning process.
at the very least, we need to get eugeni the heck out of there.
we have budget to provision another ganeti cluster, so let's use it to replace this, and hopefully more. the existing cluster has already taken more than its share by taking machines from both kvm1/textile and moly, so it's time we provision more hardware for this.
this requires a new ganeti node (fsn-node-06, legacy/trac#33907).
machines to be migrated:
* [x] alberti.torproject.org (LDAP) legacy/trac#33908
* [x] build-x86-09.torproject.org (build server) - RETIRE - replaced with build-x86-11 on gnt-fsn
* [x] eugeni.torproject.org (email) legacy/trac#32803
* [x] meronense.torproject.org (metrics) legacy/trac#33909
* [x] neriniflorum.torproject.org (DNS) legacy/trac#33910
* [x] oo-hetzner-03.torproject.org (metrics) legacy/trac#33911
* [x] pauli.torproject.org (puppet) legacy/trac#33912
* [x] rouyi.torproject.org (jenkins) legacy/trac#33913
* [x] web-hetzner-01.torproject.org (static mirror) RETIRE
* [x] weissii.torproject.org (windows build box) legacy/trac#33914
* [x] winklerianum.torproject.org (windows build box) turned off, migrated along weissii, but turned off
* [x] woronowii.torproject.org (windows build box) turned off, migrated along weissii, but turned offanarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/32803migrate eugeni to the gnt-fsn cluster2020-06-27T14:16:52Zanarcatmigrate eugeni to the gnt-fsn clusterThis is a big decision, but we might want to consider migrating eugeni out of kvm4 in the short term.
This server is highly critical: it handles incoming mail and mailing lists, and any downtime on that server is felt quickly by everyon...This is a big decision, but we might want to consider migrating eugeni out of kvm4 in the short term.
This server is highly critical: it handles incoming mail and mailing lists, and any downtime on that server is felt quickly by everyone in the Tor project.
Even though the gnt-fsn server is over-provisionned, it is technically more reliable than a single machine like kvm4 because it has redundant nodes. We might be able to afford the extra space to host eugeni right now.
The risk, of course, is that eugeni is an old server that is very important infrastructure. Migrating it might be tricky. For example its IP address might be hardcoded in a bunch of places, and moving it to the gnt-fsn will mean changing that IP address.
Such an IP change might also affect its reputation with other mail servers, making mail delivery even harder than it already is.
But we have to cross that bridge anyways with the new SMTP delivery service in legacy/trac#30608.weasel (Peter Palfrader)weasel (Peter Palfrader)https://gitlab.torproject.org/tpo/tpa/team/-/issues/32834Hetzner fault affecting hetzner-hel1-03.torproject.org2020-06-27T14:16:52ZHiroHetzner fault affecting hetzner-hel1-03.torproject.org
hetzner-hel1-03.torproject.org is currently down.
I have checked on Hetzner and it seems there is an issue with the node where this VM is hosted.
Type: Fault report
Categories: Cloud
Start: December 21, 2019 6:27:...
hetzner-hel1-03.torproject.org is currently down.
I have checked on Hetzner and it seems there is an issue with the node where this VM is hosted.
Type: Fault report
Categories: Cloud
Start: December 21, 2019 6:27:00 PM CET
End: Unknown
Description: Due to a current error, the cloud node (10567) and the cloud servers on it are not accessible.
Affected: Cloud servers running on cloud nodes 10567.
Details: You can check your cloud server on Cloud Console to see if you're affected.
hetzner-hel1-03.torproject.org is a webserver so this issue isn't critical. Hopefully it is resolved soon.https://gitlab.torproject.org/tpo/tpa/team/-/issues/32840Issue with cloud.ipnett.se2020-06-27T14:16:52ZHiroIssue with cloud.ipnett.seWe have all the VM from sunet down at the moment.
This is a problem with sunet. Open issue: http://www.nunoc.org/nunocweb/ticket.php?key=SUNETTICKET-6078
Current hosts down:
orestis.torproject.org -> onionoo.torproject.org
cdn-backen...We have all the VM from sunet down at the moment.
This is a problem with sunet. Open issue: http://www.nunoc.org/nunocweb/ticket.php?key=SUNETTICKET-6078
Current hosts down:
orestis.torproject.org -> onionoo.torproject.org
cdn-backend-sunet-01.torproject.org -> static content backend
corsicum.torproject.org -> collector.torproject.org
nutans.torproject.org -> DNS
Because all the services affected have redundancy, this shouldn't be a critical issue.