Update DNS records for .ooni.torproject.org domains
To make it easier for us to manage where these domains point to it would be great if the records for the domain explorer.ooni.torproject.org were to point to explorer.ooni.io and the record for ooni.torproject.org pointed to ooni.io.
The most high priority is the update of explorer.ooni.torproject.org as we are launching that today and we still have places where we link to explorer.ooni.torproject.org instead of explorer.ooni.io.
Activity
Another option, if we don't like having .torproject.org sites running on non-TPA machines, would be to run a tiny webserver as the .torproject.org site, which sends an http-level redirect to the external site?
I mention it because that http redirect is happening now already, just on the remote site.
hellais, do you want an actual CNAME (ie. that the user doesn't know they get redirected to ooni) or a redirect (that the user does end up on ooni.io)?
i do agree that it's unconventional to do those things for us. we usually point *.torproject.net at external resources.
is this domain used by non-HTTP clients?
Currently this is already happening though,
explorer.ooni.torproject.orgbeing a CNAME toexplorer.ooni.io, I mistakenly thought this was not currently the case when opening the ticket, but this is already happening and no change is necessary on this front.It was done this way specifically to make it easier for us to more independently handle how we serve requests to users hitting out website from the various domains that were distributed.
The website
ooni.org&ooni.io&ooni.torproject.orgis still running on tpo infrastructure, but we would like to change that to reduce the complexity of having something hosted on system where people need LDAP access to administer it.Our preference would be that we setup a CNAME record for
ooni.torproject.orgthat points toooni.ioorooni.orgso that we are able to on our own setup a redirect, if desirable, or handle the requests directly by keeping theooni.torproject.orgdomain (we probably will do this in the beginning).would be to run a tiny webserver as the .torproject.org site, which sends an http-level redirect to the external site?
hellais, do you want an actual CNAME (ie. that the user doesn't know they get redirected to ooni) or a redirect (that the user does end up on ooni.io)?
It would be preferable if we could get a CNAME record, so that we can manage how redirects are handled autonomously.
is this domain used by non-HTTP clients?
It's the domain used for our primary website. We don't make any assumption as to what type of client is going to access it. I suppose most modern browsers will do HTTPS.
-
Another record which is currently setup in a similar fashion is the
CNAMEforget.ooni.torproject.org. 3599 IN CNAME get.ooni.io.To be clear it's not a big problem if the policy WRT to setting up CNAME records has changed, I just need to be aware of it and plan according to it.
This is probably also a good opportunity to do some cleanup of other
*.ooni.torproject.orgdomains as we are trying to simplify our infrastructure and reduce our devops cognitive load by simplifying our infrastructure. -
i don't exactly know what the policy is regarding CNAMEs, to be honest. :) the best source I know of is this:
https://help.torproject.org/tsa/doc/naming-scheme/
... which outlines the distinction between TPO (torproject.org) and TPN (torproject.net) that weasel was refering to. The problem might not be CNAMEs per se, but pointing to outside stuff.
Another thing is that CNAMEs are not a great way to move stuff around, because they are transparent to clients. An web browser or crawler will not treat a CNAME as "this is now hosted over there", it's just an alias. For those kind of transitions, you want to do a HTTP redirect, that is respond with a 301 (Moved Permanently) or 302 (Found) status code:
https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_Redirection
Then we can deprecate the *.ooni.tpo namespace and eventually transition to ooni.io cleanly.
This is why I was asking about non-HTTP (and non-HTTPS) clients: those redirections will work only for HTTP clients. If you have people using this over SSH or Git or whatever non-HTTP protocol, those would break of course.
(Sorry if you already know all of this about HTTP status codes vs CNAMEs, but I thought it was useful to get back to the specs to clarify my thoughts.)
-
seems to me that just adding the CNAME will not be enough, as there are many other things to cleanup. the main procedure should be:
- remove
ooni.torproject.orgfromtor-puppet/modules/roles/misc/static-components.yaml - remove ooni from auto-dns, push
- add the CNAME, push
Other things to cleanup include:
letsencrypt-domains/domains:46:ooni.torproject.org tor-nagios/config/nagios-master.cfg:1330: name: mirror static sync - ooni tor-nagios/config/nagios-master.cfg:1331: check: "dsa_check_staticsync!ooni.torproject.org" tor-puppet/modules/sudo/files/sudoers:63:%ooni STATICMASTER=(ooni) ALL tor-puppet/modules/sudo/files/sudoers:95:%ooni STATICMASTER=(mirroradm) NOPASSWD: /usr/local/bin/static-master-update-component ooni.torproject.org, /usr/local/bin/static-update-component ooni.torproject.org tor-puppet/modules/roles/manifests/static_mirror_web.pp:74: ssl::service { 'ooni.torproject.org': ensure => 'ifstatic', notify => Exec['service apache2 reload'], key => true, } tor-puppet/modules/roles/manifests/static_mirror_onion.pp:37: 'ooni.torproject.org', tor-puppet/onions/onionbalance-services.yaml:17: [...]I'm particularly concerned about let's encrypt - wouldn't adding the cname break the X509 cert, as we would now point to another server?
- remove
-
So we looked into this with @anarcat and encountered the following issues:
- The current setup has both HSTS and certificate pinning enabled for the ooni.torproject.org website
- It is not straightforward to do custom HTTPS changes on the current ooni hosting service (netlify)
Since the maxage for the certificate pinning is set to 60 days we will need to wait for that amount of time before we are able to migrate over.
In the meantime @anarcat is going to see how to disable the certificate pinning headers from the ooni.torproject.org host config, so that we can begin waiting the 60 days after which we can proceed with the CNAME plan as mentioned above.
-
i have disabled certificate pinning on ooni.torproject.org around 15 minutes ago. it should therefore expire in 60 days exactly, which is about on saturday november 16th at 19:30UTC. assuming we don't want to do this transition on a saturday, we should probably look into this again on november 18th.
i documented a bit how HPKP works in:
-
Replying to anarcat:
we're getting ready for this transition again, which should happen some time next week.
Thanks for following up on this @anarcat!
I had marked on my calendar Nov 18th as the date we can do the migrateion
@hellais, are you around next week? should we carry on the plan as expected?
Yes I am going to be around and let's proceed as planned with doing the migration on Nov 18th. Does that work for you?
where should the CNAME point to?
ooni.org? `www.ooni.org?The CNAME should point to
ooni.organd in theory that would work with out netlify based host. I don't think I have ever done this, though, so it's useful if we are both online to coordinate on this in realtime.I am going to be mostly online on Nov 18th from ~10:00 UTC - 18:00 UTC. Should we try to meet online at around 16:00 UTC?
-
this has now been deployed, with the following three patches, in
dns/domains.git:From 471f529240673d324a66a1258f6acc257857f964 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org> Date: Mon, 18 Nov 2019 11:50:50 -0500 Subject: [PATCH] add ooni.tpo CNAME (#31718) --- torproject.org | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/torproject.org b/torproject.org index 8e62797..87623f1 100644 --- a/torproject.org +++ b/torproject.org @@ -100,6 +100,7 @@ rsync.media IN CNAME listera metrics IN CNAME meronense munin IN CNAME schmitzi nagios IN CNAME hetzner-hel1-01 +ooni IN CNAME ooni.io. get.ooni IN CNAME get.ooni.io. measurements.ooni IN CNAME measurements.ooni.io. explorer.ooni IN CNAME explorer.ooni.io. @@ -168,7 +169,6 @@ help IN CNAME static lektor-staging IN CNAME static newsletter IN CNAME static nyx IN CNAME static -; ooni A/AAAA records via services-auto openpgpkey IN CNAME static rbm IN CNAME static rpm IN CNAME static -- 2.20.1in
dns/auto-dns.git:From 7a1229bc1d0e4b92ee75712942eba146db9adee9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org> Date: Mon, 18 Nov 2019 11:48:59 -0500 Subject: [PATCH] retire ooni.tpo, will be a CNAME (#31718 --- services/ooni.torproject.org.service | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 services/ooni.torproject.org.service diff --git a/services/ooni.torproject.org.service b/services/ooni.torproject.org.service deleted file mode 100644 index ec2a1f2..0000000 --- a/services/ooni.torproject.org.service +++ /dev/null @@ -1,7 +0,0 @@ ---- -ttl: 150 -hosts: - default: - - hetzner-hel1-03.torproject.org - - listera.torproject.org -# vim:syn=yaml: -- 2.20.1... and
tor-puppet.git:From 9cc7af7889ba9b7fd9b167591c30e5baa395acf6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org> Date: Mon, 18 Nov 2019 11:44:10 -0500 Subject: [PATCH] retire ooni.tpo, will be a CNAME (#31718) --- modules/roles/misc/static-components.yaml | 3 --- 1 file changed, 3 deletions(-) diff --git a/modules/roles/misc/static-components.yaml b/modules/roles/misc/static-components.yaml index 9151c5f6..a810c5bd 100644 --- a/modules/roles/misc/static-components.yaml +++ b/modules/roles/misc/static-components.yaml @@ -47,9 +47,6 @@ components: help.torproject.org: master: staticiforme.torproject.org source: staticiforme.torproject.org:/srv/help-master.torproject.org/output - ooni.torproject.org: - master: staticiforme.torproject.org - source: staticiforme.torproject.org:/home/ooni/website openpgpkey.torproject.org: master: staticiforme.torproject.org source: alberti.torproject.org:/srv/db.torproject.org/keyrings/openpgpkey -- 2.20.1@hellais renewed the domain with netlify and the new site seems to be online and working.
i still have my own cleanup to do, but the synchronous, "OMG IS THIS GOING TO WORK" step is over, i believe.
Trac:
Status: assigned to accepted -
i removed the nagios check and let's encrypt cert, then also cleaned this up in puppet:
From b8e3ebc8f10c9b2e6654c84e85291c277b861637 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org> Date: Mon, 18 Nov 2019 12:08:12 -0500 Subject: [PATCH] remove remaining traces of ooni.tpo mirror (#31718) --- modules/roles/manifests/static_mirror_onion.pp | 3 ++- modules/roles/manifests/static_mirror_web.pp | 2 +- .../roles/templates/static-mirroring/vhost/static-vhosts.erb | 1 - modules/sudo/files/sudoers | 2 -- 4 files changed, 3 insertions(+), 5 deletions(-) diff --git a/modules/roles/manifests/static_mirror_onion.pp b/modules/roles/manifests/static_mirror_onion.pp index d9c15fce..706783cd 100644 --- a/modules/roles/manifests/static_mirror_onion.pp +++ b/modules/roles/manifests/static_mirror_onion.pp @@ -34,7 +34,6 @@ class roles::static_mirror_onion { 'nyx.torproject.org', 'onion.torproject.org', 'onionperf.torproject.org', - 'ooni.torproject.org', 'openpgpkey.torproject.org', 'rbm.torproject.org', 'research.torproject.org', @@ -56,5 +55,7 @@ class roles::static_mirror_onion { ensure => 'ifstatic'; 'spec.torproject.org': ensure => 'present'; + 'ooni.torproject.org': + ensure => 'absent'; } } diff --git a/modules/roles/manifests/static_mirror_web.pp b/modules/roles/manifests/static_mirror_web.pp index 997140b7..73859c41 100644 --- a/modules/roles/manifests/static_mirror_web.pp +++ b/modules/roles/manifests/static_mirror_web.pp @@ -65,7 +65,7 @@ class roles::static_mirror_web { ssl::service { 'nyx.torproject.org': ensure => 'ifstatic', notify => Exec['service apache2 reload'], key => true, } ssl::service { 'onion.torproject.org': ensure => 'ifstatic', notify => Exec['service apache2 reload'], key => true, } ssl::service { 'onionperf.torproject.org': ensure => 'ifstatic', notify => Exec['service apache2 reload'], key => true, } - ssl::service { 'ooni.torproject.org': ensure => 'ifstatic', notify => Exec['service apache2 reload'], key => true, } + ssl::service { 'ooni.torproject.org': ensure => 'absent', notify => Exec['service apache2 reload'], key => true, } ssl::service { 'openpgpkey.torproject.org': ensure => 'ifstatic', notify => Exec['service apache2 reload'], key => true, } ssl::service { 'rbm.torproject.org': ensure => 'ifstatic', notify => Exec['service apache2 reload'], key => true, } ssl::service { 'research.torproject.org': ensure => 'ifstatic', notify => Exec['service apache2 reload'], key => true, } diff --git a/modules/roles/templates/static-mirroring/vhost/static-vhosts.erb b/modules/roles/templates/static-mirroring/vhost/static-vhosts.erb index a49d64b5..30fd426b 100644 --- a/modules/roles/templates/static-mirroring/vhost/static-vhosts.erb +++ b/modules/roles/templates/static-mirroring/vhost/static-vhosts.erb @@ -152,7 +152,6 @@ vhost(lines, "newsletter.torproject.org") vhost(lines, "nyx.torproject.org") vhost(lines, "onion.torproject.org") vhost(lines, "onionperf.torproject.org") -vhost(lines, "ooni.torproject.org") vhost(lines, "openpgpkey.torproject.org", :extra => true) vhost(lines, "rbm.torproject.org") vhost(lines, "research.torproject.org") diff --git a/modules/sudo/files/sudoers b/modules/sudo/files/sudoers index 39156276..90b2bcbc 100644 --- a/modules/sudo/files/sudoers +++ b/modules/sudo/files/sudoers @@ -59,7 +59,6 @@ letsencrypt nevii=(dnsadm) NOPASSWD: /srv/dns.torproject.org/bin/update %metrics meronense=(metrics) ALL %onionoo ONIONOOHOSTS=(onionoo) ALL %onionoo ONIONOOHOSTS=(onionoo-unpriv) ALL -%ooni STATICMASTER=(ooni) ALL %stem STATICMASTER=(stem) ALL %nyx STATICMASTER=(nyx) ALL %rtfolks rude=(rtstuff) ALL @@ -89,7 +88,6 @@ exonerator materculae=(exonerator-web) NOPASSWD: ALL %globe STATICMASTER=(mirroradm) NOPASSWD: /usr/local/bin/static-master-update-component globe.torproject.org, /usr/local/bin/static-update-component globe.torproject.org %consensus-health henryi=(mirroradm) NOPASSWD: /usr/local/bin/static-master-update-component consensus-health.torproject.org, /usr/local/bin/static-update-component consensus-health.torproject.org %torwww,%metrics STATICMASTER=(mirroradm) NOPASSWD: /usr/local/bin/static-master-update-component onionperf.torproject.org, /usr/local/bin/static-update-component onionperf.torproject.org -%ooni STATICMASTER=(mirroradm) NOPASSWD: /usr/local/bin/static-master-update-component ooni.torproject.org, /usr/local/bin/static-update-component ooni.torproject.org %snowflake STATICMASTER=(mirroradm) NOPASSWD: /usr/local/bin/static-master-update-component snowflake.torproject.org, /usr/local/bin/static-update-component snowflake.torproject.org %stem STATICMASTER=(mirroradm) NOPASSWD: /usr/local/bin/static-master-update-component stem.torproject.org, /usr/local/bin/static-update-component stem.torproject.org %nyx STATICMASTER=(mirroradm) NOPASSWD: /usr/local/bin/static-master-update-component nyx.torproject.org, /usr/local/bin/static-update-component stem.torproject.org -- 2.20.1finally, i need to do documentation and we need to decide if/when we do HTTP redirects instead of CNAMEs here to finalize this transition. but i guess that OONI can do those redirects themselves, when they want to as well...
-
i updated the documentation on how to remove a static component here:
https://help.torproject.org/tsa/howto/static-component/
the only thing remaining is to remove the user/group, and the actual files on staticiforme (and mirrors?)
@hellais, can i remove the actual site? how about the users? shouldn't i be removing users that were created just for the purpose of managing this website?
thanks!
Some of those users are not longer with Tor/OONI like darkk.
@anarcat, should we check on servers for old accounts that are no longer being use?
-
@anarcat the only ones you should keep of the above list are:
- art
- agrabeli
-
i should note that I created legacy/trac#32558 (moved) to followup on what happens with such email accounts after lockout.
-
it turns out locking out those users was probably a mistake, as some if not all of them are still on tor-internal. my mistake. i have restored their accesses, although I have lost their passwords for now. i'm looking in the backups to see if i can restore those hashes as well. what i specifically did was:
- restore keyFingerPrint (based on the account-keyring git repo)
- delete accountStatus
- delete shadowExpire
- remove the ooni group membership from all users
I'm trying to see if i can coerce our backup system to give us a view on those old hashes now.
moved from legacy/trac#31718 (moved)
added Bug label and removed 1 deleted label
added Sysadmin label and removed 1 deleted label
