move to a private nextcloud instance

Trac:
Child Ticket(s): #32391 (moved), #32390 (moved), #32367 (moved)

added component::internal services/service - nextcloud owner::ln5 priority::medium resolution::fixed severity::normal status::closed type::project labels

riseup asked me to push a few things around here to unblock the deployment. it seems the next step is to generate a HTTPS cert and, if we want it in our namespace, requires us to pick a domain name and point it at their server.

we need to answer the following questions:

CNAME or A/AAAA?
pointing at what?
what name?

I would suggest:

CNAME: so that riseup doesn't have to coordinate with us to move the box around and we clearly see who the hoster is
the box is merlebleu.riseup.net
i would suggest keeping with the nc convention, so nc.torproject.net

I'll do this after quick validation in today's vegas.

Trac:
Owner: N/A to anarcat
Status: new to assigned

Trac:
Cc: N/A to linus@torproject.org

also cloud.tp.n or docs.tp.n could work... just throwing it there...

vegas approved nc.tpn and suggested we make a redirect from nc.tpo to avoid confusion, will do next.

nc.torproject.net is now "live" in terms of DNS, next step is to setup the cert and so on from riseup's side.

i've also deployed a nc.torproject.org HTTP redirection to nc.tpn. to quote the DNS commitlog:

commit faf9e63fe52c2efb0771e662f76247b32f946ede
Author: Antoine Beaupré <anarcat@debian.org>
Date:   Thu Oct 24 13:58:11 2019 -0400

    add nc.torproject.org for nextcloud (#32267)
    
    This points to the static mirrors for redirection, using the "vanity
    hosts" macro.
    
    The rationale is that our users might not be familiar with the .net
    and .org distinction. Even if they do, they might not know we don't
    manage this machine directly. Instead, they will probably look for
    nc.tpo and fail and complain to us. So we should help people a little
    here and make an exception for this domain which would be possibly
    heavily used.
    
    This is especially relevant since we *might* eventually use `.org` for
    this purpose and manage the Nextcloud service ourselves.
    
    But we don't do just a CNAME to the end server here: we do a redirect
    so the canonical URL remains the `.net` one, to clearly show this is
    not managed by TPA or the service admin team.

diff --git a/torproject.org b/torproject.org
index 92d68b3..b49dc6f 100644
--- a/torproject.org
+++ b/torproject.org
@@ -149,6 +149,8 @@ bugs				IN	CNAME	static
 censorshipwiki			IN	CNAME	static
 safetyboard			IN	CNAME	static
 wiki				IN	CNAME	static
+; https://trac.torproject.org/projects/tor/ticket/32267
+nc				IN	CNAME	static
 
 ; static-mirrors
 aus1				IN	CNAME	static

linus, i think you're it again on this ticket, feel free to ping if you need help.

Trac:
Owner: anarcat to ln5

Trac:
Cc: linus@torproject.org to linus@torproject.org, gaba

As an update to this issue - the cert has been setup from the riseup side.

What remains now is to plug-in a couple pieces of the setup, and create the admin access for the people who will be managing this. I assume that will be the same people who are doing that already. Then it will be a matter of migration.

Same people, yes. Would you be able to migrate the admins, including the requirement for MFA and accompanying MFA info? If not, let's try to add only me in the least insecure way.

Also, micah, do you have experience with migrating NC data from one instance to another?

From the top of my head, we would have to bring

files
calendars
contacts
decks
circles over to the new instance.

This instance can be reached by gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion

Besides the base installation, this instance was configured with the U2F and TOTP apps enabled, and forced, Onlyoffice was enabled and configured. Outgoing/incoming email has also been configured.

This new version of nextcloud has these new features that the Nextcloud admins should be aware of:

You can setup two-factor authentication after first login, admins can create one-time login tokens in the web UI and delegate this to group admins/

Remote Wipe: For Nextcloud admins, this is good functionality to know about in order to better handle people departing the organization. https://www.youtube.com/watch?v=oyWXMjb-6ik

Collaborative text: The new nextcloud collaborative text editor comes as default now as well: https://www.youtube.com/watch?v=Nr7cGN6ZJM0

ln5: can you tell me which logins for admins I should migrate? I'm unsure which should be done. I'm also unsure how reliable migrating the login/MFA information will be, but we can try it for you. See if you can login. You will need to generate new backup codes

Regarding migrating data, I have no experience with migrating data from one instance to another, it does not look trivial and my research has not produced any mechanism to make it easy.

Since you mention circles, I wanted to check in about that. This app is kind of a 'hack' that lets unprivileged users create their own groupings. It was useful on the riseup server, because no tor people were admins and could not create groups. I'd recommend considering not using it on the new server, if possible, as its long term viability is questionable.

https://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/login

http://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/ redirects to https://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/login which fails, supposedly bc 443.

See if you can login. You will need to generate new backup codes

This, after entering TOTP authn code:

The server was unable to complete your request.

If this happens again, please send the technical details below to the server administrator.

More details can be found in the server log.
Technical details

    Remote Address: <redacted>
    Request ID: PKeWiwlMWjvZrlYItQvF

Replying to ln5:

http://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/ redirects to https://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/login which fails, supposedly bc 443.

I've resolved this

Replying to ln5:

See if you can login. You will need to generate new backup codes

This, after entering TOTP authn code:

{{{ The server was unable to complete your request.

If this happens again, please send the technical details below to the server administrator.

More details can be found in the server log. Technical details
Remote Address: <redacted>
Request ID: PKeWiwlMWjvZrlYItQvF
}}}

I believe that this is because the codes are connected to the domain of the application, so we won't be able to just move those over from the database.

I've removed the TOTP setting for your user, see if you can login without it and then enable it.

I've removed the TOTP setting for your user, see if you can login without it and then enable it.

Worked fine, thanks.

Micah, did you verify outgoing email? I was hoping to get a signup email when creating a new user but haven't seen one yet. Might be greylisting but I thought I'd ask. I did see some email settings saying From @riseup.net which indicates it might not be configured yet.

Hey! How do I login into the new nextcloud? I'm getting an 'unable to connect' when I try the onion addresses in this ticket.

Replying to ln5:

Micah, did you verify outgoing email?

I did!

But then I put a restrictive firewall in place that denies all outgoing connections that aren't approved, and failed to remember that this connection needed to happen. So I've fixed that now

Replying to micah:

Replying to ln5:

Micah, did you verify outgoing email?

I did!

But then I put a restrictive firewall in place that denies all outgoing connections that aren't approved, and failed to remember that this connection needed to happen. So I've fixed that now

Works, thanks!

From: is still nc@riseup.net though. Is that intentional?

Replying to gaba:

Hey! How do I login into the new nextcloud? I'm getting an 'unable to connect' when I try the onion addresses in this ticket.

The http:// link is the one you want.

That said, if you did click the https:// link you might need to restart your Tor Browser to not automatically end up on the https:// link even when entering the http:// link. There might be less destructive ways than restarting.

UPDATE (thanks gk): The http:// link is still redirecting to https://

$ curl -x socks4a://127.0.0.1:9050/ -v http://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/ ... < HTTP/1.1 302 Found ... < Location: https://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/login

Replying to micah:

Replying to ln5:

http://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/ redirects to https://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/login which fails, supposedly bc 443.

I've resolved this

Is it crazy to redirect 443 to 80 for the onion?

Trac:
Component: Internal Services/Services Admin Team to Internal Services/Service - nextcloud
Cc: linus@torproject.org, gaba to gaba

NOTE: Before we start migrating any content, we should get LDAP integration running. This will give us all users and groups (!) from LDAP.

Micah, please comment on #32332 (moved) to let us know if any of the requirements for the "LDAP User and group backend" application are problematic.

Trac:
Description: in #31540 (moved), we agreed to delegate running a dedicated Nextcloud instance to riseup.

this ticket tracks the next steps, so far:

create the private instance (riseup)
point DNS at the private instance (TPA)
make sure HTTPS works (riseup?)
move content and users over (teams?)

to

in #31540 (moved), we agreed to delegate running a dedicated Nextcloud instance to riseup.

this ticket tracks the next steps, so far:

create the private instance (riseup)
point DNS at the private instance (TPA)
make sure HTTPS works (riseup?)
#32332 (moved) set up LDAP integration, somehow (service admins + riseup)
move content and users over (teams?)

commit faf9e63fe52c2efb0771e662f76247b32f946ede Author: Antoine Beaupré anarcat@debian.org Date: Thu Oct 24 13:58:11 2019 -0400

add nc.torproject.org for nextcloud (#32267)

This points to the static mirrors for redirection, using the "vanity
hosts" macro.

The rationale is that our users might not be familiar with the .net
and .org distinction. Even if they do, they might not know we don't
manage this machine directly. Instead, they will probably look for
nc.tpo and fail and complain to us. So we should help people a little
here and make an exception for this domain which would be possibly
heavily used.

This is especially relevant since we *might* eventually use `.org` for
this purpose and manage the Nextcloud service ourselves.

But we don't do just a CNAME to the end server here: we do a redirect
so the canonical URL remains the `.net` one, to clearly show this is
not managed by TPA or the service admin team.

diff --git a/torproject.org b/torproject.org index 92d68b3..b49dc6f 100644 --- a/torproject.org +++ b/torproject.org @@ -149,6 +149,8 @@ bugs IN CNAME static censorshipwiki IN CNAME static safetyboard IN CNAME static wiki IN CNAME static +; https://trac.torproject.org/projects/tor/ticket/32267 +nc IN CNAME static

; static-mirrors aus1 IN CNAME static

Trac:
Username: hakar11

Trac:
Cc: gaba to gaba, micah

Apologies for the delay in response, I was not in CC on this ticket.

Replying to ln5:

From: is still nc@riseup.net though. Is that intentional?

It is, but it can be setup to be something else, if you want?

Apologies for the slow response, I wasn't on CC on this issue.

UPDATE (thanks gk): The http:// link is still redirecting to https://

$ curl -x socks4a://127.0.0.1:9050/ -v http://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/ ... < HTTP/1.1 302 Found ... < Location: https://gd5tq2xucgv35k7gokt6ts2tslaosjnwferdtqf2r3cbbqv5qvbxnmyd.onion/login

This is working now, again. I had fixed that in the webserver, but it seems like there was a configuration option in nextcloud that also was causing this.

Trac:
Description: in #31540 (moved), we agreed to delegate running a dedicated Nextcloud instance to riseup.

this ticket tracks the next steps, so far:

create the private instance (riseup)
point DNS at the private instance (TPA)
make sure HTTPS works (riseup?)
#32332 (moved) set up LDAP integration, somehow (service admins + riseup)
move content and users over (teams?)

to

in #31540 (moved), we agreed to delegate running a dedicated Nextcloud instance to riseup.

this ticket tracks the next steps, so far:

create the private instance (riseup)
point DNS at the private instance (TPA)
make sure HTTPS works (riseup?)
#32332 (moved) set up LDAP integration, somehow (service admins + riseup)
move content and users over (gaba + pili + teams)

we agreed on postponing the LDAP requirement for now, as we're unsure how to proceed and don't want to block deployment. gaba will manage accounts by hand for now.

Trac:
Description: in #31540 (moved), we agreed to delegate running a dedicated Nextcloud instance to riseup.

this ticket tracks the next steps, so far:

create the private instance (riseup)
point DNS at the private instance (TPA)
make sure HTTPS works (riseup?)
#32332 (moved) set up LDAP integration, somehow (service admins + riseup)
move content and users over (gaba + pili + teams)

to

in #31540 (moved), we agreed to delegate running a dedicated Nextcloud instance to riseup.

this ticket tracks the next steps, so far:

create the private instance (riseup)
point DNS at the private instance (TPA)
make sure HTTPS works (riseup?)
~~#32332 (moved) set up LDAP integration, somehow (service admins + riseup)~~
move content and users over (gaba + pili + teams)

storm was decom'd, i think the only step left to finalize the launch here is to purge old stuff from nc.riseup.net, in #32391 (moved). that's currently assigned to micah but i suspect gaba will handle it.

i believe all this is done here. we have long since migrated to nextcloud and the cleanup was done, so it was just a matter of closing this ticket, i think. whoohoo!

Trac:
Resolution: N/A to fixed
Status: assigned to closed

closed

mentioned in issue #32332 (moved)

mentioned in issue #32367 (moved)

mentioned in issue #32390 (moved)

mentioned in issue #32391 (moved)

mentioned in issue #32392 (moved)

move to a private nextcloud instance

Child items ...

Activity