Move Tails Sysadmins' GitLab projects to Tor's GitLab
- Wait for the merge to go public
- Freeze tails/sysadmin on Tails' Gitlab
- As root on Tails' Gitlab, make an export of the tails/sysadmin project
- Wait for the export to complete, download it, and unpack the tgz
- Archive the tails/sysadmin project on Tails' Gitlab
-
Retrieve all the user id's that have been active in the project issues:
cat tree/project/issues.ndjson |jq ''|grep author_id|sed -e 's/^ *"author_id": //'|sed -e 's/,//' |sort |uniq > uids.txt -
for each uid:
- check if their username and/or email exists in tor's gitlab
- if only one of the two exists or both exist but they do not match:
- contact the user and ask how they want to resolve this
- proceed accordingly
- if both exist and match:
- add the user details to tree/project/project_members.ndjson
- use tails' user_id and set their public_email attribute
- set access_level to 10 (guest)
- if they do not exist:
- check if they had recent activity on tails' gitlab, if so:
- send them an email explaining the merge and telling them they'll receive activation instructions
- create an account for them on tor's gitlab
- else:
- add a postfix transport for them on tor's gitlab, delivering all mail to them to the discard daemon.
- create an account for them on tor's gitlab and block/deactivate it
- add the user details to tree/project/project_members.ndjson
- use tails' user_id and set their public_email attribute
- set access_level to 10 (guest)
- check if they had recent activity on tails' gitlab, if so:
- tar and gzip the export again
- On Tor's Gitlab, enable imports from other gitlab instances
- On Tor's Gitlab, create the tails/sysadmin project by importing the new tgz file
- On Tor's Gitlab, move the tails/sysadmin project to tpo/tpa/tails/sysadmin
- Raise access levels as needed
- Repeat for sysadmin-private
- Remove the postfix transports we added
- Once both projects are migrated, have gitlab.tails.boum.org/tails/sysadmin redirect to gitlab.torproject.org/tails/sysadmin , ditto for sysadmin-private.
- Finally, edit all wikis and code repositories that link to the sysadmin project as issue tracker and replace with the Tor Gitlab.
Designs
- #41721TPA-RFC-73: Tails merge (2024)
Activity
added Next label
assigned to @zen
mentioned in issue #41721 (closed)
marked this issue as related to #41721 (closed)
one thing to consider is whether (or rather to which degree) we want to preserve authorship for issues. i think preservation is a must at least for the former core workers.
there are however also quite a few issues that were created (or commented on) by volunteers and external contributors who likely do not have an account on gitlab.torproject.org. should we create accounts for them?
that is an excellent question.
when we migrated from Trac, we imported a metric ton of accounts like this that are now all disabled, and we're kind of stuck with them, thinking of killing them entirely now....
what would happen to those (often completely idle) accounts in the long run?
and what's the default state of affairs, in other words, what happen if you "just" import the repos as is without creating any users?
are are contributions mapped in the first place?
an interesting question, overall, and probably the biggest hurdle to figure out, short of namespace and ACL issues.
-
for what it's worth, the total amount of users who authored or commented on an issue is 68, and i guess a few of them already exist in the torproject gitlab.
what would happen to those (often completely idle) accounts in the long run?
i would suggest that any new account we create during the import that hasn't had any activity on the tails gitlab in the last year is created in a disabled state. that way we can keep them indefinitely without any cost to security or noteworthy resources.
and what's the default state of affairs, in other words, what happen if you "just" import the repos as is without creating any users?
then we have a whole bunch of issues authored by, commented on, and possibly assigned to 'Administrator'. that's pretty annoying for things we're still working on and outright disastrous for keeping historical records.
are are contributions mapped in the first place?
with some minor tweaking, yes, they can be mapped.
-
68 sounds like really not a lot of users, i'd say create them.
one key concern though is what happens if you create those users now, then what happens when we migrate the rest of the gitlab tails stuff... do we need to recreate those users or recheck? or you put a freeze on new accounts on the tails gitlab?
About the timing for this, it sounds to me like an "irreversible change" and we had thought about waiting until Jan 2025 to start those.
Another aspect to consider are the impacts in the GitLab workflow if we were to only move sysadmin projects and leave the rest behind (at the very least, linking would become harder).
Should we maybe consider (1) doing this later on and (2) moving everything and not only sysadmin things?
-
is this really irreversible?
what's the rollback scenario here... let's say those projects have been migrated here. they have probably been archived in tails' gitlab, but all of the other projects there are still fully functional, and you still have your users there.
there will have been a couple dozen issues (perhaps more?) created in TPO's gitlab here that will be either lost or will need to be migrated back. if they are scoped in their own namespace (as I would suggest we do), it would be relatively easy to roll back...
i guess we need to figure out a better definition of "irreversible"... if by that we mean "oh dear, more work", then we can effectively do nothing because basically any change will require some work to roll back.
this one, IMHO, doesn't feel irreversible...
-
Sure, nothing is really irreversible, and the amount of work and potential negative impacts are the "metrics" I wanted to bring attention to.
I have a tendency of being overly-concerned with the impacts in others, so it's good that you help balance me out.
I agree that gardening / triage for sysadmins will now be a completely different process than it was before.
GitLab federation would be handy, if it was already a thing.
-
I have a tendency of being overly-concerned with the impacts in others, so it's good that you help balance me out.
Oh yes, absolutely: part of the RFC process is to make sure the right stakeholders are not only informed, but have a say in changes that affect them...
GitLab federation would be handy, if it was already a thing.
ha. That will be the day. The closest thing i've seen of that so far is https://radicle.xyz/ but that takes, well, a rather radical approach to the problem, and is clearly not compatible with normal GitLab workflows.
Forgejo also made some noises about supporting this through ActivityPub, but i have yet to see any significant collaboration on that front.
-
in any case here, @groente i think the next step is to make a proposal to get everyone on the same page.
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-1-policy
feel free to test imports and so on, but before doing anything solid, seek consent.
thanks for working on this!
-
ugh.
i'm not sure.
i think we should keep moving forward on this. allocate an RFC number, create an empty page with a dummy name, then work on it in a pad or something, then communicate it internally (tpa-team@ and tails folks) so that we can move ahead. then we can publicize it in the wiki properly after.
what do you think?
-
so, i started drafting something here: https://pad.tails.net/fT560f7QT522L-1bkWjxKg?both
i'm not really sure how much detail the RFC should have, is it meant to also contain implementation details?
-
I tend to put more than less details in my proposals, which can sometimes mean they're too long.
But i'm starving for details here a bit. Where will the repos move to? what will it mean for other repos on the gitlab tails? is there a timeline planned for those?
i see that you have a nice checklist in the issue here above, that would be a nice addition to the proposal. or at least mention that you have a checklist and link to it. but i like to crystalize it in the proposal so we have a snapshot in time of what the original plan was. then the issue evolves over time and that's fine...
i was a bit hoping we'd set a plan or at least a timeline of a plan for the rest of the migration here. at least we should say something like "we're not touching the rest of the gitlab, but we will eventually merge both instances, in collaboration with [blablabla]".
ideally, i'd like to discuss the namespacing issues here: do we move to a
tails/toplevel namespace? and i would include the tails and tor-internal folks in there as well.the reasoning for this is that i've had trouble with partial migrations to gitlab before, where we had to painfully recheck all projects to see where they ended up, sometimes with errors. we still get complaints that people can't find repos that used to be on git.tpo because the redirections are broken...
if we have a good plan on namespacing, then we can say something like
gitlab.tails.boum.org/.* -> gitlab.torproject.org/tails/.*and we have a single rewrite rule. then we can move projects or issues around on gitlab.tpo and redirections will stick (i think!).what do you think?
-
tnx, i added and rewrote some stuff in hedgedoc.
i was a bit hoping we'd set a plan or at least a timeline of a plan for the rest of the migration here
nah, i'm sticking to a narrow scope of migrating two projects, those are actually easy to tackle and i wanna start making some progress here.
if we have a good plan on namespacing, then we can say something like
gitlab.tails.boum.org/.* -> gitlab.torproject.org/tails/.*and we have a single rewrite rule. then we can move projects or issues around on gitlab.tpo and redirections will stick (i think!).neat trick! <3
-
i would have prefered to have a proposal to explicitly suggest to the rest of tails the
gitlab.tails.boum.org/.* -> gitlab.torproject.org/tails/.*namespace idea, but i can live with this being out of scope for now, and having possibly two sets of rewrite rules.i must warn you that this is likely going to create a precedent and people are likely to start migrating projects on their own, piecemeal like this, and you'll have to manage redirects by project.
-
Freeze tails/sysadmin on Tails' Gitlab
this is an implementation detail that i'm fine keeping out of the RFC (which, if you want, you can send to tpa-team now), but what does that entail? are we talking about "archiving" the repository here? if so, I would just call it like that, and if not, i would actually archive the repository (and change its description) on tails' gitlab.
you don't mention redirections in the implementation plan either... how to do plan on dealing with gitlab possibly renumbering issues?
-
Freeze tails/sysadmin on Tails' Gitlab this is an implementation detail that i'm fine keeping out of the RFC (which, if you want, you can send to tpa-team now), but what does that entail? are we talking about "archiving" the repository here? if so, I would just call it like that, and if not, i would actually archive the repository (and change its description) on tails' gitlab.
i was thinking of just telling everyone to stay the fuck away from the sysadmin issues on the day of the migration. archiving right after the export is probably a good idea, though.
you don't mention redirections in the implementation plan either... how to do plan on dealing with gitlab possibly renumbering issues?
I added:
Once migrated, have gitlab.tails.boum.org/tails/sysadmin redirect to gitlab.torproject.org/tails/sysadmin , ditto for sysadmin-private.My tests indicate there's no renumbering, so that should do the trick.
marked this issue as related to #41780 (closed)
mentioned in issue #41780 (closed)
added Needs Review label and removed Doing label
added Next label and removed Needs Review label
removed Next label
added Doing label
added Tails label
-
so, after a bunch of testing, i couldn't find any way to have gitlab not send an e-mail on account creation. that's pretty annoying for all the deactivated accounts we want to move from tails' to tor's gitlab. i don't want gitlab to mail all these people with "hey, you have a new account!" and then have to manually mail them again saying "oh btw, you're deactivated"
the workaround i have in mind is collecting all the e-mail addresses of the deactived users we're going to migrate and before creating them on tor's gitlab, first adding them to a postfix transport map, setting their transport to discard. once the migration is done, we can remove them again.
-
so, after a bunch of testing, i couldn't find any way to have gitlab not send an e-mail on account creation. that's pretty annoying for all the deactivated accounts we want to move from tails' to tor's gitlab. i don't want gitlab to mail all these people with "hey, you have a new account!" and then have to manually mail them again saying "oh btw, you're deactivated"
Hey @ahf, are you familiar with this problem?
You probably created hundreds of GitLab accounts when you did the trac migration, how did you handle the email notification stuff?
the workaround i have in mind is collecting all the e-mail addresses of the deactived users we're going to migrate and before creating them on tor's gitlab, first adding them to a postfix transport map, setting their transport to discard. once the migration is done, we can remove them again.
Another idea would be to add them all with a dummy email address domain, then rewrite all emails in a batch?
...
On 2024-10-14 10:24:46, groente wrote:
I created the accounts with fake email addresses (ahf+${username}@torproject.org) IIRC and then we later changed them manually via the API or the admin interface.
This was why we had the period with all the migrated users that we had to awkwardly and manually open up. I think you, Gaba, and I did a lot of that for a longer period of time.
changed milestone to %TPA-RFC-73: Tails merge (2024)
removed Doing label
added Doing label
added Needs Information label
-
fwiw, i just noticed some authorship of comments (in particular the comments from @zen) on tpo/tpa/tails/sysadmin#16958 were broken. also some tag history and references to commits puppet-tails got lost :-/ might wanna look into that more when moving the next project.
removed Doing label
