Gitlab issueshttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues2020-06-27T14:44:06Zhttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/9Help the applications team with cloning Git projects from git.torproject.org ...2020-06-27T14:44:06ZAlexander Færøyahf@torproject.orgHelp the applications team with cloning Git projects from git.torproject.org into Gitlab@gk and @sysrqb are interested in having some of the browser related projects being automatically cloned from git.torproject.org to Gitlab such that they can begin doing code review on Gitlab.@gk and @sysrqb are interested in having some of the browser related projects being automatically cloned from git.torproject.org to Gitlab such that they can begin doing code review on Gitlab.Alexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/162FA2020-06-26T06:51:24ZGabagaba@torproject.org2FAI think we should have 2FA in all admin accounts at least.I think we should have 2FA in all admin accounts at least.https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/21publish the migration source code2021-10-21T15:14:10Zanarcatpublish the migration source codeit would be great to have the migration source code public, if that is safe at all.
it would serve a few purposes:
1. it would allow us to diagnose problems better if they come up
2. it would give good examples on how to talk to the ...it would be great to have the migration source code public, if that is safe at all.
it would serve a few purposes:
1. it would allow us to diagnose problems better if they come up
2. it would give good examples on how to talk to the API
3. it would show others how we did it
Of course the code would need to be sanitized for secrets, but it doesn't have to be clean.Alexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/24Create a solution for missing /duplicate #NNNN2020-07-09T08:51:06ZGeorg KoppenCreate a solution for missing /duplicate #NNNNWe are not hip enough with our Gitlab Community Edition to be allowed to
use the `/duplicate #NNNN` functionality. However, having that would be
very desirable.
The workflow for us for closing duplicates in Trac was:
1) Closing a ticke...We are not hip enough with our Gitlab Community Edition to be allowed to
use the `/duplicate #NNNN` functionality. However, having that would be
very desirable.
The workflow for us for closing duplicates in Trac was:
1) Closing a ticket $foo as duplicate mentioning which ticket $bar it
was a duplicate of.
2) Going to ticket $bar and adding all the folks being Cc'ed to or the
reporter of ticket $foo to ticket $bar's Cc list.
3) Mentioning in a comment to ticket $bar that ticket $foo was a duplicate.
Ideally, we would get 2) and 3) automatically once we do 1).
I've not a good handle yet on what is possible with Gitlab, but I am
willing to help with that issue, I think. @ahf had some ideas to start
with, though, IIRC.https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/31document gitlab user creation, project adhesion and permission policies2020-10-20T15:42:39ZGabagaba@torproject.orgdocument gitlab user creation, project adhesion and permission policiesWe need
* clear criterias on adding a user to a project
* clear criterias on which role/permissions to give users added to a projectWe need
* clear criterias on adding a user to a project
* clear criterias on which role/permissions to give users added to a projectGabagaba@torproject.orgGabagaba@torproject.orghttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/34Create gitlab cheat sheet2021-05-31T16:06:14ZGeorg KoppenCreate gitlab cheat sheetWe probably should create a Gitlab cheat sheet somewhere containing tips
and tricks around our Gitlab interactions which might be helpful for a
lot of us. #29 has something related to email interactions but there is
more.We probably should create a Gitlab cheat sheet somewhere containing tips
and tricks around our Gitlab interactions which might be helpful for a
lot of us. #29 has something related to email interactions but there is
more.https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/38Protected Branches are problematic for mirroring2021-10-21T14:29:39ZMatthew FinkelProtected Branches are problematic for mirroringCC: @ahf @gk
With #9 we got cloning/mirroring between gitolite and gitlab. Ignoring #36 for now, it seems we have a problem with using protected branches. Specifically, we configured protected branches for '*' and disable merging via t...CC: @ahf @gk
With #9 we got cloning/mirroring between gitolite and gitlab. Ignoring #36 for now, it seems we have a problem with using protected branches. Specifically, we configured protected branches for '*' and disable merging via the web interface, and 'Maintainers' are allowed push access. This seemed like it was close enough to what we wanted, however we now have another problem.
```
remote: remote: GitLab: You can only delete protected branches using the web interface.
remote: To dip.torproject.org:tpo/applications/tor-browser-build
remote: ! [remote rejected] 20254-update-marsigning-check-sh-to-cope-with-signed-os-x-mar-files (pre-receive hook declined)
remote: ! [remote rejected] maint-9.5 -> maint-9.5 (pre-receive hook declined)
remote: ! [remote rejected] refs/merge-requests/2/head (pre-receive hook declined)
remote: ! [remote rejected] refs/merge-requests/3/head (pre-receive hook declined)
remote: ! [remote rejected] refs/merge-requests/3/merge (pre-receive hook declined)
```
Specifically, the description of Protected Branches includes:
* prevent **anyone** from deleting the branch
* prevent **anyone** from force pushing to the branch
I guess we need a different approach.Alexander Færøyahf@torproject.orgAlexander Færøyahf@torproject.orghttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/39Milestones for non-58 ESR migration2020-07-08T17:50:31ZGabagaba@torproject.orgMilestones for non-58 ESR migrationDue on 9/22 we need to create milestones to be able to track it.Due on 9/22 we need to create milestones to be able to track it.Gabagaba@torproject.orgGabagaba@torproject.orghttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/55move gitlab project back into tpo/tpa/team2022-03-16T15:26:59Zanarcatmove gitlab project back into tpo/tpa/teamin #10, we decided to use labels to split projects in the tpo/tpa team.
yet at some point a gitlab project was created here to follow issues with the migration. that seems like a contradiction.
either we should revert the decision in #...in #10, we decided to use labels to split projects in the tpo/tpa team.
yet at some point a gitlab project was created here to follow issues with the migration. that seems like a contradiction.
either we should revert the decision in #10 and split tickets in subprojects (somehow), or we should not have subprojects unless we really need to (e.g. if they have a git repository, which the gitlab project doesn't have).
this discussion came out of #12, where it was identified we couldn't build a final redirect map until we made up our mind about the ticket moves.https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/67Gitlab mirror broken for snowflake-webext2020-10-08T16:43:48ZCecylia BocovichGitlab mirror broken for snowflake-webextI just pushed some new commits to the snowflake-webext gitweb.tp.o repo: https://gitweb.torproject.org/pluggable-transports/snowflake-webext.git/log/
But they aren't being properly mirrored to the gitlab repository: https://gitlab.torpr...I just pushed some new commits to the snowflake-webext gitweb.tp.o repo: https://gitweb.torproject.org/pluggable-transports/snowflake-webext.git/log/
But they aren't being properly mirrored to the gitlab repository: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/commits/master
I got the following error message during the push:
```
$ git push origin master
Counting objects: 8, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 2.00 KiB | 682.00 KiB/s, done.
Total 8 (delta 7), reused 0 (delta 0)
remote: == 00-sync-to-mirror ==
remote: == commit-mail ==
remote: == github-push ==
remote: == gitlab-push ==
remote: remote: GitLab: You are not allowed to force push code to a protected branch on this project.
remote: To dip.torproject.org:tpo/anti-censorship/pluggable-transports/snowflake-webext
remote: ! [remote rejected] master -> master (pre-receive hook declined)
remote: ! [remote rejected] 14-update-bug-reporting-links-for-gitlab -> 14-update-bug-reporting-links-for-gitlab (pre-receive hook declined)
remote: error: failed to push some refs to 'dip.torproject.org:tpo/anti-censorship/pluggable-transports/snowflake-webext'
remote: == irc-message ==
remote: == per-repo-hook ==
remote: == xx-jenkins-trigger ==
remote: [hook[22507]] Triggering jenkins build for (https://git.torproject.org/pluggable-transports/snowflake-webext.git, master, a2aa6acf51916e6000764dafce7197d6b85420a0).
remote: No git jobs using repository: https://git.torproject.org/pluggable-transports/snowflake-webext.git and branches: master
remote: No Git consumers using SCM API plugin for: https://git.torproject.org/pluggable-transports/snowflake-webext.git
remote: [hook[22507]] Jenkins triggers done.
To git-rw.torproject.org:pluggable-transports/snowflake-webext.git
c2c3bdb..a2aa6ac master -> master
```https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/72monitor anonymous user account2020-09-30T19:05:30ZMatthew Finkelmonitor anonymous user accountI noticed user `writecode` is now a replacement for the trac `cypherpunks` account with the password in the account's name. I'm not opposed to having an anonymous account, but I know there was concern about creating one when we migrated ...I noticed user `writecode` is now a replacement for the trac `cypherpunks` account with the password in the account's name. I'm not opposed to having an anonymous account, but I know there was concern about creating one when we migrated to gitlab. @gaba suggested we open a ticket, monitor the account's activity, and discuss it at the next gitlab meeting.https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/83Warning on Merge Requests: " Failed to get security report information. Pleas...2021-05-12T20:27:50ZNick MathewsonWarning on Merge Requests: " Failed to get security report information. Please reload the page or try again later."Hi! I see this warning at the head of all the merge requests I've looked at lately:
> Failed to get security report information. Please reload the page or try again later.
For one example, see https://gitlab.torproject.org/tpo/core/to...Hi! I see this warning at the head of all the merge requests I've looked at lately:
> Failed to get security report information. Please reload the page or try again later.
For one example, see https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/171https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/85gitlab 13.5 upgrade crashed the server2020-11-17T16:00:36Zanarcatgitlab 13.5 upgrade crashed the serverfollowing the [gitlab 13.5 release](https://about.gitlab.com/releases/2020/10/22/gitlab-13-5-released/) our server started failing with a 502 error.following the [gitlab 13.5 release](https://about.gitlab.com/releases/2020/10/22/gitlab-13-5-released/) our server started failing with a 502 error.https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/111The main branch in tpo/core/torspec does not seem to be getting updates from ...2021-11-16T19:45:43ZNick MathewsonThe main branch in tpo/core/torspec does not seem to be getting updates from gitolite.The latest commit on the `main` branch for torspec.git on git-rw.torproject.org is 7a06c12fea6e65dd0bcb411000d968844335c22f (26 August 2021).
But the latest commit on the `main` branch for tpo/core/torspec here is b244dd33a525e3c81aa867...The latest commit on the `main` branch for torspec.git on git-rw.torproject.org is 7a06c12fea6e65dd0bcb411000d968844335c22f (26 August 2021).
But the latest commit on the `main` branch for tpo/core/torspec here is b244dd33a525e3c81aa8679cda128226dcd9d6fa (30 July).
Mentioning this to @ahf and @dgoulet since I think it might have gone wrong when we renamed the branch from `master` to `main`?https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/120some gitlab runner(s?) fail to contact docker and fail jobs intermittently2023-08-09T13:40:41Zanarcatsome gitlab runner(s?) fail to contact docker and fail jobs intermittentlysometimes jobs fail with:
```
ERROR: Job failed (system failure): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (docker.go:651:120s)
```
example failing job:
https://gitlab.torprojec...sometimes jobs fail with:
```
ERROR: Job failed (system failure): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (docker.go:651:120s)
```
example failing job:
https://gitlab.torproject.org/nickm/arti/-/jobs/105400
another similar failure is:
```
ERROR: Preparation failed: adding cache volume: set volume permissions: create permission container for volume "runner-qlbl8xrr-project-647-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70": Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (linux_set.go:95:120s)
```
because of the latter, we originally thought this was related with a permission change on `/var/lib/docker/volumes`, but the permissions on that were restored in `tor-puppet.git` in:
```
commit aa7afc51dfde1daa78f75c1f529a4dade6280567
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Feb 28 10:31:52 2022 -0500
fix permissions on docker volumes
This commit created problems on runners:
commit a0c1db6df4b78a149b54b88621d03546143d2184
Author: Antoine Beaupré <anarcat@debian.org>
Date: Wed Feb 23 16:15:45 2022 -0500
fix permissions on docker volumes/images
This was seen on ci-runner-01:
Notice: /Stage[main]/Profile::Docker/File[/var/lib/docker/overlay2]/mode: mode changed '0710' to '0701'
... presumably that would happen any time puppet would run, fighting
with Docker.
It turns out this change was fine for `overlay2` but broke things when
volumes were used. An example failure is this error message:
ERROR: Preparation failed: adding cache volume: set volume permissions: create permission container for volume "runner-qlbl8xrr-project-950-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70": Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? (linux_set.go:95:120s)
in this job:
https://gitlab.torproject.org/tpo/core/doc/-/jobs/105169
That didn't happen in *all* jobs. That's possibly because some jobs
run as root? Unclear. In any case, those are the permissions on a
non-managed install (my workstation) so they should work better.
diff --git a/modules/profile/manifests/docker.pp b/modules/profile/manifests/docker.pp
index f3b8296e..14190450 100644
--- a/modules/profile/manifests/docker.pp
+++ b/modules/profile/manifests/docker.pp
@@ -94,7 +94,7 @@ class profile::docker(
}
file { '/var/lib/docker/volumes':
ensure => directory,
- mode => '0710',
+ mode => '0701',
require => Package['docker.io'],
}
file { '/var/lib/docker/overlay2':
```
yet that didn't fix the issue, because the above two jobs failed *after* the above commit was deployed.
upstream has had an issue opened about this for 4 years, but it has seen some recent activity, so this could be a regression upstream:
https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2890
it's intermittent, and we don't have a clear root cause or plan of action. keeping this ticket open to track related incidents and keeping our users informed.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/125review the shared runners collection2022-05-24T15:17:50Zanarcatreview the shared runners collectioni mistakenly thought anyone could add runners to the global shared runners pool, but i was told this is reserved to admins.
let's review the shared runners we have. maybe we should remove some that are paused or unused or unknown. and m...i mistakenly thought anyone could add runners to the global shared runners pool, but i was told this is reserved to admins.
let's review the shared runners we have. maybe we should remove some that are paused or unused or unknown. and maybe we should have a policy on what can get added there in the first place.
/cc @ahfanarcatanarcathttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/133consider security levels for runners2023-06-28T18:44:11Zanarcatconsider security levels for runnersWikimedia foundation have an interesting approach to GitLab runner security:
https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Security_Evaluation
Basically, they have three levels:
* nothing: you bring your own runners, that ...Wikimedia foundation have an interesting approach to GitLab runner security:
https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Security_Evaluation
Basically, they have three levels:
* nothing: you bring your own runners, that would be the equivalent of everyone *not* under the `tpo/` namespace for us
* members-only: access to shared runners
* special projects: allow-list of select projects which are the only ones with access to hardened runners, and *only* on the protected branch
There's all sorts of hardening on those runners, which include:
* image restrictions: an allow-list of images that are allowed to be used for jobs
* non-root gitlab-runner
* (planned) rootless docker (see #129 for our effort at podman, which could do this as well)https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/148Wrong domain of GitLab's mail server certificate2024-02-08T16:05:24ZMynacolWrong domain of GitLab's mail server certificateI wanted to reply to a GitLab issue by mail, but my mail server refused to send it, as the TLS certificate could not be verified. My mail server is configured to strictly verify the respective certificates.
The mail was headed to `[...]...I wanted to reply to a GitLab issue by mail, but my mail server refused to send it, as the TLS certificate could not be verified. My mail server is configured to strictly verify the respective certificates.
The mail was headed to `[...]@gitlab.torproject.org`. My mail server queried the MX record of gitlab.torproject.org, but only got a CNAME response, which leads to gitlab-02.torproject.org that points to the right IP addresses. Now my mail server expected a TLS certificate for gitlab.torproject.org, but your postfix provided a certificate for gitlab-02.torproject.org, which my mail server regarded as invalid.
The easiest way to fix this is to add a MX record to gitlab.torproject.org pointing at gitlab-02.torproject.org. That could even help with mail deliverability.
Alternatively, you can provide a certificate for gitlab.torproject.org from your mail server just like on the website.
Maybe the test page on [internet.nl](https://internet.nl/mail/gitlab.torproject.org/1127446/) helps you too.improve mail servicesJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/151Encrypt confidential emails2024-03-06T18:30:32Zmicahmicah@torproject.orgEncrypt confidential emailsNow that [confidential issue emails are not sent in the clear](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/23) it might be interesting to consider what it would take to modify the current process so that it sent encrypted email...Now that [confidential issue emails are not sent in the clear](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/23) it might be interesting to consider what it would take to modify the current process so that it sent encrypted emails with the actual contents of the issue, if possible.
We could modify the python script that was deployed to redact the message to add a few additional steps: query the gitlab API (requires a personal access token) to look up a user's ID, and then lookup that ID's OpenPGP key. If they have one set, then encrypt the mail to them, and if they do not, then do what we do now (send the "This issue is confidential and the contents of the message have been redacted." message).
Its actually quite easy to get the public key of a user via the API, if you know their email address. To figure out how, I wrote this script that will spit out the OpenPGP key of any email in gitlab *as long as the user's email is public*:
```python
import argparse
import requests
import sys
GITLAB_API_URL = 'https://gitlab.torproject.org/api/v4'
GITLAB_PRIVATE_TOKEN = '<redacted>'
def get_user_id(email):
"""Fetch the GitLab user ID based on the email address."""
headers = {'Private-Token': GITLAB_PRIVATE_TOKEN}
try:
response = requests.get(f"{GITLAB_API_URL}/users?search={email}", headers=headers)
response.raise_for_status() # Raises an HTTPError if the response code was unsuccessful
users = response.json()
if not users:
print(f"API returned no users for email: {email}")
return None
return users[0]['id']
except requests.exceptions.HTTPError as e:
print(f"HTTPError: {e.response.status_code} {e.response.reason}")
except requests.exceptions.RequestException as e:
print(f"RequestException: {e}")
return None
def get_user_gpg_keys(user_id):
"""Fetch the GPG keys associated with a GitLab user ID."""
headers = {'Private-Token': GITLAB_PRIVATE_TOKEN}
try:
response = requests.get(f"{GITLAB_API_URL}/users/{user_id}/gpg_keys", headers=headers)
response.raise_for_status()
return response.json()
except requests.exceptions.HTTPError as e:
print(f"HTTPError: {e.response.status_code} {e.response.reason}")
except requests.exceptions.RequestException as e:
print(f"RequestException: {e}")
return []
def main():
parser = argparse.ArgumentParser(description='Fetch a GitLab user\'s GPG key by email address.')
parser.add_argument('email', type=str, help='The email address of the user to search for.')
args = parser.parse_args()
user_id = get_user_id(args.email)
if not user_id:
sys.exit("Exiting: No user found or error occurred.")
gpg_keys = get_user_gpg_keys(user_id)
if not gpg_keys:
print(f"No GPG keys found for user with ID {user_id}")
return
for key in gpg_keys:
print(f"{key['key']}")
if __name__ == "__main__":
main()
```
Considering that is how you would pull the key from the API endpoint, then you could imagine modifying the existing deployed script to incorporate this. Here is some *untested* somewhat pseudo-code modification of the currently deployed script. Because I can't test this, there are some obvious things that are wrong, but the basic idea is there:
```python
#!/usr/bin/python3 -X utf8
import argparse
from email.parser import Parser
import email.policy
from io import StringIO
import logging
import quopri
from typing import TextIO
import subprocess
import sys
import requests
import gnupg
gpg = gnupg.GPG(gnupghome='/path/to/.gnupg')
gitlab_api_url = 'https://gitlab.torproject.org/api/v4'
gitlab_private_token = 'prviate access token'
def get_user_gpg_key(email):
"""Fetch the user's GPG key from GitLab API."""
headers = {'Private-Token': gitlab_private_token}
users = requests.get(f"{gitlab_api_url}/users?search={email}", headers=headers).json()
if not users:
return None
user_id = users[0]['id']
keys = requests.get(f"{gitlab_api_url}/users/{user_id}/gpg_keys", headers=headers).json()
if not keys:
return None
return keys[0]['key']
def encrypt_message(message, gpg_key):
"""Encrypt the message with the provided GPG key."""
imported_key = gpg.import_keys(gpg_key)
encrypted_data = gpg.encrypt(message, imported_key.fingerprints[0], always_trust=True)
return str(encrypted_data)
def main():
logging.basicConfig(level="INFO", format="%(levelname)s: %(message)s")
parser = argparse.ArgumentParser()
parser.add_argument("--stdout", action="store_true", help="send the email to stdout instead of sendmail(1)")
parser.add_argument("--from", "-f", dest="sender", help="sender address")
parser.add_argument("recipients", nargs="+", help="recipient address(es)")
args = parser.parse_args()
email_content = transform_email(sys.stdin, args.recipients[0]) # Assuming one recipient for simplicity
if args.stdout:
print(email_content)
else:
cmd = ["/usr/sbin/sendmail", "-G", "-i", "-f", args.sender, "--"] + args.recipients
try:
subprocess.Popen(cmd, stdin=subprocess.PIPE, encoding="utf-8").communicate(email_content)
except Exception as e:
import traceback
print("4.3.0 %s: %s" % (e, traceback.format_exc()))
sys.exit(75) # TEMPFAIL
def transform_email(stream: TextIO, recipient_email):
msg = Parser(policy=email.policy.compat32).parse(stream)
if msg.get('X-GitLab-ConfidentialIssue', 'false') != 'true':
return str(msg)
gpg_key = get_user_gpg_key(recipient_email)
if gpg_key:
# Encrypt the message if a GPG key is found
encrypted_msg = encrypt_message(str(msg), gpg_key)
return encrypted_msg
else:
# Redact if no GPG key is found
for part in msg.walk():
if part.is_multipart():
continue
if part.get_content_type() != 'text/plain':
continue
plain_text = part.as_string()
signature = quopri.decodestring(plain_text.split("-- ").pop()).decode("ascii")
redaction_msg = """This issue is confidential and the contents of the message have been redacted.
--
""" + signature
del msg['X-GitLab-ConfidentialIssue']
msg['X-GitLab-ConfidentialIssue'] = 'redacted'
msg.set_type("text/plain")
msg.del_param("boundary", header="Content-Type")
msg.set_payload(redaction_msg)
return str(msg)
if __name__ == "__main__":
main()
```