@@ -377,6 +377,10 @@ file. (We have considered taking this out of the template and writing
...
@@ -377,6 +377,10 @@ file. (We have considered taking this out of the template and writing
a proper Python script, but then users would have to copy that script
a proper Python script, but then users would have to copy that script
over their repo, or clone a repo in CI, and that seems impractical.)
over their repo, or clone a repo in CI, and that seems impractical.)
Another thing we considered is to set [instance-level templates](https://docs.gitlab.com/ee/user/admin_area/settings/instance_template_repository.html)
but it seems that feature is not available in GitLab's free software
version.
The CI hooks are deployed by users, which will typically include the
The CI hooks are deployed by users, which will typically include the
above template in their own `.gitlab-ci.yml` file.
above template in their own `.gitlab-ci.yml` file.
...
@@ -476,6 +480,7 @@ good copies of the websites, in any case.
...
@@ -476,6 +480,7 @@ good copies of the websites, in any case.
## Other documentation
## Other documentation
*[TPA-RFC-10][]: Jenkins retirement
* GitLab's [CI deployment mechanism](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/) blog post
* GitLab's [CI deployment mechanism](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/) blog post
*[Design and launch ticket](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40364)
*[Design and launch ticket](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40364)
* our [static mirror system][] documentation
* our [static mirror system][] documentation
...
@@ -493,13 +498,15 @@ good copies of the websites, in any case.
...
@@ -493,13 +498,15 @@ good copies of the websites, in any case.
## Overview
## Overview
The static shim was built to unblock the [Jenkins retirement
The static shim was built to unblock the [Jenkins retirement
project](https://gitlab.torproject.org/groups/tpo/-/milestones/27). A key blocker was that the [static mirror system][] was
project](https://gitlab.torproject.org/groups/tpo/-/milestones/27)([TPA-RFC-10][]). A key blocker was that the [static mirror system][] was
strongly coupled with Jenkins: many high traffic and critical websites
strongly coupled with Jenkins: many high traffic and [critical websites](policy/tpa-rfc-10-jenkins-retirement#critical-website-builds)
are built and deployed by Jenkins. Unless we wanted to completely
are built and deployed by Jenkins. Unless we wanted to completely
retire the static mirror system (in favor, say, of GitLab Pages), we
retire the static mirror system (in favor, say, of GitLab Pages), we
had to create a way for GitLab CI to deploy content to the static
had to create a way for GitLab CI to deploy content to the static
mirror system.
mirror system.
[TPA-RFC-10]:policy/tpa-rfc-10-jenkins-retirement
This section contains more in-depth discussions about the reasoning
This section contains more in-depth discussions about the reasoning
behind the project, discarded alternatives, and other ideas.
behind the project, discarded alternatives, and other ideas.
...
@@ -528,7 +535,7 @@ deployment.
...
@@ -528,7 +535,7 @@ deployment.
## Approvals required
## Approvals required
TPA.
TPA, part of [TPA-RFC-10][]: Jenkins retirement.
## Proposed Solution
## Proposed Solution
...
@@ -628,7 +635,7 @@ created very frequently, it could also be done by hand.
...
@@ -628,7 +635,7 @@ created very frequently, it could also be done by hand.
Unfortunately this design has two major flaws:
Unfortunately this design has two major flaws:
1. webhooks are designed to be fast and short-lived: most site
1. webhooks are [designed to be fast and short-lived](https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#webhook-endpoint-tips): most site
deployments take longer than the pre-configured webhook timeout (10
deployments take longer than the pre-configured webhook timeout (10
seconds) and therefore cannot be deployed synchronously, which
seconds) and therefore cannot be deployed synchronously, which
implies that...
implies that...
...
@@ -644,5 +651,26 @@ Unfortunately this design has two major flaws:
...
@@ -644,5 +651,26 @@ Unfortunately this design has two major flaws:
recent webhook calls, in Settings -> Webhooks -> Edit ->
recent webhook calls, in Settings -> Webhooks -> Edit ->
Recent deliveries. But that is rather well-hidden.
Recent deliveries. But that is rather well-hidden.
In the short term, the webhook system has be used asynchronously,
Note that it may have been possible to change the 10 seconds timeout
but we have since moved to the deployment system documented above.
with:
gitlab_rails['webhook_timeout'] = 10
in the `/etc/gitlab/gitlab.rb` file ([source](https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#webhook-fails-or-multiple-webhook-requests-are-triggered)). But static site
deployments can take a while, so it's not clear at all we can actually
wait for the full deployment.
In the short term, the webhook system has be used asynchronously (by
removing the `include-command-output-in-response` parameter in the
webhook config), but then the error reporting is even worse because
the caller doesn't even know if the deploy succeeds or fails.
We have since moved to the deployment system documented in the [design
section](#design).
### GitLab "Integrations"
Another approach we briefly considered is to write an [integration](https://docs.gitlab.com/ee/user/project/integrations/overview.html)
into GitLab. We found the documentation for this was nearly
nonexistent. It also meant maintaining a bundle of Ruby code inside