This is part of #40364

service/static-shim.md

@@ -16,24 +16,51 @@ TODO: "how do users add/remove sites"
 
 TODO: review ticket for possible howtos
 
-## Deploying a Hugo site
+## Deploying a static site from GitLab CI
 
-Normally, you should be able to deploy a hugo site by including the
-template and setting a few variables. This `.gitlab-ci.yml` file
-should be sufficient:
+First, you will need to make sure the site builds in GitLab CI. A
+`build` stage MUST be used that will produce artifacts that can be
+used by the `deploy` job provided in the [`static-shim-deploy.yml`
+template][]. How to build the website will vary according to the site,
+obviously. See the [hugo build instructions below](#building-a-hugo-site).
+
+[`static-shim-deploy.yml` template]: https://gitlab.torproject.org/tpo/tpa/ci-templates/-/blob/main/static-shim-deploy.yml
+
+TODO: link to documentation on how to build Lektor sites.
+
+It is a good idea to also add a `pages` stage to preview the
+build. The above template has an example `pages` stage.
+
+Then include the deploy job template in the `.gitlab-ci.yml` with a
+snippet like this:
 
 ```
+variables:
+  SITE_URL: example.torproject.org
+  SUBDIR: public/
+
 include:
-  project: tpo/tpa/status-site
-  file: .gitlab-ci.yml
+  project: tpo/tpa/ci-templates
+  file: static-shim-deploy.yml
+```
+
+The `SITE_URL` and `SUBDIR` parameters need to be changed to reflect,
+respectively, the FQDN of the website as defined in the
+`static-components.yml` file, and the directory where the website was
+built by the previous build stage. You may also need
+`GIT_SUBMODULE_STRATEGY: recursive` if you have submodules (e.g. hugo
+themes are often shipped as submodules). For example, for
+<https://status.torproject.org>, the `.gitlab-ci.yml` file looks like
+this (`build` stage elided for simplicity):
 
     variables:
       GIT_SUBMODULE_STRATEGY: recursive
-  SITE_URL: research.torproject.org
+      SITE_URL: status.torproject.org
       SUBDIR: public/
-```
 
-See below if this is an old hugo site, however.
+    include:
+      project: tpo/tpa/ci-templates
+      file: static-shim-deploy.yml
 
 Then you also need to set an SSH key in the project. First, generate a
 password-less key locally:
@@ -52,22 +79,40 @@ variable`, with the following parameters:
 
 Then the *public* part of that key needs to be added in Puppet. This
 can only be done by TPA, so file a ticket there if you need
-assistance. The key should be added in the `tor-puppet.git`
-repository, in the `hiera/common.yaml` file, in the
-`staticsync::gitlab_shim::ssh::sites` hash, in which the site URL is
-the key and the public key (only the key part, no `ssh-rsa` prefix or
-comment suffix) is the value. For example, this is the entry for
-`status.torproject.org`:
+assistance. For TPA, see below for the remaining instructions.
+
+You can commit the above changes to the `.gitlab-ci.yml` file, but
+when pushed, the pipeline's `deploy` stage is normal, TPA needs to do
+its magic for the deploy to work. Make sure the build works in GitLab
+pages before requesting the deploy in the static mirror system.
+
+### TPA: adding a new static site shim in Puppet
+
+The public key mentioned above should be added in the `tor-puppet.git` repository, in the
+`hiera/common.yaml` file, in the `staticsync::gitlab_shim::ssh::sites`
+hash.
+
+There, the site URL is the key and the public key (only the key part,
+no `ssh-rsa` prefix or comment suffix) is the value. For example, this
+is the entry for `status.torproject.org`:
 
     staticsync::gitlab_shim::ssh::sites:
       status.torproject.org: "AAAAB3NzaC1yc2EAAAADAQABAAABgQC3mXhQENCbOKgrhOWRGObcfqw7dUVkPlutzHpycRK9ixhaPQNkMvmWMDBIjBSviiu5mFrc6safk5wbOotQonqq2aVKulC4ygNWs0YtDgCtsm/4iJaMCNU9+/78TlrA0+Sp/jt67qrvi8WpLF/M8jwaAp78s+/5Zu2xD202Cqge/43AhKjH07TOMax4DcxjEzhF4rI19TjeqUTatIuK8BBWG5vSl2vqDz2drbsJvaLbjjrfbyoNGuK5YtvI/c5FkcW4gFuB/HhOK86OH3Vl9um5vwb3DM2HVMTiX15Hw67QBIRfRFhl0NlQD/bEKzL3PcejqL/IC4xIJK976gkZzA0wpKaE7IUZI5yEYX3lZJTTGMiZGT5YVGfIUFQBPseWTU+cGpNnB4yZZr4G4o/MfFws4mHyh4OAdsYiTI/BfICd3xIKhcj3CPITaKRf+jqPyyDJFjEZTK/+2y3NQNgmAjCZOrANdnu7GCSSz1qkHjA2RdSCx3F6WtMek3v2pbuGTns="
 
-Note that this will only allow GitLab CI to push to the shim, but will
-still fail to deploy because the static-component configuration does
-not match. Once you are satisfied your build works (try on a branch
-first to see if the GitLab pages version renders correctly), the
-static-component should be changed to point to the shim, with a commit
-like this:
+At this point, the deploy job should be able to `rsync` the content to
+the static shim, but the deploy will still fail because the
+static-component configuration does not match and the
+`static-update-component` step will fail.
+
+To fix this, the static-component entry should be added (or modified,
+if it already exists) to point to the shim. This, for example, is how
+research is configured right now:
+
+    research.torproject.org:
+      master: static-master-fsn.torproject.org
+      source: static-gitlab-shim.torproject.org:/srv/static-gitlab-shim/research.torproject.org/public
+
+It was migrated from Jenkins with a commit like this:
 
     modified   modules/roles/misc/static-components.yaml
     @@ -99,7 +99,7 @@ components:
@@ -87,17 +132,37 @@ the above case:
         ssh $host.torproject.org puppet agent --test
     done
 
-The site should now be deployed from GitLab CI. 
+The next pipeline should now succeed in deploying the site in GitLab.
 
-Make sure to [remove the old Jenkins job](service/jenkins#removing-a-job) and make sure the old
-site is cleared out from the previous static source:
+If the site is migrated from Jenkins, make sure to [remove the old
+Jenkins job](service/jenkins#removing-a-job) and make sure the old site is cleared out from the
+previous static source:
 
     ssh staticiforme.torproject.org rm -rf /srv/research.torproject.org/
 
 Typically, you will also want to [archive the git repository](howto/git#archiving-a-repository) if
 it hasn't already been [migrated to GitLab](howto/gitlab#how-to-migrate-a-git-repository-from-legacy-to-gitlab).
 
-### Deploying an old Hugo site
+## Building a Hugo site
+
+Normally, you should be able to deploy a hugo site by including the
+template and setting a few variables. This `.gitlab-ci.yml` file
+should be sufficient:
+
+```
+include:
+  project: tpo/tpa/ci-templates
+  file: static-shim-deploy.yml
+
+variables:
+  GIT_SUBMODULE_STRATEGY: recursive
+  SITE_URL: research.torproject.org
+  SUBDIR: public/
+```
+
+See below if this is an old hugo site, however.
+
+### Building an old Hugo site
 
 Unfortunately, because `research.torproject.org` was built a long time
 ago, newer Hugo releases broke its theme and the newer versions
@@ -117,8 +182,8 @@ different build system or Docker images:
 image: registry.hub.docker.com/library/debian:buster
 
 include:
-  project: tpo/tpa/status-site
-  file: .gitlab-ci.yml
+  project: tpo/tpa/ci-templates
+  file: static-shim-deploy.yml
 
 variables:
   GIT_SUBMODULE_STRATEGY: recursive