Changes

Page history
update deploy documentation to abstract SSG away authored by anarcat's avatar anarcat 
This is part of #40364
Show whitespace changes
Inline Side-by-side
service/static-shim.md
View page @ da38d9a0
......@@ -16,24 +16,51 @@ TODO: "how do users add/remove sites"
TODO: review ticket for possible howtos
## Deploying a Hugo site
## Deploying a static site from GitLab CI
Normally, you should be able to deploy a hugo site by including the
template and setting a few variables. This `.gitlab-ci.yml` file
should be sufficient:
First, you will need to make sure the site builds in GitLab CI. A
`build` stage MUST be used that will produce artifacts that can be
used by the `deploy` job provided in the [`static-shim-deploy.yml`
template][]. How to build the website will vary according to the site,
obviously. See the [hugo build instructions below](#building-a-hugo-site).
[`static-shim-deploy.yml` template]: https://gitlab.torproject.org/tpo/tpa/ci-templates/-/blob/main/static-shim-deploy.yml
TODO: link to documentation on how to build Lektor sites.
It is a good idea to also add a `pages` stage to preview the
build. The above template has an example `pages` stage.
Then include the deploy job template in the `.gitlab-ci.yml` with a
snippet like this:
```
variables:
SITE_URL: example.torproject.org
SUBDIR: public/
include:
project: tpo/tpa/status-site
file: .gitlab-ci.yml
project: tpo/tpa/ci-templates
file: static-shim-deploy.yml
```
The `SITE_URL` and `SUBDIR` parameters need to be changed to reflect,
respectively, the FQDN of the website as defined in the
`static-components.yml` file, and the directory where the website was
built by the previous build stage. You may also need
`GIT_SUBMODULE_STRATEGY: recursive` if you have submodules (e.g. hugo
themes are often shipped as submodules). For example, for
<https://status.torproject.org>, the `.gitlab-ci.yml` file looks like
this (`build` stage elided for simplicity):
variables:
GIT_SUBMODULE_STRATEGY: recursive
SITE_URL: research.torproject.org
SITE_URL: status.torproject.org
SUBDIR: public/
```
See below if this is an old hugo site, however.
include:
project: tpo/tpa/ci-templates
file: static-shim-deploy.yml
Then you also need to set an SSH key in the project. First, generate a
password-less key locally:
......@@ -52,22 +79,40 @@ variable`, with the following parameters:
Then the *public* part of that key needs to be added in Puppet. This
can only be done by TPA, so file a ticket there if you need
assistance. The key should be added in the `tor-puppet.git`
repository, in the `hiera/common.yaml` file, in the
`staticsync::gitlab_shim::ssh::sites` hash, in which the site URL is
the key and the public key (only the key part, no `ssh-rsa` prefix or
comment suffix) is the value. For example, this is the entry for
`status.torproject.org`:
assistance. For TPA, see below for the remaining instructions.
You can commit the above changes to the `.gitlab-ci.yml` file, but
when pushed, the pipeline's `deploy` stage is normal, TPA needs to do
its magic for the deploy to work. Make sure the build works in GitLab
pages before requesting the deploy in the static mirror system.
### TPA: adding a new static site shim in Puppet
The public key mentioned above should be added in the `tor-puppet.git` repository, in the
`hiera/common.yaml` file, in the `staticsync::gitlab_shim::ssh::sites`
hash.
There, the site URL is the key and the public key (only the key part,
no `ssh-rsa` prefix or comment suffix) is the value. For example, this
is the entry for `status.torproject.org`:
staticsync::gitlab_shim::ssh::sites:
status.torproject.org: "AAAAB3NzaC1yc2EAAAADAQABAAABgQC3mXhQENCbOKgrhOWRGObcfqw7dUVkPlutzHpycRK9ixhaPQNkMvmWMDBIjBSviiu5mFrc6safk5wbOotQonqq2aVKulC4ygNWs0YtDgCtsm/4iJaMCNU9+/78TlrA0+Sp/jt67qrvi8WpLF/M8jwaAp78s+/5Zu2xD202Cqge/43AhKjH07TOMax4DcxjEzhF4rI19TjeqUTatIuK8BBWG5vSl2vqDz2drbsJvaLbjjrfbyoNGuK5YtvI/c5FkcW4gFuB/HhOK86OH3Vl9um5vwb3DM2HVMTiX15Hw67QBIRfRFhl0NlQD/bEKzL3PcejqL/IC4xIJK976gkZzA0wpKaE7IUZI5yEYX3lZJTTGMiZGT5YVGfIUFQBPseWTU+cGpNnB4yZZr4G4o/MfFws4mHyh4OAdsYiTI/BfICd3xIKhcj3CPITaKRf+jqPyyDJFjEZTK/+2y3NQNgmAjCZOrANdnu7GCSSz1qkHjA2RdSCx3F6WtMek3v2pbuGTns="
Note that this will only allow GitLab CI to push to the shim, but will
still fail to deploy because the static-component configuration does
not match. Once you are satisfied your build works (try on a branch
first to see if the GitLab pages version renders correctly), the
static-component should be changed to point to the shim, with a commit
like this:
At this point, the deploy job should be able to `rsync` the content to
the static shim, but the deploy will still fail because the
static-component configuration does not match and the
`static-update-component` step will fail.
To fix this, the static-component entry should be added (or modified,
if it already exists) to point to the shim. This, for example, is how
research is configured right now:
research.torproject.org:
master: static-master-fsn.torproject.org
source: static-gitlab-shim.torproject.org:/srv/static-gitlab-shim/research.torproject.org/public
It was migrated from Jenkins with a commit like this:
modified modules/roles/misc/static-components.yaml
@@ -99,7 +99,7 @@ components:
......@@ -87,17 +132,37 @@ the above case:
ssh $host.torproject.org puppet agent --test
done
The site should now be deployed from GitLab CI.
The next pipeline should now succeed in deploying the site in GitLab.
Make sure to [remove the old Jenkins job](service/jenkins#removing-a-job) and make sure the old
site is cleared out from the previous static source:
If the site is migrated from Jenkins, make sure to [remove the old
Jenkins job](service/jenkins#removing-a-job) and make sure the old site is cleared out from the
previous static source:
ssh staticiforme.torproject.org rm -rf /srv/research.torproject.org/
Typically, you will also want to [archive the git repository](howto/git#archiving-a-repository) if
it hasn't already been [migrated to GitLab](howto/gitlab#how-to-migrate-a-git-repository-from-legacy-to-gitlab).
### Deploying an old Hugo site
## Building a Hugo site
Normally, you should be able to deploy a hugo site by including the
template and setting a few variables. This `.gitlab-ci.yml` file
should be sufficient:
```
include:
project: tpo/tpa/ci-templates
file: static-shim-deploy.yml
variables:
GIT_SUBMODULE_STRATEGY: recursive
SITE_URL: research.torproject.org
SUBDIR: public/
```
See below if this is an old hugo site, however.
### Building an old Hugo site
Unfortunately, because `research.torproject.org` was built a long time
ago, newer Hugo releases broke its theme and the newer versions
......@@ -117,8 +182,8 @@ different build system or Docker images:
image: registry.hub.docker.com/library/debian:buster
include:
project: tpo/tpa/status-site
file: .gitlab-ci.yml
project: tpo/tpa/ci-templates
file: static-shim-deploy.yml
variables:
GIT_SUBMODULE_STRATEGY: recursive
......
......