switch from librarian to g10k

in #40861 (comment 3142097), we agreed we'd use g10k to manage third-party repositories, let's do it.

i have hooks locally to do this.

specifically, the proposal is what @lavamind wrote in wiki-replica@467aa42c

i think the next steps are:

1. locally, delete all of 3rdparty/modules
2. run cd 3rdparty && g10k -puppetfile -moduledir modules/ to repopulate the directory
3. check that the actual checkout is unchanged, or audit if changes are okay (e.g. metadata from tarballs might have been removed, that's fine)
4. pin all checksums in the Puppetfile, either with the checksum of the tarball or a git commit, see my puppetfile for an example, might be possible to extract this from the lock file nope, lock files don't have checksums
5. rerun g10k, check git status, should be a noop
6. delete 3rdparty/modules from git, for real

The above should be done in a branch, most likely.

Once that's done, the branch is ready to be deployed on the server. The server needs to be configured to deploy 3rdparty modules to the right location. I'm not sure how that works with environments, but I'm using this hook to run g10k, essentially only if the Puppetfile is changed.

@lavamind, i can do the git branch, could you see how to integrate g10k on the server side?

feel free to reassign to me after checking.

To do

• switch from librarian to g10k
• fix notifications and branch protection
• move pauli back to the production environment
• switch tor-puppet-hiera-enc.git to also use the production branch
• notify tpa-team@ about the switch
• check if doc needs updating

marked this issue as related to #40861 (closed)

marked this issue as related to #29387

changed milestone to %Puppet CI

added Next Puppet Technical Debt labels

assigned to @anarcat

mentioned in issue #40861 (closed)

added Needs Review label and removed Next label

assigned to @lavamind and unassigned @anarcat

changed the description

changed the description

run cd 3rdparty && g10k -puppetfile -moduledir modules/ to repopulate the directory

fwiw, a test run of this deletes all sorts of dependencies not explicitly mentioned in the puppetfile, namely:

• archive
• concat
• epel
• firewall
• inifile
• stdlib
• translate

g10k also doesn't complete: it seems to hang on some HTTP request, presumably as it fails to contact the forge for latest release information for some modules that don't have specific version info.

pin all checksums in the Puppetfile, either with the checksum of the tarball or a git commit

i've done some of this: only the git checksums for now to retain compatibility with librarian (although I haven't bothered updating the lockfile since the point is to get rid of it anyways).

mentioned in issue #41983 (closed)

marked this issue as related to #41983 (closed)

LGTM. I can look into wiring g10k into the post-receive hooks. I think the main difference with your version is that we need to execute it in the proper environment.

I'm also wondering if we should instead first propose this functionality upstream, in the main environment deployment hook.

LGTM. I can look into wiring g10k into the post-receive hooks. I think the main difference with your version is that we need to execute it in the proper environment.

I'd very much welcome your help there, I'm a bit lost as to how environments are deployed, and how they would interact with g10k.

I'm also wondering if we should instead first propose this functionality upstream, in the main environment deployment hook.

Yeah, that sounds reasonable: I'm actually surprised that's not already featured in there...

LGTM. I can look into wiring g10k into the post-receive hooks. I think the main difference with your version is that we need to execute it in the proper environment.

looks like there's nothing upstream for g10k. there's this PR with noises about integrating r10k, so that would be a starting point:

https://github.com/theforeman/puppet-puppet/pull/853/files

oooor we go back to submodules. :p

also, for the record, i'm not sure converting to g10k is worth it.

to be more specific:

to do g10k securely, we need to populate the puppetfile with hashes for everything. that's going to be a lot of work and a significant pain in the ass.

we'll also need to double-commit to the puppetfile anytime we need to make a change to a module listed in there, also a huge PITA. at that point, i wonder why we're not using submodules, to be honest.

we could still use g10k (as opposed to librarian) to manage the modules checked in as submodules though...

sorry to reopen that can of worms, but i'm a little hesitant in jumping into rewriting the puppetfile...

i'm less and less sure than neither approach of using submodules or g10k with a blank 3rdparty/modules/ directory makes sense:

1. submodules will be slow
2. g10k with pins will take a long time to setup, and a pain to maintain

I wonder if a better approach might not be to simply keep our current monorepo, basically as is, but:

1. move modules from 3rdparty/modules to modules/
2. keep the Puppetfile, but only to keep maintaining the modules directory with g10k

I also note that we could just get rid of the Puppetfile altogether and use the regular puppet module subcommand to maintain modules. This, for example, is what it has to say about our current module set:

anarcat@angela:~/s/t/tor-puppet> puppet module --modulepath 3rdparty/modu
les/ list
Warning: Module 'puppet-systemd' (v4.2.0) fails to meet some dependencies:
  'puppet-prometheus' (v12.4.0) requires 'puppet-systemd' (>= 1.1.1 < 4.0.0)
/home/anarcat/src/tor/tor-puppet/3rdparty/modules
├── LeLutin-fail2ban (v4.0.1)
├── attestra-pgbackrest (v1.0.0)
├── bzed-dehydrated (v0.1.11)
├── cirrax-postfix (v2.1.1)
├── dalen-puppetdbquery (v3.0.1)
├── duritong-trocla (v1.0.1)
├── edestecd-clamav (v2.0.0)
├── fortunes (???)
├── hetzner-needrestart (v1.0.4)
├── mtail (???)
├── oxc-dovecot (v3.1.0)
├── oxc-rspamd (v1.3.0)
├── puppet-archive (v4.6.0)
├── puppet-epel (v3.1.0)
├── puppet-extlib (v6.2.0)
├── puppet-gitlab (v5.0.0)
├── puppet-gitlab_ci_runner (v3.0.0)
├── puppet-grafana (v8.0.0)
├── puppet-kmod (v3.2.0)
├── puppet-nginx (v3.0.1-rc0)
├── puppet-prometheus (v12.4.0)
├── puppet-prometheus_reporter (v2.0.0)
├── puppet-ssh_keygen (v6.0.0)
├── puppet-systemd (v4.2.0)  invalid
├── puppet-unattended_upgrades (v8.1.0)
├── puppetlabs-apt (v7.6.0)
├── puppetlabs-augeas_core (v1.5.0)
├── puppetlabs-concat (v6.4.0)
├── puppetlabs-cron_core (v1.2.0)
├── puppetlabs-firewall (v3.6.0)
├── puppetlabs-inifile (v4.4.0)
├── puppetlabs-mailalias_core (v1.1.0)
├── puppetlabs-mysql (v13.0.1)
├── puppetlabs-postgresql (v6.10.2)
├── puppetlabs-puppetdb (v7.13.0)
├── puppetlabs-sshkeys_core (v2.5.0)
├── puppetlabs-stdlib (v6.6.0)
├── puppetlabs-translate (v2.2.0)
├── puppetlabs-vcsrepo (v6.1.0)
├── saz-sudo (v7.0.2)
├── smash-ganeti (v1.0.8)
├── smash-kgb (v0.1.0)
├── smash-tor (v3.1.1)
└── theforeman-puppet (v18.0.0)

It doesn't show which modules are out of date, but neither does g10k or submodules or any solution proposed so far. One could run puppet module upgrade MODULE to upgrade a module, for example to get rid of the above warning, i could do:

puppet module --modulepath 3rdparty/modules/ upgrade puppet-prometheus

plop!

so i think i'll make the case for keeping the monorepo, pretty much as is, except maybe:

1. removing the librarian lockfile (and ditching librarian, which i think we all agree on, but haven't done yet)
2. removing the Puppetfile (and using only puppet module to manage modules)
3. OR keeping the Puppetfile (and using g10k to manage modules)

i tend to lean towards the latter, because i like g10k's -clonegit flag, but the problem with g10k right now is it just hangs, even on -help (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1093015)... so i feel we might want to just ignore the entire puppefile stuff.. but then again, puppet module install can't checkout git repositories and it relies on md5 (!?) for integrity (ugh?!)...

thoughts?

we discussed this at our meeting today, and the next steps are:

1. set checksums for every entry in the Puppetfile
2. add missing dependencies in the Puppetfile, at this point, running g10k -puppetfile -clonegit on the repository is a noop
3. replace environment deployment hook with some g10k wrapper

added Next label and removed Needs Review label

assigned to @zen and unassigned @lavamind

added Needs Review label and removed Next label

comment moved to the right place, the merge rquest in https://gitlab.torproject.org/tpo/tpa/puppet-control/-/merge_requests/22