Skip to content
Snippets Groups Projects

switch from librarian to g10k

    • Closed Issue created by anarcat @anarcat

    in #40861 (comment 3142097), we agreed we'd use g10k to manage third-party repositories, let's do it.

    i have hooks locally to do this.

    specifically, the proposal is what @lavamind wrote in wiki-replica@467aa42c

    i think the next steps are:

    1. locally, delete all of 3rdparty/modules
    2. run cd 3rdparty && g10k -puppetfile -moduledir modules/ to repopulate the directory
    3. check that the actual checkout is unchanged, or audit if changes are okay (e.g. metadata from tarballs might have been removed, that's fine)
    4. pin all checksums in the Puppetfile, either with the checksum of the tarball or a git commit, see my puppetfile for an example, might be possible to extract this from the lock file nope, lock files don't have checksums
    5. rerun g10k, check git status, should be a noop
    6. delete 3rdparty/modules from git, for real

    The above should be done in a branch, most likely.

    Once that's done, the branch is ready to be deployed on the server. The server needs to be configured to deploy 3rdparty modules to the right location. I'm not sure how that works with environments, but I'm using this hook to run g10k, essentially only if the Puppetfile is changed.

    @lavamind, i can do the git branch, could you see how to integrate g10k on the server side?

    feel free to reassign to me after checking.

    To do

    • switch from librarian to g10k
    • fix notifications and branch protection
    • move pauli back to the production environment
    • switch tor-puppet-hiera-enc.git to also use the production branch
    • notify tpa-team@ about the switch
    • check if doc needs updating
    ✓ 6 of 6 checklist items completed · Edited

    Designs

      Child items 0

      2. No child items are currently assigned. Use child items to break down this issue into smaller parts.

      Activity

      • anarcat marked this issue as related to #40861 (closed)

        marked this issue as related to #40861 (closed)

      • anarcat marked this issue as related to #29387

        marked this issue as related to #29387

      • anarcat changed milestone to %Puppet CI

        changed milestone to %Puppet CI

      • anarcat added Next Puppet Technical Debt labels

        added Next Puppet Technical Debt labels

      • anarcat assigned to @anarcat

        assigned to @anarcat

      • anarcat mentioned in issue #40861 (closed)

        mentioned in issue #40861 (closed)

      • anarcat added Needs Review label and removed Next label

        added Needs Review label and removed Next label

      • anarcat assigned to @lavamind and unassigned @anarcat

        assigned to @lavamind and unassigned @anarcat

      • anarcat changed the description

        changed the description

      • anarcat changed the description

        changed the description

        • anarcat
          anarcat @anarcat ·
          Author Owner
          Resolved by zen

          run cd 3rdparty && g10k -puppetfile -moduledir modules/ to repopulate the directory

          fwiw, a test run of this deletes all sorts of dependencies not explicitly mentioned in the puppetfile, namely:

          • archive
          • concat
          • epel
          • firewall
          • inifile
          • stdlib
          • translate

          g10k also doesn't complete: it seems to hang on some HTTP request, presumably as it fails to contact the forge for latest release information for some modules that don't have specific version info.

      • anarcat
        anarcat @anarcat ·
        Author Owner

        pin all checksums in the Puppetfile, either with the checksum of the tarball or a git commit

        i've done some of this: only the git checksums for now to retain compatibility with librarian (although I haven't bothered updating the lockfile since the point is to get rid of it anyways).

      • Jérôme Charaoui mentioned in issue #41983 (closed)

        mentioned in issue #41983 (closed)

      • Jérôme Charaoui marked this issue as related to #41983 (closed)

        marked this issue as related to #41983 (closed)

        • Jérôme Charaoui
          Jérôme Charaoui @lavamind ·
          Owner

          LGTM. I can look into wiring g10k into the post-receive hooks. I think the main difference with your version is that we need to execute it in the proper environment.

          I'm also wondering if we should instead first propose this functionality upstream, in the main environment deployment hook.

        • anarcat
          anarcat @anarcat ·
          Author Owner

          LGTM. I can look into wiring g10k into the post-receive hooks. I think the main difference with your version is that we need to execute it in the proper environment.

          I'd very much welcome your help there, I'm a bit lost as to how environments are deployed, and how they would interact with g10k.

          I'm also wondering if we should instead first propose this functionality upstream, in the main environment deployment hook.

          Yeah, that sounds reasonable: I'm actually surprised that's not already featured in there...

        • anarcat
          anarcat @anarcat ·
          Author Owner

          LGTM. I can look into wiring g10k into the post-receive hooks. I think the main difference with your version is that we need to execute it in the proper environment.

          looks like there's nothing upstream for g10k. there's this PR with noises about integrating r10k, so that would be a starting point:

          https://github.com/theforeman/puppet-puppet/pull/853/files

          oooor we go back to submodules. :p

        • Please register or sign in to reply
        • anarcat
          anarcat @anarcat ·
          Author Owner
          Resolved by zen

          also, for the record, i'm not sure converting to g10k is worth it.

          to be more specific:

          to do g10k securely, we need to populate the puppetfile with hashes for everything. that's going to be a lot of work and a significant pain in the ass.

          we'll also need to double-commit to the puppetfile anytime we need to make a change to a module listed in there, also a huge PITA. at that point, i wonder why we're not using submodules, to be honest.

          we could still use g10k (as opposed to librarian) to manage the modules checked in as submodules though...

          sorry to reopen that can of worms, but i'm a little hesitant in jumping into rewriting the puppetfile...

        • anarcat
          anarcat @anarcat ·
          Author Owner
          Resolved by anarcat

          i'm less and less sure than neither approach of using submodules or g10k with a blank 3rdparty/modules/ directory makes sense:

          1. submodules will be slow
          2. g10k with pins will take a long time to setup, and a pain to maintain

          I wonder if a better approach might not be to simply keep our current monorepo, basically as is, but:

          1. move modules from 3rdparty/modules to modules/
          2. keep the Puppetfile, but only to keep maintaining the modules directory with g10k

          I also note that we could just get rid of the Puppetfile altogether and use the regular puppet module subcommand to maintain modules. This, for example, is what it has to say about our current module set:

          anarcat@angela:~/s/t/tor-puppet> puppet module --modulepath 3rdparty/modu
les/ list
Warning: Module 'puppet-systemd' (v4.2.0) fails to meet some dependencies:
  'puppet-prometheus' (v12.4.0) requires 'puppet-systemd' (>= 1.1.1 < 4.0.0)
/home/anarcat/src/tor/tor-puppet/3rdparty/modules
├── LeLutin-fail2ban (v4.0.1)
├── attestra-pgbackrest (v1.0.0)
├── bzed-dehydrated (v0.1.11)
├── cirrax-postfix (v2.1.1)
├── dalen-puppetdbquery (v3.0.1)
├── duritong-trocla (v1.0.1)
├── edestecd-clamav (v2.0.0)
├── fortunes (???)
├── hetzner-needrestart (v1.0.4)
├── mtail (???)
├── oxc-dovecot (v3.1.0)
├── oxc-rspamd (v1.3.0)
├── puppet-archive (v4.6.0)
├── puppet-epel (v3.1.0)
├── puppet-extlib (v6.2.0)
├── puppet-gitlab (v5.0.0)
├── puppet-gitlab_ci_runner (v3.0.0)
├── puppet-grafana (v8.0.0)
├── puppet-kmod (v3.2.0)
├── puppet-nginx (v3.0.1-rc0)
├── puppet-prometheus (v12.4.0)
├── puppet-prometheus_reporter (v2.0.0)
├── puppet-ssh_keygen (v6.0.0)
├── puppet-systemd (v4.2.0)  invalid
├── puppet-unattended_upgrades (v8.1.0)
├── puppetlabs-apt (v7.6.0)
├── puppetlabs-augeas_core (v1.5.0)
├── puppetlabs-concat (v6.4.0)
├── puppetlabs-cron_core (v1.2.0)
├── puppetlabs-firewall (v3.6.0)
├── puppetlabs-inifile (v4.4.0)
├── puppetlabs-mailalias_core (v1.1.0)
├── puppetlabs-mysql (v13.0.1)
├── puppetlabs-postgresql (v6.10.2)
├── puppetlabs-puppetdb (v7.13.0)
├── puppetlabs-sshkeys_core (v2.5.0)
├── puppetlabs-stdlib (v6.6.0)
├── puppetlabs-translate (v2.2.0)
├── puppetlabs-vcsrepo (v6.1.0)
├── saz-sudo (v7.0.2)
├── smash-ganeti (v1.0.8)
├── smash-kgb (v0.1.0)
├── smash-tor (v3.1.1)
└── theforeman-puppet (v18.0.0)

          It doesn't show which modules are out of date, but neither does g10k or submodules or any solution proposed so far. One could run puppet module upgrade MODULE to upgrade a module, for example to get rid of the above warning, i could do:

          puppet module --modulepath 3rdparty/modules/ upgrade puppet-prometheus

          plop!

          so i think i'll make the case for keeping the monorepo, pretty much as is, except maybe:

          1. removing the librarian lockfile (and ditching librarian, which i think we all agree on, but haven't done yet)
          2. removing the Puppetfile (and using only puppet module to manage modules)
          3. OR keeping the Puppetfile (and using g10k to manage modules)

          i tend to lean towards the latter, because i like g10k's -clonegit flag, but the problem with g10k right now is it just hangs, even on -help (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1093015)... so i feel we might want to just ignore the entire puppefile stuff.. but then again, puppet module install can't checkout git repositories and it relies on md5 (!?) for integrity (ugh?!)...

          thoughts?

      • anarcat
        anarcat @anarcat ·
        Author Owner

        we discussed this at our meeting today, and the next steps are:

        1. set checksums for every entry in the Puppetfile
        2. add missing dependencies in the Puppetfile, at this point, running g10k -puppetfile -clonegit on the repository is a noop
        3. replace environment deployment hook with some g10k wrapper
      • anarcat added Next label and removed Needs Review label

        added Next label and removed Needs Review label

      • anarcat assigned to @zen and unassigned @lavamind

        assigned to @zen and unassigned @lavamind

      • zen added Needs Review label and removed Next label

        added Needs Review label and removed Next label

      • anarcat
        anarcat @anarcat ·
        Author Owner

        comment moved to the right place, the merge rquest in https://gitlab.torproject.org/tpo/tpa/puppet-control/-/merge_requests/22

        Edited by anarcat
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      Please register or sign in to reply