Skip to content
Snippets Groups Projects
New look: Off

Deploy new PuppetDB server

    • Closed Issue created by Jérôme Charaoui @lavamind

    As part of #40696 (closed) we decided to deploy a new PuppetDB server alongside the old puppet master.

    The hope is we can make progress on the Puppet upgrade plan by replacing the current PuppetDB running on pauli with this new one.

    Designs

      Child items ...

      2. Activity

      • Jérôme Charaoui changed milestone to %Debian 12 bookworm upgrade

        changed milestone to %Debian 12 bookworm upgrade

      • Jérôme Charaoui added Puppet lifecycle labels

        added Puppet lifecycle labels

      • Jérôme Charaoui assigned to @lavamind

        assigned to @lavamind

      • Jérôme Charaoui added Doing label

        added Doing label

      • anarcat marked this issue as related to #40696 (closed)

        marked this issue as related to #40696 (closed)

      • anarcat mentioned in issue #41321 (closed)

        mentioned in issue #41321 (closed)

      • Jérôme Charaoui
        Jérôme Charaoui @lavamind ·
        Author Owner

        The machine has been deployed on gnt-dal and a new role/class pair specifically for managing PuppetDB has been pushed to tor-puppet. So far it seems to be working great!

        Tomorrow I'll start testing data migration from the old PuppetDB.

        Edited by Jérôme Charaoui
      • anarcat mentioned in issue #41311 (closed)

        mentioned in issue #41311 (closed)

      • Jérôme Charaoui
        Jérôme Charaoui @lavamind ·
        Author Owner

        I copied the PuppetDB database from pauli to puppetdb-01, using pg_dump. At startup PuppetDB automatically executed the necessary database migrations, and did so successfully.

      • Jérôme Charaoui
        Jérôme Charaoui @lavamind ·
        Author Owner

        On pauli, I've added submit_only_server_urls = https://puppetdb-01.torproject.org:8081 to /etc/puppet/puppetdb.conf to test whether our current Puppet master is able to submit reports and catalog to the new PuppetDB and it appears this is also working without issues. Upon enabling HTTP session logging I can see submissions being received and processed without any sign of errors.

      • Jérôme Charaoui
        Jérôme Charaoui @lavamind ·
        Author Owner

        To view the PuppetDB dashboard, one can ssh -NL8080:localhost:8080 puppetdb-01.torproject.org & and open http://localhost:8080/pdb/dashboard/index.html in a browser. It's also possible to use that connection for cumin queries.

        • Jérôme Charaoui
          Jérôme Charaoui @lavamind ·
          Author Owner

          I've disabled the puppet across all nodes, switched to the new PuppetDB and tested:

          puppet agent --enable && time puppet agent --test --noop && puppet agent --disable "testing new bookworm puppetdb endpoint"

          This works. Resources are found and applied as expected. Yay!

          However, using the new PuppetDB, puppet agent --test is noticeably slower, approximately 15 to 20 seconds more per run, which is a bit confounding. I'm not sure if this is due to PuppetDB itself, PostgreSQL, Puppet master, or what. Perhaps at least part of the explanation is that pauli is on gnt-fsn and puppetdb-01 is on gnt-dal. I'm wondering if it would be worth it to move pauli to gnt-dal?

        • Jérôme Charaoui
          Jérôme Charaoui @lavamind ·
          Author Owner

          At @anarcat's suggestions I picked one of the queries logged by PuppetDB and figured out an equivalent curl command.

          Locally it's near instantaneous, and it's not noticeably slower when requested remotely, on pauli.

          I'm enclined to chalk this one up to our very old Puppet master... With any luck upgrading to 5.5.23 might help?

        • Jérôme Charaoui
          Jérôme Charaoui @lavamind ·
          Author Owner

          I ran more tests, enabling --debug on the Puppet master.

          With puppetdb-01 :

          puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Profile::Prometheus::Node_exporter])) Collected 1 Ferm::Rule resource in 0.47 seconds
puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Nagios::Client])) Collected 1 Concat::Fragment resource in 0.42 seconds
puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Nagios::Client])) Collected 1 Concat::Fragment resource in 0.44 seconds
puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Nagios::Client])) Collected 1 Ferm::Rule resource in 0.43 seconds
puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Nagios::Client])) Collected 1 Ferm::Rule resource in 0.42 seconds
puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Nagios::Client])) Collected 1 Ferm::Rule resource in 0.42 seconds
puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Ssh])) Collected 89 Ferm::Rule::Simple resources in 0.48 seconds
puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Bacula::Client])) Collected 1 Bacula::Client::Director resource in 0.44 seconds
puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Bacula::Client])) Collected 1 Ferm::Rule::Simple resource in 0.44 seconds
puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Profile::Gitlab::Runner])) Collected 1 Ferm::Rule resource in 0.42 seconds
puppet-master[8458]: Creating new connection for https://puppetdb-01.torproject.org:8081
puppet-master[8458]: (Scope(Class[Profile::Gitlab::Runner])) Collected 1 Ferm::Rule::Simple resource in 0.44 seconds

          With the current PuppetDB:

          puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Profile::Prometheus::Node_exporter])) Collected 1 Ferm::Rule resource in 0.02 seconds
puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Nagios::Client])) Collected 1 Concat::Fragment resource in 0.02 seconds
puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Nagios::Client])) Collected 1 Concat::Fragment resource in 0.02 seconds
puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Nagios::Client])) Collected 1 Ferm::Rule resource in 0.01 seconds
puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Nagios::Client])) Collected 1 Ferm::Rule resource in 0.01 seconds
puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Nagios::Client])) Collected 1 Ferm::Rule resource in 0.02 seconds
puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Ssh])) Collected 89 Ferm::Rule::Simple resources in 0.03 seconds
puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Bacula::Client])) Collected 1 Bacula::Client::Director resource in 0.02 seconds
puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Bacula::Client])) Collected 1 Ferm::Rule::Simple resource in 0.01 seconds
puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Profile::Gitlab::Runner])) Collected 1 Ferm::Rule resource in 0.01 seconds
puppet-master[8985]: Creating new connection for https://puppet.torproject.org:8081
puppet-master[8985]: (Scope(Class[Profile::Gitlab::Runner])) Collected 1 Ferm::Rule::Simple resource in 0.01 seconds
        • Jérôme Charaoui
          Jérôme Charaoui @lavamind ·
          Author Owner

          And now curl -o /dev/null -w "%{time_connect},%{time_total},%{speed_download},%{http_code},%{size_download},%{url_effective}\n" on pauli:

          0.130676,0.748194,113491.000,200,84892,https://puppetdb-01.torproject.org:8081/pdb/query/v4/resources?query=[...]

          0.005655,0.025170,3395680.000,200,84892,https://puppet.torproject.org:8081/pdb/query/v4/resources?query=[...]

          And when I run the same command on puppetdb-01:

          0.006655,0.068492,1239444,200,84892,https://puppetdb-01.torproject.org:8081/pdb/query/v4/resources?query=

          Considering this I'm leaning towards pinning the blame on the gnt-dal and gnt-fsn connection.

        • Jérôme Charaoui
          Jérôme Charaoui @lavamind ·
          Author Owner

          To be clear, we've got a difference of ~750ms for one PuppetDB query, based on the origin of the query. There are a lot of PuppetDB queries in one Puppet run, so I'm not surprised this can add up to 15 seconds of extra delay.

          @anarcat Do you see any serious issues in migrating pauli to gnt-dal ?

        • anarcat
          anarcat @anarcat ·
          Owner

          That shouldn't be a problem, and you might even use the process to act as a backup: you could migrate the host to gnt-dal, then upgrade it there, and revert to the previous cluster if all goes to shit.

          Just an idea.

          ...

          On 2023-10-03 18:48:23, Jérôme Charaoui (@lavamind) wrote:

          @anarcat Do you see any serious issues in migrating pauli to gnt-dal ?

        • Please register or sign in to reply
      • Jérôme Charaoui mentioned in issue #41346 (closed)

        mentioned in issue #41346 (closed)

      • Jérôme Charaoui marked this issue as related to #41346 (closed)

        marked this issue as related to #41346 (closed)

        • Jérôme Charaoui
          Jérôme Charaoui @lavamind ·
          Author Owner

          With pauli now migrated to gnt-dal, and the Puppet master configured to talk to puppetdb-01, agent run timings are now similar.

          On pauli:

          0.004966,0.069372,1230217.000,200,84885,https://puppetdb-01.torproject.org:8081/pdb/query/v4/resources?query=

          Total query time is now ~6-7ms instead of ~2-3ms, which I think is acceptable (and probably inevitable).

        • anarcat
          anarcat @anarcat ·
          Owner

          [...]

          Total query time is now ~6-7ms instead of ~2-3ms, which I think is acceptable (and probably inevitable).

          Just to make sure I understand what you're saying here: we have a 3-fold performance penalty after the upgrade, and that's inevitable because... upgrade? :)

          ...

          On 2023-10-04 14:29:52, Jérôme Charaoui (@lavamind) wrote:

        • Jérôme Charaoui
          Jérôme Charaoui @lavamind ·
          Author Owner

          I think labeling this as a "3-fold performance penalty" as you're doing here is quite unfair because individual query performance is not a useful metric: what we care about is the performance of puppet agent runs and this has not changed in any significant or perceptible way.

          And no, it's not just "because... upgrade?". We discussed the reasons previously:

          • is recommended by upstream
          • breaks down the already-complex puppet upgrade roadmap
          • immediately fixes (afaict) the intermittent agent run failures
          • immediately moves us away from an eol puppetdb
          • opens the door to implementing a second puppetdb node for reliability
        • Jérôme Charaoui
          Jérôme Charaoui @lavamind ·
          Author Owner

          And to be precise, I was saying inevitable because the Puppet master <-> PuppetDB connections are now hitting the (real-world) wire and opposed to being local to the same machine.

        • anarcat
          anarcat @anarcat ·
          Owner

          Sorry, I was a little harsh there. I was just trying to understand the underlying cause and impact.

          So what are those? Cause is moving PuppetDB to a different VM?

          Impact on run time is how much? twice slower?

          I don't mean to question the track we've taken, I still think it's the right direction. There's probably many optimizations we can make to the setup if we want to improve runtime from here on too.

          ...

          On 2023-10-04 16:30:16, Jérôme Charaoui (@lavamind) wrote:

          And to be precise, I was saying inevitable because the Puppet master <-> PuppetDB connections are now hitting the (real-world) wire and opposed to being local to the same machine.

        • Jérôme Charaoui
          Jérôme Charaoui @lavamind ·
          Author Owner

          So what are those? Cause is moving PuppetDB to a different VM?

          Yes.

          Impact on run time is how much? twice slower?

          Negligible, the puppet agent runs are now more of less the same now that we move pauli to gnt-dal, in the same ganeti cluster as puppetdb-01.

        • anarcat
          anarcat @anarcat ·
          Owner

          Awesome, thanks!

        • Please register or sign in to reply
      • Jérôme Charaoui mentioned in commit wiki-replica@4e8a2058

        mentioned in commit wiki-replica@4e8a2058

      • Jérôme Charaoui mentioned in commit wiki-replica@2836159b

        mentioned in commit wiki-replica@2836159b

      • Jérôme Charaoui closed

        closed

      • anarcat marked this issue as related to #41819 (closed)

        marked this issue as related to #41819 (closed)

      • anarcat mentioned in issue #41819 (closed)

        mentioned in issue #41819 (closed)

      Please register or sign in to reply