2020 - 06 ($41) · Snippets

Crm-int-01 had some obsolete packages that needed a clean up.

The unanttended upgrade puppet module is now tracking upstream again instead of the git repository.

Updaded gitlab-ce and jenkins during the weekend.

Filled bug with openvswitch-switch debian package. (https://trac.torproject.org/projects/tor/ticket/34185)

Onion-Location header (https://trac.torproject.org/projects/tor/ticket/33705)

Gitlab now allows now to reply to tickets via email

Create new ldap account for opara (https://trac.torproject.org/projects/tor/ticket/34074)

RT aliases for newsletter and giving https://trac.torproject.org/projects/tor/ticket/34074

Started to work on gitlab monitoring with prometheus.

https://trac.torproject.org/projects/tor/ticket/33921

diff --git a/modules/profile/manifests/gitlab/app.pp b/modules/profile/manifests/gitlab/app.pp
index 44447a97..ac7744aa 100644
--- a/modules/profile/manifests/gitlab/app.pp
+++ b/modules/profile/manifests/gitlab/app.pp
@@ -45,6 +45,9 @@ class profile::gitlab::app {
         incoming_email_port       => 143,
         incoming_email_email      => 'git@gitlab.torproject.org',
         incoming_email_password   => "${trocla('gitlab_mail_password', 'plain')}",
+        monitoring_whitelist      => [
+          '195.201.139.202',
+        ],
       },
       grafana                     => {
         enable => false,
@@ -55,6 +58,27 @@ class profile::gitlab::app {
       node_exporter               => {
         enable => false,
       },
+      gitlab_exporter             => {
+        enable         => true,
+        listen_address => '0.0.0.0:9168',
+      },
+      sidekiq                     => {
+        listen_address => '0.0.0.0',
       node_exporter               => {
         enable => false,
       },
+      gitlab_exporter             => {
+        enable         => true,
+        listen_address => '0.0.0.0:9168',
+      },
+      sidekiq                     => {
+        listen_address => '0.0.0.0',
+      },
+      redis_exporter              => {
+        enable         => true,
+        listen_address => '0.0.0.0:9121',
+      },
+      postgres_exporter           => {
+        enable         => true,
+        listen_address => '0.0.0.0:9187',
+      },
+      gitaly                      => {
+        prometheus_listen_addr => '0.0.0.0:9236'.
+      },
+      gitlab_workhorse            => {
+        prometheus_listen_addr => '0.0.0.0:9229'
+      },
   }
 
   file {
diff --git a/modules/profile/manifests/prometheus/server/internal.pp b/modules/profile/manifests/prometheus/server/internal.pp
index 6ae3f838..a33d492a 100644
--- a/modules/profile/manifests/prometheus/server/internal.pp
+++ b/modules/profile/manifests/prometheus/server/internal.pp
@@ -1,6 +1,6 @@
 # a prometheus monitoring server, which collects samples from multiple targets
 #
-# @param vhost_name the name of the "vanity" virtual host for this service 
+# @param vhost_name the name of the "vanity" virtual host for this service
 class profile::prometheus::server::internal (
   String $vhost_name = $::fqdn,
 ) {
@@ -38,6 +38,10 @@ class profile::prometheus::server::internal (
     { 'job_name' => 'postfix' },
     { 'job_name' => 'postgres' },
     { 'job_name' => 'mtail' },
+    { 'job_name' => 'gitlab_exporter' },
+    { 'job_name' => 'redis_exporter' },
+    { 'job_name' => 'gitaly' },
+    { 'job_name' => 'gitlab_workhorse' },
   ]
   class { 'profile::prometheus::server::common':
     vhost_name          => $vhost_name,
@@ -57,5 +61,9 @@ class profile::prometheus::server::internal (
     'postgres': port => 9187;
     'bind': port => 9119;
     'mtail': port => 3903;
+    'gitlab_exporter': port => 9168;
+    'redis_exporter': port => 9121;
+    'gitaly': port => 9236;
+    'gitlab_workhorse': port => 9229;
   }
 }

Moved the gitlab exporter to the static targets.

$scrape_configs =
  [
    {
      'job_name'       => 'prometheus',
      'static_configs' => [
        {
          'targets' => ['localhost:9090'],
          'labels'  => {
            'alias' => 'Prometheus',
          },
        },
      ],
    },
    {
      'job_name'       => 'grafana',
      'static_configs' => [
        {
          'targets' => ['localhost:3000'],
          'labels'  => {
            'alias' => 'Grafana',
          },
        },
      ],
    },
    {
      'job_name'       => 'gitlab_exporter',
      'static_configs' => [
        {
          'targets' => ['gitlab-02.torproject.org:9168'],
          'labels'  => {
            'alias' => 'Gitlab-Exporter',
          },
        },
      ],
    },
  ]

But looks like gitlab-ce package is getting in the middle of exporting this endpoint. Since on gitlab-02 I see:

down: gitlab-exporter: 1s, normally up, want up; run: log: (pid 32306) 342718s

Updated gpg key for boklm: https://trac.torproject.org/projects/tor/ticket/34348

And updated documentation on the wiki.

Finally added the node and the gitlab exporters to prom.

https://trac.torproject.org/projects/tor/ticket/33921

Apparently the correct syntax was:

  gitlab_exporter             => {
        enable         => true,
        listen_address => '0.0.0.0',
        listen_port    => '9178',
      },

Wondering why different exporters have a different syntax in the same gitlab configs.

Exported gitaly and gitlab dashboard to gradana. See https://gitlab.com/gitlab-org/grafana-dashboards/tree/master/omnibus Should be added to puppet.

Gitlab dashboard: https://grafana.torproject.org/d/QrDJktiMz/gitlab-omnibus?orgId=1&refresh=5m

Gitaly dashboard: https://grafana.torproject.org/d/x6Z50y-iz/gitlab-gitaly?orgId=1&refresh=1m

Gitlab Node dashboard: https://grafana.torproject.org/d/Z7T7Cfemz/node-exporter-full?orgId=1&var-job=gitlab&var-node=gitlab-02.torproject.org&var-port=9101

Installed prometheus-nginx-exporter on gitlab-02. The exporter doesn't run as daemon so it needs to be configured.

Gitlab-02 is having disk issues. Needs more space.

A number of ganeti nodes and instances went down last night at 2 UTC. Not sure yet why. Investigating. Checking the last reboot time doesn't look like these have been rebooted. Apparently it was an issue with hetzner.

Type: 	Fault report 	
Categories: 	Network
Start: 	June 9, 2020 6:45:00 PM CEST
End: 	June 9, 2020 6:55:00 PM CEST
Description: 	We are currently experiencing a fault on our switches fsn1-dc7-sw_718 and fsn1-dc7-sw_719. Our technicians are working hard on analyzing the cause of the interruption in order to solve the issue.

Please bear with us. As soon as more information is available, we shall inform you via this website.

We apologize for any inconvenience. Thank you for your understanding.
Update: 	June 9, 2020 6:55:00 PM CEST 	
	The fault has been fixed.

All systems are accessible again

Working on backups for gitlab:

https://trac.torproject.org/projects/tor/ticket/34420#ticket

Sysadmin sync today. Updated tickets and roadmap.

Gitlab wasn't sending backups to the right folder and was filling up its disk. Also gitlab-backup dir needs to be owned by the git user.

Added new onionperf hosts entries on the DNS.

Some last minute gitlab configuration.

Doing the reboot dance.