GitLab is used only for code review, issue tracking and project management. Canonical locations for source code are still https://gitweb.torproject.org/ https://git.torproject.org/ and git-rw.torproject.org.

  • Crm-int-01 had some obsolete packages that needed a clean up.

  • The unanttended upgrade puppet module is now tracking upstream again instead of the git repository.

  • Updaded gitlab-ce and jenkins during the weekend.

  • Filled bug with openvswitch-switch debian package. (https://trac.torproject.org/projects/tor/ticket/34185)

  • Gitlab now allows now to reply to tickets via email

  • Create new ldap account for opara (https://trac.torproject.org/projects/tor/ticket/34074)

  • RT aliases for newsletter and giving https://trac.torproject.org/projects/tor/ticket/34074

  • Started to work on gitlab monitoring with prometheus.

    https://trac.torproject.org/projects/tor/ticket/33921

    diff --git a/modules/profile/manifests/gitlab/app.pp b/modules/profile/manifests/gitlab/app.pp
    index 44447a97..ac7744aa 100644
    --- a/modules/profile/manifests/gitlab/app.pp
    +++ b/modules/profile/manifests/gitlab/app.pp
    @@ -45,6 +45,9 @@ class profile::gitlab::app {
             incoming_email_port       => 143,
             incoming_email_email      => 'git@gitlab.torproject.org',
             incoming_email_password   => "${trocla('gitlab_mail_password', 'plain')}",
    +        monitoring_whitelist      => [
    +          '195.201.139.202',
    +        ],
           },
           grafana                     => {
             enable => false,
    @@ -55,6 +58,27 @@ class profile::gitlab::app {
           node_exporter               => {
             enable => false,
           },
    +      gitlab_exporter             => {
    +        enable         => true,
    +        listen_address => '0.0.0.0:9168',
    +      },
    +      sidekiq                     => {
    +        listen_address => '0.0.0.0',
           node_exporter               => {
             enable => false,
           },
    +      gitlab_exporter             => {
    +        enable         => true,
    +        listen_address => '0.0.0.0:9168',
    +      },
    +      sidekiq                     => {
    +        listen_address => '0.0.0.0',
    +      },
    +      redis_exporter              => {
    +        enable         => true,
    +        listen_address => '0.0.0.0:9121',
    +      },
    +      postgres_exporter           => {
    +        enable         => true,
    +        listen_address => '0.0.0.0:9187',
    +      },
    +      gitaly                      => {
    +        prometheus_listen_addr => '0.0.0.0:9236'.
    +      },
    +      gitlab_workhorse            => {
    +        prometheus_listen_addr => '0.0.0.0:9229'
    +      },
       }
     
       file {
    diff --git a/modules/profile/manifests/prometheus/server/internal.pp b/modules/profile/manifests/prometheus/server/internal.pp
    index 6ae3f838..a33d492a 100644
    --- a/modules/profile/manifests/prometheus/server/internal.pp
    +++ b/modules/profile/manifests/prometheus/server/internal.pp
    @@ -1,6 +1,6 @@
     # a prometheus monitoring server, which collects samples from multiple targets
     #
    -# @param vhost_name the name of the "vanity" virtual host for this service 
    +# @param vhost_name the name of the "vanity" virtual host for this service
     class profile::prometheus::server::internal (
       String $vhost_name = $::fqdn,
     ) {
    @@ -38,6 +38,10 @@ class profile::prometheus::server::internal (
         { 'job_name' => 'postfix' },
         { 'job_name' => 'postgres' },
         { 'job_name' => 'mtail' },
    +    { 'job_name' => 'gitlab_exporter' },
    +    { 'job_name' => 'redis_exporter' },
    +    { 'job_name' => 'gitaly' },
    +    { 'job_name' => 'gitlab_workhorse' },
       ]
       class { 'profile::prometheus::server::common':
         vhost_name          => $vhost_name,
    @@ -57,5 +61,9 @@ class profile::prometheus::server::internal (
         'postgres': port => 9187;
         'bind': port => 9119;
         'mtail': port => 3903;
    +    'gitlab_exporter': port => 9168;
    +    'redis_exporter': port => 9121;
    +    'gitaly': port => 9236;
    +    'gitlab_workhorse': port => 9229;
       }
     }
  • Moved the gitlab exporter to the static targets.

    $scrape_configs =
      [
        {
          'job_name'       => 'prometheus',
          'static_configs' => [
            {
              'targets' => ['localhost:9090'],
              'labels'  => {
                'alias' => 'Prometheus',
              },
            },
          ],
        },
        {
          'job_name'       => 'grafana',
          'static_configs' => [
            {
              'targets' => ['localhost:3000'],
              'labels'  => {
                'alias' => 'Grafana',
              },
            },
          ],
        },
        {
          'job_name'       => 'gitlab_exporter',
          'static_configs' => [
            {
              'targets' => ['gitlab-02.torproject.org:9168'],
              'labels'  => {
                'alias' => 'Gitlab-Exporter',
              },
            },
          ],
        },
      ]

    But looks like gitlab-ce package is getting in the middle of exporting this endpoint. Since on gitlab-02 I see:

    down: gitlab-exporter: 1s, normally up, want up; run: log: (pid 32306) 342718s
    Edited by Hiro
  • Updated gpg key for boklm: https://trac.torproject.org/projects/tor/ticket/34348

    And updated documentation on the wiki.

  • Finally added the node and the gitlab exporters to prom.

    https://trac.torproject.org/projects/tor/ticket/33921

    Apparently the correct syntax was:

      gitlab_exporter             => {
            enable         => true,
            listen_address => '0.0.0.0',
            listen_port    => '9178',
          },

    Wondering why different exporters have a different syntax in the same gitlab configs.

  • Exported gitaly and gitlab dashboard to gradana. See https://gitlab.com/gitlab-org/grafana-dashboards/tree/master/omnibus Should be added to puppet.

  • Edited by Hiro
  • Installed prometheus-nginx-exporter on gitlab-02. The exporter doesn't run as daemon so it needs to be configured.

    Gitlab-02 is having disk issues. Needs more space.

  • A number of ganeti nodes and instances went down last night at 2 UTC. Not sure yet why. Investigating. Checking the last reboot time doesn't look like these have been rebooted. Apparently it was an issue with hetzner.

    Type: 	Fault report 	
    Categories: 	Network
    Start: 	June 9, 2020 6:45:00 PM CEST
    End: 	June 9, 2020 6:55:00 PM CEST
    Description: 	We are currently experiencing a fault on our switches fsn1-dc7-sw_718 and fsn1-dc7-sw_719. Our technicians are working hard on analyzing the cause of the interruption in order to solve the issue.
    
    Please bear with us. As soon as more information is available, we shall inform you via this website.
    
    We apologize for any inconvenience. Thank you for your understanding.
    Update: 	June 9, 2020 6:55:00 PM CEST 	
    	The fault has been fixed.
    
    All systems are accessible again
    Edited by Hiro
  • Sysadmin sync today. Updated tickets and roadmap.

  • Gitlab wasn't sending backups to the right folder and was filling up its disk. Also gitlab-backup dir needs to be owned by the git user.

  • Added new onionperf hosts entries on the DNS.

  • Some last minute gitlab configuration.

  • Doing the reboot dance.

Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment