tpa-gitlab-ci-housekeeping postgresql complaints
i got this mail last night:
Date: Tue, 23 Apr 2024 02:50:45 +0000
From: root@gitlab-02.torproject.org (Cron Daemon)
To: root@gitlab-02.torproject.org
Subject: Cron <root@gitlab-02> chronic /usr/local/sbin/tpa-gitlab-ci-housekeeping.sh
Processing scheduled pipelines in tpo/core/debian/tor (1218) ...
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql_adapter.rb:87:in `rescue in new_client': connection to server at "127.0.0.1", port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections (ActiveRecord::ConnectionNotEstablished)
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql_adapter.rb:77:in `new_client'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:656:in `public_send'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:656:in `new_connection'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:700:in `checkout_new_connection'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:679:in `try_to_checkout_new_connection'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:640:in `acquire_connection'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:341:in `checkout'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `connection'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/primary_host.rb:44:in `connection'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:64:in `read'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:67:in `schema_cache'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:580:in `load_schema!'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/attributes.rb:264:in `load_schema!'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/encryption/encryptable_record.rb:122:in `load_schema!'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sha_attribute.rb:52:in `load_schema!'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:566:in `block in load_schema'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:563:in `synchronize'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:563:in `load_schema'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:429:in `attribute_types'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:455:in `type_for_attribute'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/type_caster/map.rb:16:in `type_for_attribute'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/arel/table.rb:107:in `type_for_attribute'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/table_metadata.rb:18:in `type'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:59:in `build'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:54:in `[]'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:126:in `block in expand_from_hash'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:79:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:79:in `flat_map'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:79:in `expand_from_hash'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:25:in `build_from_hash'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/query_methods.rb:1333:in `build_where_clause'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/query_methods.rb:774:in `where!'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/query_methods.rb:769:in `where'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/querying.rb:22:in `where'
from stdin:3:in `<main>'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/runner/runner_command.rb:39:in `eval'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/runner/runner_command.rb:39:in `perform'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/thor-1.3.1/lib/thor/command.rb:28:in `run'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/thor-1.3.1/lib/thor/invocation.rb:127:in `invoke_command'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/thor-1.3.1/lib/thor.rb:527:in `dispatch'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/command/base.rb:87:in `perform'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/command.rb:48:in `invoke'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/commands.rb:18:in `<top (required)>'
from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
from bin/rails:4:in `<main>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/pg-1.5.6/lib/pg/connection.rb:701:in `async_connect_or_reset': connection to server at "127.0.0.1", port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections (PG::ConnectionBad)
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/pg-1.5.6/lib/pg/connection.rb:833:in `connect_to_hosts'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/pg-1.5.6/lib/pg/connection.rb:764:in `new'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/pg-1.5.6/lib/pg.rb:63:in `connect'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql_adapter.rb:78:in `new_client'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:656:in `public_send'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:656:in `new_connection'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:700:in `checkout_new_connection'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:679:in `try_to_checkout_new_connection'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:640:in `acquire_connection'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:341:in `checkout'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `connection'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/primary_host.rb:44:in `connection'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:64:in `read'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:67:in `schema_cache'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:580:in `load_schema!'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/attributes.rb:264:in `load_schema!'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/encryption/encryptable_record.rb:122:in `load_schema!'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sha_attribute.rb:52:in `load_schema!'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:566:in `block in load_schema'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:563:in `synchronize'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:563:in `load_schema'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:429:in `attribute_types'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:455:in `type_for_attribute'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/type_caster/map.rb:16:in `type_for_attribute'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/arel/table.rb:107:in `type_for_attribute'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/table_metadata.rb:18:in `type'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:59:in `build'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:54:in `[]'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:126:in `block in expand_from_hash'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:79:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:79:in `flat_map'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:79:in `expand_from_hash'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/predicate_builder.rb:25:in `build_from_hash'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/query_methods.rb:1333:in `build_where_clause'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/query_methods.rb:774:in `where!'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/relation/query_methods.rb:769:in `where'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/querying.rb:22:in `where'
from stdin:3:in `<main>'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/runner/runner_command.rb:39:in `eval'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/runner/runner_command.rb:39:in `perform'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/thor-1.3.1/lib/thor/command.rb:28:in `run'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/thor-1.3.1/lib/thor/invocation.rb:127:in `invoke_command'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/thor-1.3.1/lib/thor.rb:527:in `dispatch'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/command/base.rb:87:in `perform'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/command.rb:48:in `invoke'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/commands.rb:18:in `<top (required)>'
from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
from bin/rails:4:in `<main>'
it's happened 4 times so far, and seems to happen more frequently:
Yest. 22:50 [1/1] Cron Daemon Cron <root@gitlab-02> chronic /usr/local/sbin/tpa-gitlab-ci-housekeeping.sh (cron rapports tor)
Sun. 22:50 [1/1] Cron Daemon Cron <root@gitlab-02> chronic /usr/local/sbin/tpa-gitlab-ci-housekeeping.sh (cron rapports tor)
Fri. 22:50 [1/1] Cron Daemon Cron <root@gitlab-02> chronic /usr/local/sbin/tpa-gitlab-ci-housekeeping.sh (cron rapports tor)
April 09 [1/1] Cron Daemon Cron <root@gitlab-02> chronic /usr/local/sbin/tpa-gitlab-ci-housekeeping.sh (cron rapports tor)