GraphQL endpoints broken after GitLab upgrade
Triage Ops is currently failing due to the following error from our Ruby GraphQL code in a lot of the common policies used in many projects:
* Total after limiting: 0 resources
Triaging 'tpo/web/tpo' using the policy found in 'policies/tpo/web/tpo/01-warn-unlabeled.yaml' ...
/usr/local/bundle/gems/graphql-2.0.32/lib/graphql/schema/loader.rb:17:in `fetch': key not found: "data" (KeyError)
from /usr/local/bundle/gems/graphql-2.0.32/lib/graphql/schema/loader.rb:17:in `load'
from /usr/local/bundle/gems/graphql-2.0.32/lib/graphql/schema.rb:104:in `from_introspection'
from /usr/local/bundle/gems/graphql-client-0.26.0/lib/graphql/client.rb:53:in `load_schema'
from /usr/local/bundle/gems/graphql-client-0.26.0/lib/graphql/client.rb:62:in `load_schema'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/network_adapters/graphql_adapter.rb:85:in `schema'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/network_adapters/graphql_adapter.rb:89:in `client'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/network_adapters/graphql_adapter.rb:40:in `parse'
from /builds/tpo/tpa/triage-ops/src/tor.rb:96:in `execute_graphql_query'
from /builds/tpo/tpa/triage-ops/src/tor.rb:131:in `status'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/resource/context.rb:27:in `eval'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/resource/context.rb:25:in `instance_eval'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/resource/context.rb:25:in `eval'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/filters/ruby_conditions_filter.rb:24:in `calculate'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:493:in `block in filter_resource'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:463:in `each'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:463:in `filter_resource'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:456:in `block in filter_resources'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:455:in `select'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:455:in `filter_resources'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:390:in `block in resources_for_rule'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/expand_condition.rb:15:in `each'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/expand_condition.rb:15:in `perform'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:379:in `resources_for_rule'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:271:in `block in process_rules'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:270:in `each'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:270:in `process_rules'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:103:in `block in perform'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:94:in `each'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/lib/gitlab/triage/engine.rb:94:in `perform'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/bin/gitlab-triage:23:in `block in <top (required)>'
from /usr/local/lib/ruby/3.0.0/set.rb:344:in `each_key'
from /usr/local/lib/ruby/3.0.0/set.rb:344:in `each'
from /usr/local/bundle/gems/gitlab-triage-1.48.0/bin/gitlab-triage:12:in `<top (required)>'
from /usr/local/bundle/bin/gitlab-triage:25:in `load'
from /usr/local/bundle/bin/gitlab-triage:25:in `<main>'Scrolling through the many failed jobs, the following thing shows up. (Currently) on page 14 of you can see where things are going wrong: https://gitlab.torproject.org/tpo/tpa/triage-ops/-/pipelines?page=14&scope=all
- On 2026-02-10 @ 23:43 we had the last build that went well: https://gitlab.torproject.org/tpo/tpa/triage-ops/-/pipelines/361025
- on 2026-02-10 @ 23:49 we have the first build that fails. This one fails with an error indicating that GitLab is being upgraded: https://gitlab.torproject.org/tpo/tpa/triage-ops/-/pipelines/361026
- On 2026-02-10 @ 23:57 we have the first error with the GraphQL error: https://gitlab.torproject.org/tpo/tpa/triage-ops/-/pipelines/361028
This has happened since this time. It looks like the GL upgrade around 2026-02-10 @ 23:49 caused something to break with GraphQL.
I'm too tired today to look into this, but this ticket may be useful to keep track of the issue. Right now everything that uses the work item status system is not working with the Triage Bot.