| ... | ... | @@ -1472,6 +1472,83 @@ is being exceeded. |
|
|
|
By default the token lifetime is 5 minutes. This setting can be changed via the
|
|
|
|
GitLab admin web interface, in the Container registry configuration section.
|
|
|
|
|
|
|
|
### Gitaly is unavailable
|
|
|
|
|
|
|
|
If you see this error when browsing GitLab:
|
|
|
|
|
|
|
|
Error: Gitaly is unavailable. Contact your administrator.
|
|
|
|
|
|
|
|
Run this rake task to see what's going on:
|
|
|
|
|
|
|
|
gitlab-rake gitlab:gitaly:check
|
|
|
|
|
|
|
|
You might, for example, see this error:
|
|
|
|
|
|
|
|
```
|
|
|
|
root@gitlab-02:~# gitlab-rake gitlab:gitaly:check
|
|
|
|
Checking Gitaly ...
|
|
|
|
|
|
|
|
Gitaly: ... default ... FAIL: 14:connections to all backends failing; last error: UNKNOWN: ipv4:204.8.99.149:9999: Failed to connect to remote host: Connection refused. debug_error_string:{UNKNOWN:Error received from peer {grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:204.8.99.149:9999: Failed to connect to remote host: Connection refused", grpc_status:14, created_time:"2025-07-18T01:25:42.139054855+00:00"}}
|
|
|
|
storage1 ... FAIL: 14:connections to all backends failing; last error: UNKNOWN: ipv6:%5B2620:7:6002:0:466:39ff:fe74:2f50%5D:9999: Failed to connect to remote host: Connection refused. debug_error_string:{UNKNOWN:Error received from peer {created_time:"2025-07-18T01:25:44.578932647+00:00", grpc_status:14, grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv6:%5B2620:7:6002:0:466:39ff:fe74:2f50%5D:9999: Failed to connect to remote host: Connection refused"}}
|
|
|
|
|
|
|
|
Checking Gitaly ... Finished
|
|
|
|
```
|
|
|
|
|
|
|
|
In this case, the firewall on `gitaly-01` was broken by an error in
|
|
|
|
the Puppet configuration. Fixing the error and running Puppet on both
|
|
|
|
nodes (`gitaly-01` and `gitlab-02`) a couple times fixed the issue.
|
|
|
|
|
|
|
|
Check if you can open a socket to the Gitaly server. In this case, for
|
|
|
|
example, you'd run something like this from `gitlab-02`:
|
|
|
|
|
|
|
|
nc -zv gitaly-01.torproject.org 9999
|
|
|
|
|
|
|
|
Example success:
|
|
|
|
|
|
|
|
```
|
|
|
|
root@gitlab-02:~# nc -zv gitaly-01.torproject.org 9999
|
|
|
|
Connection to gitaly-01.torproject.org (2620:7:6002:0:466:39ff:fe74:2f50) 9999 port [tcp/*] succeeded!
|
|
|
|
```
|
|
|
|
|
|
|
|
Example failure:
|
|
|
|
|
|
|
|
```
|
|
|
|
root@gitlab-02:~# nc -zv gitaly-01.torproject.org 9999
|
|
|
|
nc: connect to gitaly-01.torproject.org (2620:7:6002:0:466:39ff:fe74:2f50) port 9999 (tcp) failed: Connection refused
|
|
|
|
nc: connect to gitaly-01.torproject.org (204.8.99.167) port 9999 (tcp) failed: Connection refused
|
|
|
|
```
|
|
|
|
|
|
|
|
Connection failures could be anything from the firewall causing issues
|
|
|
|
or Gitaly itself being stopped or refusing connections. Check that the
|
|
|
|
service is running on the Gitaly side:
|
|
|
|
|
|
|
|
systemctl status gitaly
|
|
|
|
|
|
|
|
... and the latest logs:
|
|
|
|
|
|
|
|
journalctl -u gitaly -e
|
|
|
|
|
|
|
|
Check the load on the server as well.
|
|
|
|
|
|
|
|
Note that, as of this writing, the `gitlab:gitaly:check` job actually
|
|
|
|
raises an error:
|
|
|
|
|
|
|
|
```
|
|
|
|
root@gitlab-02:~# gitlab-rake gitlab:gitaly:check
|
|
|
|
Checking Gitaly ...
|
|
|
|
|
|
|
|
Gitaly: ... default ... FAIL: 14:connections to all backends failing; last error: UNKNOWN: ipv4:204.8.99.149:9999: Failed to connect to remote host: Connection refused. debug_error_string:{UNKNOWN:Error received from peer {grpc_message:"connections to all backends failing; last error: UNKNOWN: ipv4:204.8.99.149:9999: Failed to connect to remote host: Connection refused", grpc_status:14, created_time:"2025-07-18T01:34:28.590049422+00:00"}}
|
|
|
|
storage1 ... OK
|
|
|
|
|
|
|
|
Checking Gitaly ... Finished
|
|
|
|
```
|
|
|
|
|
|
|
|
This is normal: the `default` storage backend is the legacy Gitaly
|
|
|
|
server on `gitlab-02` which was disabled in the `gitaly-01`
|
|
|
|
migration. The configuration was kept because [GitLab requires a
|
|
|
|
default repository storage](https://docs.gitlab.com/administration/gitaly/configure_gitaly/#gitlab-requires-a-default-repository-storage), a [known (and 2019) issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36175). See
|
|
|
|
[anarcat's latest comment on this](https://gitlab.com/gitlab-org/gitlab/-/issues/36175#note_2634803728).
|
|
|
|
|
|
|
|
## Disaster recovery
|
|
|
|
|
|
|
|
In case the entire GitLab machine is destroyed, a new server should be
|
| ... | ... | |
