document mail troubleshooting tricks (gitlab#139)

howto/gitlab.md

@@ -620,7 +620,7 @@ To limit this to `job.log`, of course, you can do:
 
     find -name "job.log" -mtime +14 -print0 | du --files0-from=- -c -h | tee find-mtime+14-joblog-du.log
 
-### Email routing
+### Incoming email routing
 
 Incoming email get routed through either eugeni or the submission
 service, then end up on the Postfix server on `gitlab-02`, and from
@@ -645,6 +645,63 @@ A working run will look something like this:
 
 Emails should be processed every minute or so.
 
+### Outgoing email
+
+Follow the [email not sent](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/smtp.md#email-not-sent) procedure. TL;DR:
+
+    sudo gitlab-rails console
+
+(Yes it takes forever.) Then check if the settings are sane:
+
+```
+--------------------------------------------------------------------------------
+ Ruby:         ruby 3.0.5p211 (2022-11-24 revision ba5cf0f7c5) [x86_64-linux]
+ GitLab:       15.10.0 (496a1d765be) FOSS
+ GitLab Shell: 14.18.0
+ PostgreSQL:   12.12
+------------------------------------------------------------[ booted in 28.31s ]
+Loading production environment (Rails 6.1.7.2)
+irb(main):003:0> ActionMailer::Base.delivery_method
+=> :smtp
+irb(main):004:0> ActionMailer::Base.smtp_settings
+=> 
+{:user_name=>nil,
+ :password=>nil,
+ :address=>"localhost",
+ :port=>25,
+ :domain=>"localhost",
+ :enable_starttls_auto=>false,
+ :tls=>false,
+ :ssl=>false,
+ :openssl_verify_mode=>"none",
+ :ca_file=>"/opt/gitlab/embedded/ssl/certs/cacert.pem"}
+```
+
+Then test an email delivery:
+
+    Notify.test_email('noreply@torproject.org', 'Hello World', 'This is a test message').deliver_now
+
+A working delivery will look something like this, with the last line
+in *green*:
+
+```
+irb(main):001:0> Notify.test_email('noreply@torproject.org', 'Hello World', 'This is a test message').deliver_now
+Delivered mail 64219bdb6e919_10e66548d042948@gitlab-02.mail (20.1ms)
+=> #<Mail::Message:296420, Multipart: false, Headers: <Date: Mon, 27 Mar 2023 13:36:27 +0000>, <From: GitLab <git@gitlab.torproject.org>>, <Reply-To: GitLab <noreply@torproject.org>>, <To: noreply@torproject.org>, <Message-ID: <64219bdb6e919_10e66548d042948@gitlab-02.mail>>, <Subject: Hello World>, <Mime-Version: 1.0>, <Content-Type: text/html; charset=UTF-8>, <Content-Transfer-Encoding: 7bit>, <Auto-Submitted: auto-generated>, <X-Auto-Response-Suppress: All>>
+```
+
+A *failed* delivery will *also* say `Delivered mail` *but* will
+include an error message as well. For example, in [issue 139][] we had
+this error:
+
+```
+irb(main):006:0> Notify.test_email('noreply@torproject.org', 'Hello World', 'This is a test message').deliver_now
+Delivered mail 641c797273ba1_86be948d03829@gitlab-02.mail (7.2ms)
+/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:46:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)
+```
+
+[issue 139]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/139
+
 ## Disaster recovery
 
 In case the entire GitLab machine is destroyed, a new server should be