[[_TOC_]]

[PostgreSQL](https://www.postgresql.org/) is an advanced database server that is robust and
fast, although possibly less well-known and popular than it's eternal
rival in the free software world, MySQL.

# Tutorial

Those are quick reminders on easy things to do in a cluster.

## Connecting

Our PostgreSQL setup is fairly standard so connecting to the database
is like any other Debian machine:

    sudo -u postres psql

This drops you in a [psql shell](https://www.postgresql.org/docs/11/app-psql.html) where you can issue SQL queries
and so on.

# How-to

## Find what is taking up space

This will report size and count information for all "relations", which
includes indexes:

    SELECT relname AS objectname
         , relkind AS objecttype
         , reltuples AS "#entries"
         , pg_size_pretty(relpages::bigint*8*1024) AS size
         FROM pg_class
         WHERE relpages >= 8
         ORDER BY relpages DESC;

It might be difficult to track the *total* size of a table because it
doesn't add up index size, which is typically small but *can* grow
quite significantly.

This will report the same, but with aggregated results:

    SELECT table_name
        , row_estimate
        , pg_size_pretty(total_bytes) AS total
        , pg_size_pretty(table_bytes) AS TABLE
        , pg_size_pretty(index_bytes) AS INDEX
        , pg_size_pretty(toast_bytes) AS toast
      FROM (
      SELECT *, total_bytes-index_bytes-COALESCE(toast_bytes,0) AS table_bytes FROM (
          SELECT c.oid,nspname AS table_schema, relname AS TABLE_NAME
                  , c.reltuples AS row_estimate
                  , pg_total_relation_size(c.oid) AS total_bytes
                  , pg_indexes_size(c.oid) AS index_bytes
                  , pg_total_relation_size(reltoastrelid) AS toast_bytes
              FROM pg_class c
              LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
              WHERE relkind = 'r'
      ) a
    ) a ORDER BY total_bytes DESC LIMIT 10;

Same with databases:

    SELECT d.datname AS Name,  pg_catalog.pg_get_userbyid(d.datdba) AS Owner,
        CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')
            THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))
            ELSE 'No Access'
        END AS SIZE
    FROM pg_catalog.pg_database d
        ORDER BY
        CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')
            THEN pg_catalog.pg_database_size(d.datname)
            ELSE NULL
        END DESC -- nulls first
        LIMIT 20;

Source: [PostgreSQL wiki](https://wiki.postgresql.org/wiki/Disk_Usage). See also [the upstream manual](https://www.postgresql.org/docs/11/disk-usage.html).

## Show running queries

If the server seems slow, it's possible to inspect running queries
with this query:

    SELECT datid,datname,pid,query_start,now()-query_start as age,state,query FROM pg_stat_activity;

If the `state` is `waiting`, it might be worth looking at the
`wait_event`, and `wait_event_type` columns as well. We're looking for
deadlocks here.

## Killing a slow query

This will kill all queries to `database_name`:

```
SELECT 
    pg_terminate_backend(pid) 
FROM 
    pg_stat_activity 
WHERE 
    -- don't kill my own connection!
    pid <> pg_backend_pid()
    -- don't kill the connections to other databases
    AND datname = 'database_name'
    ;
```

A more selective approach is to list threads (above) and then kill
only one PID, say:

```
SELECT 
    pg_terminate_backend(pid) 
FROM 
    pg_stat_activity 
WHERE 
    -- don't kill my own connection!
    pid = 1234;
```

## Diagnosing performance issues

Some ideas from the `#postgresql` channel on Libera:

 * look at `query_start` and `state`, and if `state` is `waiting`,
   `wait_event`, and `wait_event_type`, in [`pg_stat_activity`](https://www.postgresql.org/docs/14/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW),
   possibly looking for locks here. this is done by the query above,
   in [Show running queries](#show-running-queries)

 * enable [`pg_stat_statements`](https://www.postgresql.org/docs/14/pgstatstatements.html) to see where the time is going,
   and then dig into the queries/functions found there, possibly with
   [`auto_explain`](https://www.postgresql.org/docs/current/auto-explain.html) and `auto_explain.log_nested_statements=on`

In general, we have a few Grafana dashboards specific to PostgreSQL
(see [logs and metrics](#logs-and-metrics), below) that might help tracing performance
issues as well. Obviously, system-level statistics (disk, CPU, memory
usage) can help pinpoint where the bottleneck is as well, so basic
node-level Grafana dashboards are useful there as well.

## Running a full backup

Backups are normally automatically ran on the backup server (currently
`bungei`) but you can also run a backup by hand. This will run a
backup of `meronense`, for example:

    sudo -u torbackup postgres-make-one-base-backup $(grep ^meronense.torproject.org $(which postgres-make-base-backups ))

See the [reference](#reference) section for details.

## Monitoring the VACUUM processes

In PostgreSQL, the [VACUUM command](https://www.postgresql.org/docs/current/sql-vacuum.html) "reclaims storage occupied by
dead tuples". To quote the excellent PostgreSQL documentation:

> In normal PostgreSQL operation, tuples that are deleted or obsoleted
> by an update are not physically removed from their table; they
> remain present until a VACUUM is done. Therefore it's necessary to
> do VACUUM periodically, especially on frequently-updated tables.

By default, the [autovacuum launcher](https://www.postgresql.org/docs/9.1/routine-vacuuming.html#AUTOVACUUM) is enabled in PostgreSQL (and
in our deployments), which should automatically take care of this
problem.

This will show that the autovacuum daemon is running:

    # ps aux | grep [v]acuum
    postgres   534  0.5  4.7 454920 388012 ?       Ds   05:31   3:08 postgres: 11/main: autovacuum worker   bacula
    postgres 17259  0.0  0.1 331376 10984 ?        Ss   Nov12   0:10 postgres: 11/main: autovacuum launcher   

In the above, the `launcher` is running, and we can see a worker has
been started to vacuum the `bacula` table.

If you don't see the launcher, check that it's enabled:

    bacula=# SELECT name, setting FROM pg_settings WHERE name='autovacuum' or name='track_counts';
     autovacuum   | on
     track_counts | on

Both need to be `on` for the autovacuum workers to operate. It's
possible that some tables might have autovacuum disabled, however,
see:

    SELECT reloptions FROM pg_class WHERE relname='my_table';

In the above scenario, the `autovacuum worker bacula` process had been
running for *hours*, which was concerning. One way to diagnose is to
figure out how much data there is to vacuum.

This query will show the tables with dead tuples that need to be
cleaned up by the VACUUM process:

    SELECT relname, n_dead_tup FROM pg_stat_user_tables where n_dead_tup > 0 order by n_dead_tup DESC LIMIT 1;

In our case, there were tens of millions of rows to clean:

    bacula=# SELECT relname, n_dead_tup FROM pg_stat_user_tables where n_dead_tup > 0 order by n_dead_tup DESC LIMIT 1;
     file    |  183278595

That is 200 *million* tuples to cleanup!

We can see details of the vacuum operation with this funky query,
taken from [this amazing blog post](https://dataegret.com/2017/10/deep-dive-into-postgres-stats-pg_stat_progress_vacuum/):

```
SELECT
p.pid,
now() - a.xact_start AS duration,
coalesce(wait_event_type ||'.'|| wait_event, 'f') AS waiting,
CASE
WHEN a.query ~*'^autovacuum.*to prevent wraparound' THEN 'wraparound'
WHEN a.query ~*'^vacuum' THEN 'user'
ELSE 'regular'
END AS mode,
p.datname AS database,
p.relid::regclass AS table,
p.phase,
pg_size_pretty(p.heap_blks_total * current_setting('block_size')::int) AS table_size,
pg_size_pretty(pg_total_relation_size(relid)) AS total_size,
pg_size_pretty(p.heap_blks_scanned * current_setting('block_size')::int) AS scanned,
pg_size_pretty(p.heap_blks_vacuumed * current_setting('block_size')::int) AS vacuumed,
round(100.0 * p.heap_blks_scanned / p.heap_blks_total, 1) AS scanned_pct,
round(100.0 * p.heap_blks_vacuumed / p.heap_blks_total, 1) AS vacuumed_pct,
p.index_vacuum_count,
round(100.0 * p.num_dead_tuples / p.max_dead_tuples,1) AS dead_pct
FROM pg_stat_progress_vacuum p
JOIN pg_stat_activity a using (pid)
ORDER BY now() - a.xact_start DESC
```

For example, the above vacuum on the Bacula director is in this state
at the time of writing:

```
bacula=# \x
Expanded display is on.
bacula=# SELECT [...]
-[ RECORD 1 ]------+----------------
pid                | 534
duration           | 10:55:24.413986
waiting            | f
mode               | regular
database           | bacula
table              | file
phase              | scanning heap
table_size         | 55 GB
total_size         | 103 GB
scanned            | 29 GB
vacuumed           | 16 GB
scanned_pct        | 52.2
vacuumed_pct       | 29.3
index_vacuum_count | 1
dead_pct           | 93.8
```

This is a lot of information, but basically the worker with PID 513
has been running for 10h55m on the `bacula` database. It is in the
`scanning heap` phase, second out of 8 phases of the [vacuuming
process](https://www.postgresql.org/docs/11/progress-reporting.html#VACUUM-PROGRESS-REPORTING). It's working on the `file` table which has has 55GB of
data on the "heap" and a total size of 103 GB (including indexes). It
scanned 29 GB of data (52%), vacuumed 16GB out of that (29%). The
`dead_pct` indicates that the [`maintenance_work_mem` buffer](https://www.postgresql.org/docs/11/runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM) is
94% full, which could indicate raising that buffer could improve
performance. I am not sure what the `waiting` and `index_vacuum_count`
fields are for.

Naturally, this will return information for very large VACUUM
operations, which typically do not take this long. This one VACUUM
operation was especially slow because we suddenly removed almost half
of the old clients in the Bacula database, see [ticket
40525](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40525) for more information.

One more trick: this will show last VACUUM dates on tables:

    SELECT relname, last_vacuum, last_autovacuum FROM pg_stat_user_tables WHERE last_vacuum IS NOT NULL or last_autovacuum IS NOT NULL ORDER BY relname;

Some of the ideas above were found on [this datadog post](https://www.datadoghq.com/blog/postgresql-vacuum-monitoring/).

Finally, note that the Debian 10 ("buster") version of PostgreSQL (11) does
not support reporting on "FULL" VACUUM, that feature was introduced in
PostgreSQL 12. Debian 11 ("bullseye") has PostgreSQL 13, but progress
there is reported in the `pg_stat_progress_cluster` table, so the
above might not work even there.

## Checking for wasted space

PostgreSQL is particular as a database in the sense that it never
actually returns free space to the operating system unless explicitly
asked for. Modern PostgreSQL releases (8.1+) have an "auto-vacuum"
daemon which takes care of cleaning up `DELETE` and related operations
to reclaim that disk space, but this only marks those regions of the
database as usable: it doesn't actually returns those blocks to the
operating system.

Because databases typically either stay the same size or grow over
their lifetime, this typically does not matter: the next `INSERT` will
use that space and no space is actually wasted.

But sometimes that disk space can grow too large. How do we check if
our database is wasting space? There are many ways...

### check_postgresql

There's actually a Nagios plugin, which we are not currently using,
which does this. It is called [check_postgresql](https://bucardo.org/check_postgres/check_postgres.pl.html) and features a
[bloat check](https://bucardo.org/check_postgres/check_postgres.pl.html#bloat) which can run regularly.

### Running bloat query by hand

The above script might be annoying to deploy for an ad-hoc
situation. You can just run the query by hand instead:

```
SELECT
  current_database(), schemaname, tablename, /*reltuples::bigint, relpages::bigint, otta,*/
  ROUND((CASE WHEN otta=0 THEN 0.0 ELSE sml.relpages::float/otta END)::numeric,1) AS tbloat,
  CASE WHEN relpages < otta THEN 0 ELSE bs*(sml.relpages-otta)::BIGINT END AS wastedbytes,
  iname, /*ituples::bigint, ipages::bigint, iotta,*/
  ROUND((CASE WHEN iotta=0 OR ipages=0 THEN 0.0 ELSE ipages::float/iotta END)::numeric,1) AS ibloat,
  CASE WHEN ipages < iotta THEN 0 ELSE bs*(ipages-iotta) END AS wastedibytes
FROM (
  SELECT
    schemaname, tablename, cc.reltuples, cc.relpages, bs,
    CEIL((cc.reltuples*((datahdr+ma-
      (CASE WHEN datahdr%ma=0 THEN ma ELSE datahdr%ma END))+nullhdr2+4))/(bs-20::float)) AS otta,
    COALESCE(c2.relname,'?') AS iname, COALESCE(c2.reltuples,0) AS ituples, COALESCE(c2.relpages,0) AS ipages,
    COALESCE(CEIL((c2.reltuples*(datahdr-12))/(bs-20::float)),0) AS iotta -- very rough approximation, assumes all cols
  FROM (
    SELECT
      ma,bs,schemaname,tablename,
      (datawidth+(hdr+ma-(case when hdr%ma=0 THEN ma ELSE hdr%ma END)))::numeric AS datahdr,
      (maxfracsum*(nullhdr+ma-(case when nullhdr%ma=0 THEN ma ELSE nullhdr%ma END))) AS nullhdr2
    FROM (
      SELECT
        schemaname, tablename, hdr, ma, bs,
        SUM((1-null_frac)*avg_width) AS datawidth,
        MAX(null_frac) AS maxfracsum,
        hdr+(
          SELECT 1+count(*)/8
          FROM pg_stats s2
          WHERE null_frac<>0 AND s2.schemaname = s.schemaname AND s2.tablename = s.tablename
        ) AS nullhdr
      FROM pg_stats s, (
        SELECT
          (SELECT current_setting('block_size')::numeric) AS bs,
          CASE WHEN substring(v,12,3) IN ('8.0','8.1','8.2') THEN 27 ELSE 23 END AS hdr,
          CASE WHEN v ~ 'mingw32' THEN 8 ELSE 4 END AS ma
        FROM (SELECT version() AS v) AS foo
      ) AS constants
      GROUP BY 1,2,3,4,5
    ) AS foo
  ) AS rs
  JOIN pg_class cc ON cc.relname = rs.tablename
  JOIN pg_namespace nn ON cc.relnamespace = nn.oid AND nn.nspname = rs.schemaname AND nn.nspname <> 'information_schema'
  LEFT JOIN pg_index i ON indrelid = cc.oid
  LEFT JOIN pg_class c2 ON c2.oid = i.indexrelid
) AS sml
ORDER BY wastedbytes DESC
```

### Another way

It is rumored, however, that this is not very accurate. A [better
option](https://blog.ioguix.net/postgresql/2014/03/28/Playing-with-indexes-and-better-bloat-estimate.html) seems to be this ... more complicated query:

```
-- change to the max number of field per index if not default.
\set index_max_keys 32
-- (readonly) IndexTupleData size
\set index_tuple_hdr 2
-- (readonly) ItemIdData size
\set item_pointer 4
-- (readonly) IndexAttributeBitMapData size
\set index_attribute_bm (:index_max_keys + 8 - 1) / 8

SELECT current_database(), nspname, c.relname AS table_name, index_name, bs*(sub.relpages)::bigint AS totalbytes,
  CASE WHEN sub.relpages <= otta THEN 0 ELSE bs*(sub.relpages-otta)::bigint END                                    AS wastedbytes,
  CASE WHEN sub.relpages <= otta THEN 0 ELSE bs*(sub.relpages-otta)::bigint * 100 / (bs*(sub.relpages)::bigint) END AS realbloat
FROM (
  SELECT bs, nspname, table_oid, index_name, relpages, coalesce(
    ceil((reltuples*(:item_pointer+nulldatahdrwidth))/(bs-pagehdr::float)) +
      CASE WHEN am.amname IN ('hash','btree') THEN 1 ELSE 0 END , 0 -- btree and hash have a metadata reserved block
    ) AS otta
  FROM (
    SELECT maxalign, bs, nspname, relname AS index_name, reltuples, relpages, relam, table_oid,
      ( index_tuple_hdr_bm +
          maxalign - CASE /* Add padding to the index tuple header to align on MAXALIGN */
            WHEN index_tuple_hdr_bm%maxalign = 0 THEN maxalign
            ELSE index_tuple_hdr_bm%maxalign
          END
        + nulldatawidth + maxalign - CASE /* Add padding to the data to align on MAXALIGN */
            WHEN nulldatawidth::integer%maxalign = 0 THEN maxalign
            ELSE nulldatawidth::integer%maxalign
          END
      )::numeric AS nulldatahdrwidth, pagehdr
    FROM (
      SELECT
        i.nspname, i.relname, i.reltuples, i.relpages, i.relam, s.starelid, a.attrelid AS table_oid,
        current_setting('block_size')::numeric AS bs,
        /* MAXALIGN: 4 on 32bits, 8 on 64bits (and mingw32 ?) */
        CASE
          WHEN version() ~ 'mingw32' OR version() ~ '64-bit' THEN 8
          ELSE 4
        END AS maxalign,
        /* per page header, fixed size: 20 for 7.X, 24 for others */
        CASE WHEN substring(current_setting('server_version') FROM '#"[0-9]+#"%' FOR '#')::integer > 7
          THEN 24
          ELSE 20
        END AS pagehdr,
        /* per tuple header: add index_attribute_bm if some cols are null-able */
        CASE WHEN max(coalesce(s.stanullfrac,0)) = 0
          THEN :index_tuple_hdr
          ELSE :index_tuple_hdr + :index_attribute_bm
        END AS index_tuple_hdr_bm,
        /* data len: we remove null values save space using it fractionnal part from stats */
        sum( (1-coalesce(s.stanullfrac, 0)) * coalesce(s.stawidth, 2048) ) AS nulldatawidth
      FROM pg_attribute AS a
        JOIN pg_statistic AS s ON s.starelid=a.attrelid AND s.staattnum = a.attnum
        JOIN (
          SELECT nspname, relname, reltuples, relpages, indrelid, relam, regexp_split_to_table(indkey::text, ' ')::smallint AS attnum
          FROM pg_index
            JOIN pg_class ON pg_class.oid=pg_index.indexrelid
            JOIN pg_namespace ON pg_namespace.oid = pg_class.relnamespace
        ) AS i ON i.indrelid = a.attrelid AND a.attnum = i.attnum
      WHERE a.attnum > 0
      GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9
    ) AS s1
  ) AS s2
    LEFT JOIN pg_am am ON s2.relam = am.oid
) as sub
JOIN pg_class c ON c.oid=sub.table_oid
ORDER BY wastedbytes;
```

It was modified to sort the output by `wastedbytes`.

### Grouped output

One disadvantage of the above query is that tables and indexes are
displayed separately. How do we know which belongs to which? It also
makes it less obvious what the big tables are, and which ones are
important.

This one comes from the [pgx_scripts GitHub repo](https://github.com/pgexperts/pgx_scripts), and is a [130+
line SQL query](https://github.com/pgexperts/pgx_scripts/blob/master/bloat/table_bloat_check.sql):

```
-- new table bloat query
-- still needs work; is often off by +/- 20%
WITH constants AS (
    -- define some constants for sizes of things
    -- for reference down the query and easy maintenance
    SELECT current_setting('block_size')::numeric AS bs, 23 AS hdr, 8 AS ma
),
no_stats AS (
    -- screen out table who have attributes
    -- which dont have stats, such as JSON
    SELECT table_schema, table_name, 
        n_live_tup::numeric as est_rows,
        pg_table_size(relid)::numeric as table_size
    FROM information_schema.columns
        JOIN pg_stat_user_tables as psut
           ON table_schema = psut.schemaname
           AND table_name = psut.relname
        LEFT OUTER JOIN pg_stats
        ON table_schema = pg_stats.schemaname
            AND table_name = pg_stats.tablename
            AND column_name = attname 
    WHERE attname IS NULL
        AND table_schema NOT IN ('pg_catalog', 'information_schema')
    GROUP BY table_schema, table_name, relid, n_live_tup
),
null_headers AS (
    -- calculate null header sizes
    -- omitting tables which dont have complete stats
    -- and attributes which aren't visible
    SELECT
        hdr+1+(sum(case when null_frac <> 0 THEN 1 else 0 END)/8) as nullhdr,
        SUM((1-null_frac)*avg_width) as datawidth,
        MAX(null_frac) as maxfracsum,
        schemaname,
        tablename,
        hdr, ma, bs
    FROM pg_stats CROSS JOIN constants
        LEFT OUTER JOIN no_stats
            ON schemaname = no_stats.table_schema
            AND tablename = no_stats.table_name
    WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
        AND no_stats.table_name IS NULL
        AND EXISTS ( SELECT 1
            FROM information_schema.columns
                WHERE schemaname = columns.table_schema
                    AND tablename = columns.table_name )
    GROUP BY schemaname, tablename, hdr, ma, bs
),
data_headers AS (
    -- estimate header and row size
    SELECT
        ma, bs, hdr, schemaname, tablename,
        (datawidth+(hdr+ma-(case when hdr%ma=0 THEN ma ELSE hdr%ma END)))::numeric AS datahdr,
        (maxfracsum*(nullhdr+ma-(case when nullhdr%ma=0 THEN ma ELSE nullhdr%ma END))) AS nullhdr2
    FROM null_headers
),
table_estimates AS (
    -- make estimates of how large the table should be
    -- based on row and page size
    SELECT schemaname, tablename, bs,
        reltuples::numeric as est_rows, relpages * bs as table_bytes,
    CEIL((reltuples*
            (datahdr + nullhdr2 + 4 + ma -
                (CASE WHEN datahdr%ma=0
                    THEN ma ELSE datahdr%ma END)
                )/(bs-20))) * bs AS expected_bytes,
        reltoastrelid
    FROM data_headers
        JOIN pg_class ON tablename = relname
        JOIN pg_namespace ON relnamespace = pg_namespace.oid
            AND schemaname = nspname
    WHERE pg_class.relkind = 'r'
),
estimates_with_toast AS (
    -- add in estimated TOAST table sizes
    -- estimate based on 4 toast tuples per page because we dont have 
    -- anything better.  also append the no_data tables
    SELECT schemaname, tablename, 
        TRUE as can_estimate,
        est_rows,
        table_bytes + ( coalesce(toast.relpages, 0) * bs ) as table_bytes,
        expected_bytes + ( ceil( coalesce(toast.reltuples, 0) / 4 ) * bs ) as expected_bytes
    FROM table_estimates LEFT OUTER JOIN pg_class as toast
        ON table_estimates.reltoastrelid = toast.oid
            AND toast.relkind = 't'
),
table_estimates_plus AS (
-- add some extra metadata to the table data
-- and calculations to be reused
-- including whether we cant estimate it
-- or whether we think it might be compressed
    SELECT current_database() as databasename,
            schemaname, tablename, can_estimate, 
            est_rows,
            CASE WHEN table_bytes > 0
                THEN table_bytes::NUMERIC
                ELSE NULL::NUMERIC END
                AS table_bytes,
            CASE WHEN expected_bytes > 0 
                THEN expected_bytes::NUMERIC
                ELSE NULL::NUMERIC END
                    AS expected_bytes,
            CASE WHEN expected_bytes > 0 AND table_bytes > 0
                AND expected_bytes <= table_bytes
                THEN (table_bytes - expected_bytes)::NUMERIC
                ELSE 0::NUMERIC END AS bloat_bytes
    FROM estimates_with_toast
    UNION ALL
    SELECT current_database() as databasename, 
        table_schema, table_name, FALSE, 
        est_rows, table_size,
        NULL::NUMERIC, NULL::NUMERIC
    FROM no_stats
),
bloat_data AS (
    -- do final math calculations and formatting
    select current_database() as databasename,
        schemaname, tablename, can_estimate, 
        table_bytes, round(table_bytes/(1024^2)::NUMERIC,3) as table_mb,
        expected_bytes, round(expected_bytes/(1024^2)::NUMERIC,3) as expected_mb,
        round(bloat_bytes*100/table_bytes) as pct_bloat,
        round(bloat_bytes/(1024::NUMERIC^2),2) as mb_bloat,
        table_bytes, expected_bytes, est_rows
    FROM table_estimates_plus
)
-- filter output for bloated tables
SELECT databasename, schemaname, tablename,
    can_estimate,
    est_rows,
    pct_bloat, mb_bloat,
    table_mb
FROM bloat_data
-- this where clause defines which tables actually appear
-- in the bloat chart
-- example below filters for tables which are either 50%
-- bloated and more than 20mb in size, or more than 25%
-- bloated and more than 4GB in size
WHERE ( pct_bloat >= 50 AND mb_bloat >= 10 )
    OR ( pct_bloat >= 25 AND mb_bloat >= 1000 )
ORDER BY mb_bloat DESC;
```

It will show only tables which have significant bloat, which is
defined in the last few lines above. It makes the output much more
readable.

There's also [this other query](https://github.com/ioguix/pgsql-bloat-estimation) we haven't evaluated.

## Recovering disk space

In some cases, you do need to reclaim actual operating system disk
space from the PostgreSQL server (see above to see if you do). This
can happen for example,for example if you have [removed years of old
data from a database](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40525)).

### VACUUM FULL

Typically this is done with the `VACUUM FULL` command (instead of
plain `VACUUM`, which the auto-vacuum does, see [this discussion for
details](https://www.postgresql.org/docs/9.1/routine-vacuuming.html#VACUUM-FOR-SPACE-RECOVERY)). This will actually rewrite all the tables to make sure
only the relevant data is actually stored on this. It's roughly the
equivalent of a dump/restore, except it is faster.

### pg_repack

For very large changes (say, a dozens of terabytes) however, `VACUUM
FULL` (and even plain `VACUUM`) can be prohibitively slow (think
days). And while `VACUUM` doesn't require an exclusive lock on the
tables it's working on, `VACUUM FULL` *does* which implies a
significant outage.

An alternative to that method is the [pg_repack extension](https://github.com/reorg/pg_repack), which
is [packaged in Debian](https://tracker.debian.org/pkg/pg-repack). In Debian 10 buster, the following
procedure was used on `bacula-director-01` to purge old data about
removed Bacula clients that hadn't been cleaned up in years:

    apt install postgresql-11-repack

Then install the extension on the database, as the postgres user (sudo
-u postgres -i), this needs to be done only once:

    psql -c "CREATE EXTENSION pg_repack" -d bacula

Then, for each table:

    pg_repack  -d bacula --table media

It is a good idea to start with a small table we can afford to lose,
just in case something goes wrong. That job took about 2 hours on a
very large table (150GB, `file`). The entire Bacula database went from
using 161GB to 91GB after that cleanup, see [this ticket for
details](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40525#note_2763212).

When done, drop the `pg_repack` extension:

    DROP EXTENSION pg_repack;

Also note that, after the repack, `VACUUM` performance improved
significantly, going from hours (if not days) to minutes.

Note that [pg_squeeze](https://github.com/cybertec-postgresql/pg_squeeze) is another alternative to `pg_repack`, but
it isn't available in Debian.

## Pager playbook

### BASE-IS-OLD

Our nagios check warns us when a backup server has not successfully
fetched a base backup recently. The causes often are that either the
postgres server or the backup host went down or was down during the
time of the weekly cronjob. The TLS certificate could have also been
renewed without the postgres daemon having been restarted.

To re-run a base backup for a specific cluster, log into the backup
server(s) (currently `bungei`), then find the port for the affected
cluster:

    tail /usr/local/bin/postgres-make-base-backups

Then run this backup in a screen:

    sudo -u torbackup /usr/local/bin/postgres-make-base-backups <host>:<port>

### MISSING-BASE

WARNING: this procedure was taken from the [DSA wiki](https://dsa.debian.org/howto/postgres-backup/) and is
untested.

If you get a Nagios warning like this:

    [fasolo, dak] MISSING-BASE: dak.BASE.backuphost.debian.org-20180211-012002-fasolo.debian.org-dak-9.6-backup.tar.gz

This means that we started doing a base backup (as witnessed by a
.backup file next to a WAL), but for some reason we don't have the
corresponding base file.

    root@backuphost:/srv/backups/pg/fasolo# ls -l *backup*
    -rw------- 1 debbackup debbackup 9201093916 Jan 14 06:18 dak.BASE.backuphost.debian.org-20180114-012001-fasolo.debian.org-dak-9.6-backup.tar.gz
    -rw------- 1 debbackup debbackup 9227651542 Jan 21 06:25 dak.BASE.backuphost.debian.org-20180121-012001-fasolo.debian.org-dak-9.6-backup.tar.gz
    -rw------- 1 debbackup debbackup 9266306750 Jan 28 07:59 dak.BASE.backuphost.debian.org-20180128-012001-fasolo.debian.org-dak-9.6-backup.tar.gz
    -rw------- 1 debbackup debbackup 9312602089 Feb  5 11:00 dak.BASE.backuphost.debian.org-20180204-012001-fasolo.debian.org-dak-9.6-backup.tar.gz
    -rw------- 1 debbackup debbackup 9346830509 Feb 12 10:25 dak.BASE.backuphost.debian.org-20180212-094930-fasolo.debian.org-dak-9.6-backup.tar.gz
    -rw------- 1 debbackup debbackup        353 Jan 14 06:18 dak.WAL.0000000100000033000000A6.00000028.backup
    -rw------- 1 debbackup debbackup        350 Jan 20 11:20 dak.WAL.00000001000000350000008C.00000028.backup
    -rw------- 1 debbackup debbackup        353 Jan 21 06:25 dak.WAL.000000010000003600000068.00000028.backup
    -rw------- 1 debbackup debbackup        353 Jan 28 07:59 dak.WAL.0000000100000038000000E3.00000028.backup
    -rw------- 1 debbackup debbackup        353 Feb  5 11:00 dak.WAL.000000010000003B00000090.00000028.backup
    -rw------- 1 debbackup debbackup        350 Feb  5 15:49 dak.WAL.000000010000003B0000009B.00000108.backup
    -rw------- 1 debbackup debbackup        353 Feb 11 10:09 dak.WAL.000000010000003D000000AC.00000028.backup
    -rw------- 1 debbackup debbackup        353 Feb 12 10:25 dak.WAL.000000010000003E00000027.00000178.backup

`.backup` files are created on the postgres server and shipped to the
backup hosts whenever a base backup is initiated. We do some
labelling, so we know which backup host the corresponding tarball
should end up with. For example:

    root@backuphost:/srv/backups/pg/fasolo# cat dak.WAL.000000010000003B00000090.00000028.backup
    START WAL LOCATION: 3B/90000028 (file 000000010000003B00000090)
    STOP WAL LOCATION: 3B/97CF2138 (file 000000010000003B00000097)
    CHECKPOINT LOCATION: 3B/90000098
    BACKUP METHOD: streamed
    BACKUP FROM: master
    START TIME: 2018-02-05 10:25:28 UTC
    LABEL: backuphost.debian.org-20180204-012001-fasolo.debian.org-dak-9.6-backup
    STOP TIME: 2018-02-05 10:59:50 UTC

To fix this, verify we have a later base tarball, or that we are fine
for some other reason, and remove the corresponding `.backup` file from
the backup host. In the case above, we would remove
`dak.WAL.000000010000003D000000AC.00000028.backup`.

### WAL-MISSING-AFTER

Example message:

    [troodi, main] WAL-MISSING-AFTER: troodi/main.WAL.00000001000000D9000000AD

This means that a WAL file is missing after the specified
file. Specifically, in the above scenario, the following files are
present, in chronological order:

    -rw------- 1 torbackup torbackup   16777216 May 10 05:08 main.WAL.00000001000000D9000000AA
    -rw------- 1 torbackup torbackup   16777216 May 10 05:47 main.WAL.00000001000000D9000000AB
    -rw------- 1 torbackup torbackup   16777216 May 10 06:20 main.WAL.00000001000000D9000000AC
    -rw------- 1 torbackup torbackup   16777216 May 10 06:26 main.WAL.00000001000000D9000000AD
    -rw------- 1 torbackup torbackup   16777216 May 10 13:57 main.WAL.00000001000000D9000000B5

Notice the jump from `...AD` to `...B5`. We're missing `AE`, `AF`,
`B1`, `B2`, `B3`, `B4`, specifically. We can also tell that something
happened between 6:26 and 13:57 on that day.

 1. List the files in chronological order:

        ls -ltr /srv/backups/pg/troodi/ | less

 2. Find the file warned about, using `/` then the filename
    (`main.WAL.00000001000000D9000000AD`), above

 3. Look for a `.BASE.` file *following* the missing file, using `/`
    again

 4. Either:

    * if a `.BASE.` backup is present after the missing files, it is
      harmless to the extent that the missing timeframe is not
      necessary. to remove the warnings, previous WAL files *do* need
      to be removed, by hand.

    * if a `.BASE.` backup is *not* present after the missing files,
      the backup integrity is faulty, and a new base backup needs to
      be performed. See [Running a full
      backup](#running-a-full-backup) above.

Here is an example of the second case above, and recovery:

    root@bungei:~# cd /srv/backups/pg/meronense
    root@bungei:/srv/backups/pg/meronense# /usr/lib/nagios/plugins/dsa-check-backuppg -e -v
    [meronense, main] WAL-MISSING-AFTER: meronense/main.WAL.00000001000005A900000073
    [meronense, main] NO-BASE: no base backup found?
    [meronense, main] NO-BACKUP: no backups! (no .backup files found)
    NOT-CONFIGURED: meronense-11
    IGNORED: .nobackup
    IGNORED: lost+found
    IGNORED: BASE-meronense.torproject.org:5432-G0uKzfYhtH.tar.gz
    root@bungei:/srv/backups/pg/meronense# ls -altr
    total 98312
    -rw-------  1 torbackup torbackup 16777216 Jun 23 13:49 main.WAL.00000001000005A90000006F
    -rw-------  1 torbackup torbackup 16777216 Jun 23 13:49 main.WAL.00000001000005A900000070
    -rw-------  1 torbackup torbackup 16777216 Jun 23 13:58 main.WAL.00000001000005A900000071
    -rw-------  1 torbackup torbackup 16777216 Jun 23 13:59 main.WAL.00000001000005A900000072
    -rw-------  1 torbackup torbackup 16777216 Jun 23 13:59 main.WAL.00000001000005A900000073
    -rw-------  1 torbackup torbackup 16777216 Jun 23 14:14 main.WAL.00000001000005A90000007F
    drwxr-xr-x 10 torbackup torbackup     4096 Jun 23 14:14 ..
    drwxr-xr-x  2 torbackup torbackup     4096 Jun 23 14:14 .
    root@bungei:/srv/backups/pg/meronense# rm main.WAL.00000001000005A90000006F main.WAL.00000001000005A900000070 main.WAL.00000001000005A900000071 main.WAL.00000001000005A900000072 main.WAL.00000001000005A900000073
    root@bungei:/srv/backups/pg/meronense# /usr/lib/nagios/plugins/dsa-check-backuppg -e -v
    [meronense, main] NO-BASE: no base backup found?
    [meronense, main] NO-BACKUP: no backups! (no .backup files found)
    NOT-CONFIGURED: meronense-11
    IGNORED: .nobackup
    IGNORED: lost+found
    IGNORED: BASE-meronense.torproject.org:5432-G0uKzfYhtH.tar.gz

There was a gap between `main.WAL.00000001000005A900000073` and
`main.WAL.00000001000005A90000007F` so everything up to and
including the former were removed by hand. Then a full backup was
performed. The reason why the BASE backup was missing is this was
following a failed upgrade (see [tpo/tpa/team#40809](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40809)).

## Disaster recovery

If all fails, we need to restore PostgreSQL from backups. We have two
ways of restoring PostgreSQL backups, **indirect** and **direct**.

The **indirect** procedures first transfers the backup files to the
server and then runs the restore, so it require more space than the
direct procedure. It might also be slower than the direct restore
procedure because files are copied around twice: once from the backup
server, and again loaded in the database.

The **direct** procedure *streams* the backup files directly from the
backup server. The `BASE` file is streamed into the `tar` command for
restore of the base snapshot, then the PostgreSQL restore command
copies each log directly from the backup server as well. It requires
minimal amount of space, as files are not copied to a temporary
location on the new server. The downside is it might be more difficult
to deploy and diagnose, as it has more moving parts.

Both procedures are adaptations of the [official recovery
procedure](https://www.postgresql.org/docs/11/continuous-archiving.html#BACKUP-PITR-RECOVERY), which can be refered to for more information.

### Indirect restore procedures

Reminder: this procedure **copies** the backup files from the backup
server to the database server and restores from those.

 1. First, you will need to give the backup server access to the new
    postgresql server, which we'll call the "client" for now. First,
    login to the client and allow the backup server to connect, and
    show the public hostkey:

        iptables -I INPUT -s $BACKUP_SERVER -j ACCEPT
        cat /etc/ssh/ssh_host_rsa_key.pub

 2. Then load the server's private key in an agent and show it to
    allow on the client. On the server, assuming `$IP` is the IP of
    the client and `$HOSTKEY` is its hostkey:

        ssh-agent bash
        ssh-add /etc/ssh/ssh_host_rsa_key
        mkdir -p ~/.ssh
        echo "$IP $HOSTKEY" >> ~/.ssh/known_hosts
        cat /etc/ssh/ssh_host_rsa_key.pub

 3. And on the client, allow the server `$HOSTKEY` (the above `cat
    /etc/ssh/ssh_host_rsa_key.pub` on the backup server):

        echo "$HOSTKEY" >> /etc/ssh/userkeys/root.more

 4. Then, we can transfer files over from the backup server to the
    client:

        cd /srv/backups/pg
        rsync -aP $CLIENT $CLIENT:/var/lib/postgresql/restore

 5. Disable Puppet so you have control on when PostgreSQL is running:

        puppet agent --disable 'keeping control of postgresql startup -- anarcat 2019-10-09'

 6. Then, on the client, install the software, stop the server and
    move the template cluster out of the way:

        apt install postgres rsync
        service postgresql stop
        mv /var/lib/postgresql/*/main{,.orig}
        su -c 'mkdir -m 0700 /var/lib/postgresql/9.6/main' postgres

    We'll be restoring files in that `main` directory.

    Make sure you run the SAME MAJOR VERSION of PostgreSQL than the
    backup! You cannot restore across versions. This might mean
    installing from backports or an older version of Debian.

 7. Then you need to find the right `BASE` file to restore from. Each
    `BASE` file has a timestamp in its filename, so just sorting them
    by name should be enough to find the latest one. Uncompress the
    `BASE` file in place:

        cat ~postgres/restore/$CLIENT/main.BASE.bungei.torproject.org-20190805-145239-$CLIENT.torproject.org-main-9.6-backup.tar.gz | su postgres -c 'tar -C /var/lib/postgresql/9.6/main -x -z -f -'

    (Use `pv` instead of `cat` for a progress bar with large backups.)

 8. Make sure the `pg_xlog` directory doesn't contain any files.

 9. Then you need to create a `recovery.conf` file in
    `/var/lib/postgresql/9.6/main` that will tell postgres where to
    find the WAL files. At least the `restore_command` need to be
    specified. Something like this should work:

        restore_command = 'cp /var/lib/postgresql/restore/subnotabile/main.WAL.%f %p'

    You can specify a specific recovery point in the `recovery.conf`,
    see the [upstream documentation](https://www.postgresql.org/docs/11/recovery-target-settings.html) for more information.

 10. Then start the server and look at the logs to follow the recovery
     process:

        service postgresql start
        tail -f /var/log/postgresql/*

 11. re-enable Puppet, which should clean up the extra SSH key and
     firewall rules:

         puppet agent -t
    
     make sure it's okay in `/etc/ssh/userkeys/root` and `iptables -L`.

When the restore succeeds, the `recovery.conf` file will be renamed to
`recovery.done` and you will see something like:

    2019-08-12 21:36:53.453 UTC [16901] LOG:  selected new timeline ID: 2
    2019-08-12 21:36:53.470 UTC [16901] LOG:  archive recovery complete
    cp: cannot stat '/var/lib/postgresql/restore/subnotablie/main.WAL.00000001.history': No such file or directory
    2019-08-12 21:36:53.577 UTC [16901] LOG:  MultiXact member wraparound protections are now enabled
    2019-08-12 21:36:53.584 UTC [16900] LOG:  database system is ready to accept connections

Ignore the error from `cp` complaining about the `.history` file, it's
harmless.

#### Troubleshooting

If you find the following error in the logs:

    FATAL:  could not locate required checkpoint record

It's because postgres cannot find the WAL logs to restore from. There
could be many causes for this, but the ones I stumbled upon were:

 * wrong location for `recovery.conf` (it's in `/var`, not `/etc`),
   probably the case if you don't see an error from `cp`
 * wrong permissions on the archive (put the WAL files in `~postgres`,
   not `~root`)
 * wrong path or pattern for `restore_command` (double-check the path
   and make sure to include the right prefix, e.g. `main.WAL`)

### Direct restore procedure

Reminder: this procedure **streams** the files from the backup server
to the database server and restore directly. See above for more
information on how to pick a restore procedure.

 1. First, disable Puppet so you have control on when PostgreSQL is
    running:
 
        puppet agent --disable 'keeping control of postgresql startup -- anarcat 2019-10-09'

 2. Then install the right PostgreSQL version and stop the server:

        apt install postgres-9.6
        service postgresql stop

    Make sure you run the SAME MAJOR VERSION of PostgreSQL than the
    backup! You cannot restore across versions. This might mean
    installing from backports or an older version of Debian.

 3. On that new PostgreSQL server, show the `postgres` server public
    key, creating it if missing:

        [ -f ~postgres/.ssh/id_rsa.pub ] || sudo -u postgres ssh-keygen
        cat ~postgres/.ssh/*.pub

 4. Then on the backup server, allow the user to access backups of the
    old server:

        echo "command="/usr/local/bin/debbackup-ssh-wrap --read-allow=/srv/backups/pg/$OLDSERVER $CLIENT",restrict $HOSTKEY" > /etc/ssh/userkeys/torbackup.more

    This assumes we connect to a *previous* server's backups, named
    `$OLDSERVER` (e.g. `dictyotum`). The `$HOSTKEY` is the public key
    found on the postgres server above.

    Warning: the above will fail if the key is already present in
    `/etc/ssh/userkeys/torbackup`, edit the key in there instead in
    that case.

 5. Then you need to find the right `BASE` file to restore from. Each
    `BASE` file has a timestamp in its filename, so just sorting them
    by name should be enough to find the latest one. Uncompress the
    `BASE` file in place, as the `postgres` user:

        sudo -u postgres -i
        mkdir -m 0700 /var/lib/postgresql/9.6/main
        ssh torbackup@$BACKUPSERVER $(hostname) retrieve-file pg $OLDSERVER $BASEFILE \
          | sed '0,/^$/d' \
          | tar -C /var/lib/postgresql/9.6/main -x -z -f -

    Add a `pv` before the `tar` call in the pipeline for a progress bar
    with large backups, and replace:

     * `$BACKUPSERVER` with the backupserver name and username
        (currently `bungei.torproject.org`)
     * `$OLDSERVER` with the old server's (short) hostname
       (e.g. `dictyotum`)
     * `$CLUSTERNAME` with the name of the cluster to restore
       (e.g. usually `main`)
     * `$BASEFILE` with something like
       `$CLUSTERNAME.BASE.$BACKUPSERVER-20191004-062226-$OLDSERVER.torproject.org-$CLUSTERNAME-9.6-backup.tar.gz`
       or, fully expanded:
       `bacula.BASE.bungei.torproject.org-20191010-184205-dictyotum.torproject.org-bacula-9.6-backup.tar.gz`
       where the `$CLUSTERNAME` is the non-default `bacula`

    The above might hang for a while (yes, maybe even a minute) in the
    beginning, but it should eventually get started and complete. The
    "hang" is because `retrieve-file` sends a header which includes a
    `sha512sum` and it takes a while to compute. The `sed` command in
    that pipeline is there to skip that header. Example of a
    successful restore operated in [#31786][]:
   
        ssh torbackup@bungei.torproject.org bacula-director-01 retrieve-file pg dictyotum bacula.BASE.bungei.torproject.org-20191010-184205-dictyotum.torproject.org-bacula-9.6-backup.tar.gz | pv -s 33G | sed '0,/^$/d' | tar -C /var/lib/postgresql/9.6/main -x -z -f -

    If the above doesn't work, use the indirect procedure to restore
    the `BASE` file.

[#31786]: https://bugs.torproject.org/31786

 6. Make sure the `pg_xlog` directory doesn't contain any files.
 
        rm -rf -- /var/lib/postgresql/9.6/main/pg_xlog/*

 7. Then you need to create a `recovery.conf` file in
    `/var/lib/postgresql/9.6/main` that will tell postgres where to
    find the WAL files. At least the `restore_command` need to be
    specified. Something like this should work:

        restore_command = '/usr/local/bin/pg-receive-file-from-backup $OLDSERVER $CLUSTERNAME.WAL.%f %p'

    ... where:

     * `$OLDSERVER` should be replaced by the previous postgresql
       server name (e.g. `dictyotum`)
     * `$CLUSTERNAME` should be replaced by the previous cluster name
       (e.g. `main`, generally)

    You can specify a specific recovery point in the `recovery.conf`,
    see the [upstream documentation](https://www.postgresql.org/docs/11/recovery-target-settings.html) for more information. Also
    make sure the file is owned by postgres:

        $EDITOR /var/lib/postgresql/9.6/main/recovery.conf
        chown postgres /var/lib/postgresql/9.6/main/recovery.conf

 8. Then start the server and look at the logs to follow the recovery
    process:

        service postgresql start
        tail -f /var/log/postgresql/*

    You should see something like this:

        2019-10-09 21:17:47.335 UTC [9632] LOG:  database system was interrupted; last known up at 2019-10-04 08:12:28 UTC
        2019-10-09 21:17:47.517 UTC [9632] LOG:  starting archive recovery
        2019-10-09 21:17:47.524 UTC [9633] [unknown]@[unknown] LOG:  incomplete startup packet
        2019-10-09 21:17:48.032 UTC [9639] postgres@postgres FATAL:  the database system is starting up
        2019-10-09 21:17:48.538 UTC [9642] postgres@postgres FATAL:  the database system is starting up
        2019-10-09 21:17:49.046 UTC [9645] postgres@postgres FATAL:  the database system is starting up
        2019-10-09 21:17:49.354 UTC [9632] LOG:  restored log file "00000001000005B200000074" from archive
        2019-10-09 21:17:49.552 UTC [9648] postgres@postgres FATAL:  the database system is starting up
        2019-10-09 21:17:50.058 UTC [9651] postgres@postgres FATAL:  the database system is starting up
        2019-10-09 21:17:50.565 UTC [9654] postgres@postgres FATAL:  the database system is starting up
        2019-10-09 21:17:50.836 UTC [9632] LOG:  redo starts at 5B2/74000028
        2019-10-09 21:17:51.071 UTC [9659] postgres@postgres FATAL:  the database system is starting up
        2019-10-09 21:17:51.577 UTC [9665] postgres@postgres FATAL:  the database system is starting up
        2019-10-09 21:20:35.790 UTC [9632] LOG:  restored log file "00000001000005B20000009F" from archive
        2019-10-09 21:20:37.745 UTC [9632] LOG:  restored log file "00000001000005B2000000A0" from archive
        2019-10-09 21:20:39.648 UTC [9632] LOG:  restored log file "00000001000005B2000000A1" from archive
        2019-10-09 21:20:41.738 UTC [9632] LOG:  restored log file "00000001000005B2000000A2" from archive
        2019-10-09 21:20:43.773 UTC [9632] LOG:  restored log file "00000001000005B2000000A3" from archive

    ... and so on.

    TODO: this warning comes up from time to time, problem?

        2019-10-09 23:47:13.446 UTC [16973] LOG:  could not link file "pg_xlog/00000001000005B3000000C3" to "pg_xlog/00000001000005B3000000F9": File exists

    Then the recovery will complete with something like this:

        2019-10-10 01:30:55.460 UTC [16953] LOG:  redo done at 5B8/9C5BE738
        2019-10-10 01:30:55.460 UTC [16953] LOG:  last completed transaction was at log time 2019-10-10 01:04:23.238233+00
        2019-10-10 01:31:03.536 UTC [16953] LOG:  restored log file "00000001000005B80000009C" from archive
        2019-10-10 01:31:06.458 UTC [16953] LOG:  selected new timeline ID: 2
        2019-10-10 01:31:17.485 UTC [16953] LOG:  archive recovery complete
        2019-10-10 01:32:11.975 UTC [16953] LOG:  MultiXact member wraparound protections are now enabled
        2019-10-10 01:32:12.438 UTC [16950] LOG:  database system is ready to accept connections
        2019-10-10 01:32:12.439 UTC [26501] LOG:  autovacuum launcher started

 9. Then remove the temporary SSH access on the backup server, either
    by removing the `.more` key file or restoring the previous key
    configuration:

        rm /etc/ssh/userkeys/torbackup.more

 10. re-enable Puppet:

        puppet agent -t

#### Troubleshooting

If you get this error in the PostgreSQL restore logs:

    No directories from which read is allowed given on cmdline.
    Unkown format in line 1 () when getting dictyotum:main.WAL.00000005.history from backup host

It's because the SSH keys deployed on the server does not have the
`--read-allowed` argument.

If the restore doesn't work, try to connect by hand:

    sudo -u postgres /usr/local/bin/pg-receive-file-from-backup dictyotum bacula.WAL.00000001000005AB0000003A /tmp/foo

If you get this kind of errors, it's because you forgot to restore the
`BASE` file first:

    2019-10-08 21:25:43.711 UTC [14320] [unknown]@[unknown] LOG:  incomplete startup packet
    2019-10-08 21:25:44.215 UTC [14326] postgres@postgres FATAL:  the database system is starting up
    2019-10-08 21:25:44.451 UTC [14316] WARNING:  WAL was generated with wal_level=minimal, data may be missing
    2019-10-08 21:25:44.451 UTC [14316] HINT:  This happens if you temporarily set wal_level=minimal without taking a new base backup.
    2019-10-08 21:25:44.451 UTC [14316] LOG:  consistent recovery state reached at 0/153A3F0
    2019-10-08 21:25:44.451 UTC [14316] LOG:  invalid record length at 0/153A3F0: wanted 24, got 0
    2019-10-08 21:25:44.451 UTC [14316] LOG:  redo is not required
    2019-10-08 21:25:44.721 UTC [14334] postgres@postgres FATAL:  the database system is starting up
    2019-10-08 21:25:45.226 UTC [14340] postgres@postgres FATAL:  the database system is starting up
    2019-10-08 21:25:45.549 UTC [14316] LOG:  selected new timeline ID: 6
    2019-10-08 21:25:45.732 UTC [14343] postgres@postgres FATAL:  the database system is starting up
    2019-10-08 21:25:45.765 UTC [14316] LOG:  archive recovery complete
    2019-10-08 21:25:46.238 UTC [14349] postgres@postgres FATAL:  the database system is starting up
    2019-10-08 21:25:46.466 UTC [14316] LOG:  MultiXact member wraparound protections are now enabled
    2019-10-08 21:25:46.467 UTC [14315] LOG:  database system is ready to accept connections
    2019-10-08 21:25:46.467 UTC [14351] LOG:  autovacuum launcher started

See also the "Direct restore procedure" troubleshooting section, which
also applies here.

# Reference

## Installation

PostgreSQL servers are installed by hand on TPO servers, from Debian
packages. For example, in Debian bullseye:

    apt install postgresql-13

To ensure backups are properly performed, the server must also have
the `profile::postgres` class. The storage backup server also needs
`REPLICATION` access to make full backups. To set this up, follow the
steps below:

First, add the Puppet class to the new server, and run puppet on that
server and on the storage server (currently `bungei`) to create the
replication password (stored in trocla).

Then, create a backup user, using that password:

    sudo -u postgres createuser -D -E -P -R -S tor-backup

Give the role replication access:

    sudo -u postgres psql -c 'ALTER ROLE "tor-backup" REPLICATION;'

Add an entry to `pg_hba.conf` to allow access from the storage server
(currently `bungei`):

    hostssl replication     tor-backup      2a01:4f9:2b:1a05::2/128 md5
    hostssl replication     tor-backup      95.216.116.122/32       md5

Ensure pg is listening on all interfaces, that it is using TLS and a
proper auto-ca cert. This is generally with something like:

```
root@metrics-psqlts-01:~# cat /etc/postgresql/13/main/conf.d/listen.conf
listen_addresses = '*'
ssl_cert_file = '/etc/ssl/torproject-auto/servercerts/thishost.crt'
ssl_key_file = '/etc/ssl/torproject-auto/serverkeys/thishost.key'
```

Also ensure you have archival setup to backup to the storage server,
which generally looks like:

```
root@metrics-psqlts-01:~# cat /etc/postgresql/13/main/conf.d/wal.conf
track_counts = yes
archive_mode = on
wal_level = archive
#wal_level = hot_standby
max_wal_senders = 3
#wal_keep_segments = 8
archive_timeout = 1h
archive_command = '/usr/local/bin/pg-backup-file main WAL %p'
```

Reload db server.

Rerun Puppet on the new server so it realizes the firewall rules
exported from the storage server.

Make a full backup on the storage server to see if that works
correctly:

    sudo -u torbackup postgres-make-one-base-backup $(grep ^metrics-psqlts-01.torproject.org $(which postgres-make-base-backups ))

### Prometheus PostgreSQL exporter deployment

Prometheus metrics collection is not enabled by default. To enable it,
include the following line in `/etc/postgresql/11/main/pg_hba.conf`:

    local   all             prometheus                              peer

Then run the following SQL queries as the `postgres` user, for example
after `sudo -u postgres psql`, you first create the monitoring user to
match the above:

    -- To use IF statements, hence to be able to check if the user exists before
    -- attempting creation, we need to switch to procedural SQL (PL/pgSQL)
    -- instead of standard SQL.
    -- More: https://www.postgresql.org/docs/9.3/plpgsql-overview.html
    -- To preserve compatibility with <9.0, DO blocks are not used; instead,
    -- a function is created and dropped.
    CREATE OR REPLACE FUNCTION __tmp_create_user() returns void as $$
    BEGIN
      IF NOT EXISTS (
              SELECT                       -- SELECT list can stay empty for this
              FROM   pg_catalog.pg_user
              WHERE  usename = 'prometheus') THEN
        CREATE USER prometheus;
      END IF;
    END;
    $$ language plpgsql;
    
    SELECT __tmp_create_user();
    DROP FUNCTION __tmp_create_user();

This will make the user connect to the right database by default:

    ALTER USER prometheus SET SEARCH_PATH TO postgres_exporter,pg_catalog;
    GRANT CONNECT ON DATABASE postgres TO prometheus;

... and grant the required accesses to do the probes:

    GRANT pg_monitor to prometheus;

Note the procedure was modified from [the upstream procedure](https://github.com/prometheus-community/postgres_exporter/blob/972d8e885171e5d9d06e66a0d3e83454a7328002/README.md#running-as-non-superuser) to
use the `prometheus` user (instead of `postgres_exporter`), and to
remove the hardcoded password (since we rely on the "peer"
authentication method).

A previous version of this documentation mistakenly recommended
creating views and other complex objects that were only required in
PostgreSQL < 10, and were never actually necessary. Those can be
cleaned up with the following:

    DROP SCHEMA postgres_exporter CASCADE;
    DROP FUNCTION get_pg_stat_replication;
    DROP FUNCTION get_pg_stat_statements;
    DROP FUNCTION get_pg_stat_activity;

... and it wouldn't hurt then to rerun the above install procedure to
grant the correct rights to the `prometheus` user.

Then restart the exporter to be sure everything still works:

    systemctl restart prometheus-postgres-exporter.service

## SLA

No service level is defined for this service.

## Design

We use PostgreSQL for a handful of services. Each service has its own
PostgreSQL server installed, with no high availability or replication,
currently, although we use the "write-ahead log" to keep a binary dump
of databases on the backup server.

It should be noted for people unfamiliar with PostgreSQL that it (or
at least the Debian package) can manage multiple "clusters" of
distinct databases with overlapping namespaces, running on different
ports. To quote the [upstream documentation](https://www.postgresql.org/docs/11/tutorial-concepts.html):

> PostgreSQL is a *relational database management system*
> (RDBMS). That means it is a system for managing data stored in
> *relations*. Relation is essentially a mathematical term for
> *table*. \[...\]
>
> Each table is a named collection of *rows*. Each row of a given table
> has the same set of named *columns*, and each column is of a specific
> data type. \[...\]
>
> Tables are grouped into databases, and a collection of databases
> managed by a single PostgreSQL server instance constitutes a
> database *cluster*.

See also the [PostgreSQL architecture fundamentals](https://www.postgresql.org/docs/11/tutorial-arch.html).

## Issues

There is no issue tracker specifically for this project, [File][] or
[search][] for issues in the [team issue tracker][search] with the
~PostgreSQL label.

 [File]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new
 [search]: https://gitlab.torproject.org/tpo/tpa/team/-/issues?label_name%5B%5D=PostgreSQL

## Maintainer, users, and upstream

PostgreSQL services are part of the core services maintained by
TPA. The `postgres` Puppet module and associated backup
synchronisation code was written by Peter Palfrader.

The [PostgreSQL project](https://www.postgresql.org/) itself is a major database, free software
project, which calls itself "The World's Most Advanced Open Source
Relational Database, with regular releases and a healthy community.

## Monitoring and testing

The [Nagios](howto/nagios) monitoring server makes sure that a PostgreSQL process
is running on the host when the host is added to the relevant host
group (e.g. `postgres11-hosts`).

It also monitors the [bacula](howto/backups) storage server to make sure snapshots
are up to date. Backups are checked for freshness in Nagios using the
`dsa-check-backuppg` plugin with its configuration stored in
`/etc/dsa/postgresql-backup/dsa-check-backuppg.conf.d/`, per
cluster. The Nagios plugin also takes care of expiring backups when
they are healthy.

## Logs and metrics

PostgreSQL keeps log files in `/var/log/postgresql/`, one per
"cluster". Since it logs failed queries, logs may contain PII in the
form of SQL queries. The log rotation policy is the one set by the
Debian package and keeps logs for 10 weeks.

A [Prometheus](howto/prometheus) exporter is installed on the [GitLab](howto/gitlab) server by
the GitLab omnibus package, but metrics are not collected on other
Prometheus servers. The [Grafana](howto/grafana) server has a handful of
dashboards in various working states:

 * [Postgres Overview](https://grafana.torproject.org/d/wGgaPlciz/postgres-overview) - basic dashboard with minimal metrics
 * [PostgreSQL Overview (Percona)](https://grafana.torproject.org/d/IvhES05ik/postgresql-overview-percona) - mostly working
 * [GitLab Omnibus - PostgreSQL](https://grafana.torproject.org/d/c_LJgXfmk/gitlab-omnibus-postgresql) - broken

We do have a Puppet class (`profile::prometheus::postgres_exporter`
which can monitor PostgreSQL servers, but it is not deployed on all
hosts. This is because there is still a manual step in the
configuration of the exporter. See "installation" above.

It would also be preferable if that procedure was automated in Puppet,
but that requires haggling with the `postgresql` module which we do
not actually use, currently. It does not directly conflict with our
`postgres` module, but it does rewrite the `pg_hba.conf` and
`postges.conf`. If we do eventually fix this oddity, know that we
considered using `postgresql::server::grant`, but it might be better
to call `postgresql_psql` directly.

Note that there is a program called [pgstatsmon](https://github.com/joyent/pgstatsmon) which can provide
very detailed information about the state of a PostgreSQL database,
see [this blog post for details](https://www.davepacheco.net/blog/2019/visualizing-postgresql-vacuum-progress/).

## Backups

We use upstream's [Continuous Archiving and Point-in-Time Recovery
(PITR)](https://www.postgresql.org/docs/11/continuous-archiving.html) which relies on postgres's "write-ahead log" (WAL) to write
regular "transaction logs" of the cluster to the backup host. (Think
of transaction logs as incremental backups.) This is configured in
`postgresql.conf`, using a configuration like this:

    track_counts = yes
    archive_mode = on
    wal_level = archive
    max_wal_senders = 3
    archive_timeout = 6h
    archive_command = '/usr/local/bin/pg-backup-file main WAL %p'

The latter is a site-specific script which reads a config file in
`/etc/dsa/pg-backup-file.conf` where the backup host is specified
(currently `torbackup@bungei.torproject.org`). That command passes the
WAL logs onto the backup server, over SSH. A WAL file is shipped
immediately when it is full (16MB of data by default) but no later
than 6 hours (varies, see `archive_timeout` on each host) after it was
first written to. On the backup server, the `command` is set to
`debbackup-ssh-wrap` in the `authorized_keys` file and takes the
`store-file pg` argument to write the file to the right location.

WAL files are written to `/srv/backups/pg/$HOSTNAME` where `$HOSTNAME`
(without `.torproject.org`). WAL files are prefixed with `main.WAL.`
(where `main` is the cluster name) with a long unique string after,
e.g. `main.WAL.00000001000000A40000007F`.

For that system to work, we also need *full* backups to happen on a
regular basis. This happens straight from the backup server (still
`bungei`) which connects to the various postgres servers and runs a
[pg_basebackup](https://manpages.debian.org/pg_basebackup) to get a complete snapshot of the cluster. This
happens *weekly* (every 7 to 10 days) in the wrapper
`postgres-make-base-backups`, which is a wrapper (based on a Puppet
`concat::fragment` template) that calls
`postgres-make-one-base-backup` for each postgres server.

The base files are written to the same directory as WAL file and are
named using the template:

    $CLUSTER.BASE.$SERVER_FQDN-$DATE-$ID-$CLIENT_FQDN-$CLUSTER-$VERSION-backup.tar.gz

... for example:

    main.BASE.bungei.torproject.org-20190804-214510-troodi.torproject.org-main-9.6-backup.tar.gz

All of this works because SSH public keys and postgres credentials are
passed around between servers. That is handled in the Puppet
`postgresql` module for the most part, but some bits might still be
configured manually on some servers.

The Nagios plugin (see above) takes care of expiring snapshots.

## Other documentation

See also:

 * [PostgreSQL official documentation](https://www.postgresql.org/docs/manuals/)
 * [PostgreSQL wiki](https://wiki.postgresql.org/wiki/Main_Page)
 * [Debian DSA documentation](https://dsa.debian.org/howto/postgres-backup/)

# Discussion

## Overview

<!-- describe the overall project. should include a link to a ticket -->
<!-- that has a launch checklist -->

<!-- if this is an old project being documented, summarize the known -->
<!-- issues with the project. to quote the "audit procedure":

 5. When was the last security review done on the project? What was
    the outcome? Are there any security issues currently? Should it
    have another security review?

 6. When was the last risk assessment done? Something that would cover
    risks from the data stored, the access required, etc.

 7. Are there any in-progress projects? Technical debt cleanup?
    Migrations? What state are they in? What's the urgency? What's the
    next steps?

 8. What urgent things need to be done on this project?

-->

## Goals

<!-- include bugs to be fixed -->

### Must have

### Nice to have

### Non-Goals

## Approvals required

<!-- for example, legal, "vegas", accounting, current maintainer -->

## Proposed Solution

## Cost

## Alternatives considered

<!-- include benchmarks and procedure if relevant -->
