spellcheck backups howto

Longer version:

 1. enter the console on the bacula director:

 1. enter the console on the Bacula director:

        ssh -tt bacula-director-01.torproject.org bconsole

        OK to run? (yes/mod/no): yes

        Job queued. JobId=113225

 4. bacula confirms the job is `queued`. you can see the status of the

 4. Bacula confirms the job is `queued`. you can see the status of the

    job with `status director`, which should show set of lines like

    this in the middle:

point in time.

The first thing to know is that restores are done from the server to

the client, ie. they are restored directly on the machine that is

the client, i.e. they are restored directly on the machine that is

backed up. Note that by default files will be owned by the `bacula`

user because the file daemon runs as `bacula` in our configuration. If

that's a problem for large backups, the override (in

        Select the Client (1-117): 87

 5. you now get dropped in a file browser where you use the `mark` and

    `unmark` commands to mark and unmark files for restore. the

    `unmark` commands to mark and un-mark files for restore. the

    commands support wildcards like `*`. use `mark *` to mark all

    files in the current directory, see also the [full list of

    commands](https://www.bacula.org/9.4.x-manuals/en/main/Restore_Command.html#SECTION0026130000000000000000):

In this case you might have to manually recreate a few configuration files both on `bacula-director-01` and on `bungei`.

On bungei:

On `bungei`:

```

cd /etc/bacula/storage-conf.d

```

Create a conf file like the following (you can also copy and edit one of the files from the other hosts):

Create a configuration file like the following (you can also copy and edit one of the files from the other hosts):

```

##

}

```

Disable puppet and restart bacula-sd:

Disable puppet and restart `bacula-sd`:

```

puppet agent --disable 'adding conf for <hostname> manually'

service bacula-sd restart

```

On bacula-director-01:

On `bacula-director-01`:

```

cd /etc/conf.d

```

Create two conf file:

Create two configuration files:

```

<hostname>.torproject.org.conf

<hostname>.torproject.org_storage.conf

File-<hostname>.torproject.org

```

Disable puppet and restart bacula-director:

Disable puppet and restart `bacula-director`:

```

puppet agent --disable 'adding conf for <hostname> manually'

### Get files without a director

If you want to get to files stored on the bacula storgage host without

If you want to get to files stored on the Bacula storage host without

involving the director, they can be accessed directly as well.  Remember

that to bacula everything is a tape, and `/srv/backups/bacula` is full

that to Bacula everything is a tape, and `/srv/backups/bacula` is full

of directories of tapes.  You can see the contents of a tape using

[bls](https://www.bacula.org/7.2.x-manuals/en/utility/Volume_Utility_Tools.html#115),

that is, `bls <file>`, with a fully qualified filename, i.e. involving all the

    bextract /srv/backups/bacula/dictyotum.torproject.org/torproject-inc-dictyotum.torproject.org.2019-09-25_11:53 /var/tmp/restore

This will extract the entire tape to `/var/tmp/restore`.  If you want only a few files,

put their names into a file such as `include` and call bextract with `-i`:

put their names into a file such as `include` and call `bextract` with `-i`:

    bextract -i ~/include /srv/backups/bacula/dictyotum.torproject.org/torproject-inc-dictyotum.torproject.org.2019-09-25_11:53 /var/tmp/restore

If a job is behaving strangely, you can inspect its job log to see

what's going on. For example, today Nagios warned about the backups

being too old on colchicifolium:

being too old on `colchicifolium`:

    10:02:58 <nsa> tor-nagios: [colchicifolium] backup - bacula - last full backup is WARNING: WARN: Last backup of colchicifolium.torproject.org/F is 45.16 days old.

Looking at the bacula director status, it says this:

Looking at the Bacula director status, it says this:

    Console connected using TLS at 10-Jan-20 18:19

     JobId  Type Level     Files     Bytes  Name              Status

    120468  Back Diff     30,694    3.353 G gitlab-01.torproject.org is running

    ====

Which is strange because those JobId numbers are very low compared to

(say) the gitlab backup job. To inspect the job log, you use the

Which is strange because those `JobId` numbers are very low compared to

(say) the GitLab backup job. To inspect the job log, you use the

`list` command:

    *list joblog jobid=120225

        sudo -u postgres psql -c '\password bacula-dictyotum-reader'

 9. reset the password of the bacula director, as it changed in

 9. reset the password of the Bacula director, as it changed in

    puppet:

        grep dbpassword /etc/bacula/bacula-dir.conf | cut -f2 -d\"

       `archive_command1`

     * the `ssl_cert_file` and `ssl_key_file` point to valid SSL certs

 11. Once you have the postgres database cluster restored, start the

 11. Once you have the PostgreSQL database cluster restored, start the

     director:

         systemctl start bacula-director

        killall sleep

 16. switch the nagios checks over the new director: grep for the old

     director name in the nagios configuration and fix up some of the

 16. switch the Nagios checks over the new director: grep for the old

     director name in the Nagios configuration and fix up some of the

     checks

        git -C tor-nagios grep dictyotum

 17. you will also need to restore the password file for the nagios

 17. you will also need to restore the password file for the Nagios

     check in `/etc/nagios/bacula-database`

 18. switch the director in `/etc/dsa/bacula-reader-database` or

TODO: `15:19:55 <weasel> and once that's up and running, it'd probably be smart to upgrade it to 11.  pg_upgradecluster -m upgrade --link`

TODO: some psql users still refer to host-specific usernames like

`bacula-dictyotum-reader`, maybe they should refer to role-specif

`bacula-dictyotum-reader`, maybe they should refer to role-specific

names instead?

### Troubleshooting

    port=5433

to that file, or specify the `dbname` and `port` manually in the

config file.

configuration file.

If the scheduler is sending you an email every three minutes with this

error:

compromise on a machine shouldn't allow an attacker to delete backups

from the backup server.

Bacula splits the different responsabilities of the backup system

Bacula splits the different responsibilities of the backup system

among multiple components, namely:

 * storage daemon (`bacula::storage` in Puppet, currently `bungei`)

the compressed database dump, which is sufficient to re-bootstrap the

director.

See the [introductio to Bacula](https://www.bacula.org/9.4.x-manuals/en/main/What_is_Bacula.html#SECTION00220000000000000000) for more information on those

See the [introduction to Bacula](https://www.bacula.org/9.4.x-manuals/en/main/What_is_Bacula.html#SECTION00220000000000000000) for more information on those

distinctions.

### PostgreSQL backup system

Database backups are handled specially. We use PostgreSQL (postgres)

everywhere apart from a few rare exceptions (currently only CiviCRM)

and therefore use postgres-specific configurations to do backups of

all our servers.

Database backups are handled specially. We use PostgreSQL everywhere

apart from a few rare exceptions (currently only CiviCRM) and

therefore use postgres-specific configurations to do backups of all

our servers.

See [howto/postgresql](howto/postgresql) for that server's specific backup/restore

instructions.