# Dealing with OpenStack

We were granted access to Linaro's OpenStack cluster. The following
instructions were originally written to create virtual machines in
that cluster, but were adapted to also work on OSUOSL's clusters.

We provide command line instructions below because they are easier to
document, but an equivalent configuration can be performed through the
web interface as well.

## Preparation

You first need a `adminrc.sh` file with the right configuration and
credentials. 

In general, the credentials it can be downloaded in API access page of
an OpenStack web UI (`/project/api_access/`) by clicking on the
`Download OpenStack RC File`. We call the downloaded file the
`adminrc.sh` file, but it can be named anything, as long as it's
source in your shell for the following commands.

Here are platform-specific instructions:

 * the credentials for Linaro were extracted from [ticket 453](https://servicedesk.linaro.org/servicedesk/customer/portal/11/DC-453) and
   the password prompted is the `login.linaro.org` SSO password stored
   in `tor-passwords.git` (note: the `domain` is `linaro`).
 * the OSUOSL password is in `tor-passwords.git`

Then you need to install some OpenStack clients:

    apt install openstack-clients

Yes, that installs 74 packages, no kidding.

Add your SSH key to the server:

    openstack keypair create --public-key=~/.ssh/id_rsa.pub anarcat

If your key is stored in GnuPG:

    openstack keypair create --public-key=<(gpg --export-ssh-key anarcat@debian.org) anarcat

You will probably need to edit the default security group (or create a
new one) to allow ingress traffic as well. For example, this will
create an "allow all" ingress rule on IPv4:

    openstack security group rule create default

During this entire process, it's useful to take a look at the effect
of the various steps through the web interface.

## Launching an instance

This procedure will create a new VM in the OpenStack cluster. Make
sure you first source the `adminrc.sh` script you found in the
previous step.

 1. list the known flavors and images:
 
        openstack flavor list
        openstack image list

    let's say we deploy a `uk.nano` flavor with
    `debian-10-openstack-arm64` image. 

 2. create the server (known as an "instance" in the GUI):
 
        openstack server create --key-name=anarcat --security-group=default --image=debian-10-openstack-arm64 --flavor=uk.nano build-arm-10.torproject.org

    In the above:
    
     * `--keypair=anarcat` refers to the keypair created in the
       preparation
     * `--security-group` is taken from `openstack security group
       list` output, which typically has a `default` one. in previous
       installs, we setup a security group through the web interface
       possibly to allow the floating IP routing (unclear)
     * `--image` and `--flavor` were picked from the previous step

 3. you can see the status of the process with:
 
        openstack server list

 5. inspect the server console log to fetch the SSH public keys:

        openstack console log show build-arm-10.torproject.org | sed '0,/-----BEGIN SSH HOST KEY KEYS-----/d;/-----END SSH HOST KEY KEYS-----/,$d;s/^/213.146.141.28 /' >> ~/.ssh/known_hosts

    Note: the above doesn't actually work. In my tests (on OSUOSL) the
    keys do show up in the web console, but not in the above
    command. Use this command to load the web console:
    
        openstack console url show build-arm-10.torproject.org

 6. the VM should be up by now, and you should be able to SSH in:
 
        openstack server ssh -l debian build-arm-10.torproject.org

    You unfortunately have to blindly TOFU (Trust On First Use) the
    SSH server's public key because it's not visible in the API or web
    interface. The `debian` user has `sudo` access.

Note that the above might fail on OSUOSL's OpenStack cluster
sometimes. The symptom is that the host would be named
"unassigned-hostname" (visible in the console) and SSH login would be
impossible. Sometimes, the console would also display this message:

    no authorized SSH keys fingerprints found for user debian

This is cloud-init failing to fetch the configuration from the
metadata service. This is an upstream issue with OSUOSL, file an issue
with them (aarch64-hosting-request@osuosl.org), documenting the
problem. Our previous ticket for this was `[support.osuosl.org
#31901]` and was resolved upstream by restarting the metadata service.

## Floating IP configuration

The above may fail in some OpenStack clusters that allocate RFC1918
private IP addresses to new instances. In those case, you need to
allocate a floating IP and route it to the instance.

 1. create a floating IP

        openstack floating ip create ext-net

    The IP address will be shown in the output:
    
    | floating_ip_address | 213.146.141.28 |

    The network name (`ext-net` above) can be found in the network list:
    
        openstack network list

 2. link the router in the private network if not already done:
    
        openstack router add subnet router-tor 7452852a-8b5c-43f6-97f1-72b1248b2638

    The subnet UUID comes from the `Subnet` column in the output of
    `openstack network list` for the "internal network" (the one that
    is not `ext-net`.

 3. map the floating IP address to the server:

        openstack server add floating ip build-arm-10.torproject.org 213.146.141.28

# References

 * [How to launch an instance](https://docs.openstack.org/install-guide/launch-instance.html)
