openstack.md

Dealing with OpenStack

We were granted access to Linaro's OpenStack cluster. The following instructions were originally written to create virtual machines in that cluster, but were adapted to also work on OSUOSL's clusters.

We provide command line instructions below because they are easier to document, but an equivalent configuration can be performed through the web interface as well.

Preparation

You first need a adminrc.sh file with the right configuration and credentials.

In general, the credentials it can be downloaded in API access page of an OpenStack web UI (/project/api_access/) by clicking on the Download OpenStack RC File. We call the downloaded file the adminrc.sh file, but it can be named anything, as long as it's source in your shell for the following commands.

Here are platform-specific instructions:

• the credentials for Linaro were extracted from ticket 453 and the password prompted is the login.linaro.org SSO password stored in tor-passwords.git (note: the domain is linaro).
• the OSUOSL password is in tor-passwords.git

Then you need to install some OpenStack clients:

apt install openstack-clients

Yes, that installs 74 packages, no kidding.

Add your SSH key to the server:

openstack keypair create --public-key=~/.ssh/id_rsa.pub anarcat

If your key is stored in GnuPG:

openstack keypair create --public-key=<(gpg --export-ssh-key anarcat@debian.org) anarcat

You will probably need to edit the default security group (or create a new one) to allow ingress traffic as well. For example, this will create an "allow all" ingress rule on IPv4:

openstack security group rule create default

During this entire process, it's useful to take a look at the effect of the various steps through the web interface.

Launching an instance

This procedure will create a new VM in the OpenStack cluster. Make sure you first source the adminrc.sh script you found in the previous step.

1. list the known flavors and images:

   openstack flavor list
   openstack image list

   let's say we deploy a uk.nano flavor with debian-10-openstack-arm64 image.

2. create the server (known as an "instance" in the GUI):

   openstack server create --key-name=anarcat --security-group=default --image=debian-10-openstack-arm64 --flavor=uk.nano build-arm-10.torproject.org

   In the above:

   • --keypair=anarcat refers to the keypair created in the preparation
   • --security-group is taken from openstack security group list output, which typically has a default one. in previous installs, we setup a security group through the web interface possibly to allow the floating IP routing (unclear)
   • --image and --flavor were picked from the previous step

3. you can see the status of the process with:

   openstack server list

4. inspect the server console log to fetch the SSH public keys:

   openstack console log show build-arm-10.torproject.org | sed '0,/-----BEGIN SSH HOST KEY KEYS-----/d;/-----END SSH HOST KEY KEYS-----/,$d;s/^/213.146.141.28 /' >> ~/.ssh/known_hosts

   Note: the above doesn't actually work. In my tests (on OSUOSL) the keys do show up in the web console, but not in the above command. Use this command to load the web console:

   openstack console url show build-arm-10.torproject.org

5. the VM should be up by now, and you should be able to SSH in:

   openstack server ssh -l debian build-arm-10.torproject.org

   You unfortunately have to blindly TOFU (Trust On First Use) the SSH server's public key because it's not visible in the API or web interface. The debian user has sudo access.

Note that the above might fail on OSUOSL's OpenStack cluster sometimes. The symptom is that the host would be named "unassigned-hostname" (visible in the console) and SSH login would be impossible. Sometimes, the console would also display this message:

no authorized SSH keys fingerprints found for user debian

This is cloud-init failing to fetch the configuration from the metadata service. This is an upstream issue with OSUOSL, file an issue with them (aarch64-hosting-request@osuosl.org), documenting the problem. Our previous ticket for this was [support.osuosl.org #31901] and was resolved upstream by restarting the metadata service.

Floating IP configuration

The above may fail in some OpenStack clusters that allocate RFC1918 private IP addresses to new instances. In those case, you need to allocate a floating IP and route it to the instance.

1. create a floating IP

   openstack floating ip create ext-net

   The IP address will be shown in the output:

   | floating_ip_address | 213.146.141.28 |

   The network name (ext-net above) can be found in the network list:

   openstack network list

2. link the router in the private network if not already done:

   openstack router add subnet router-tor 7452852a-8b5c-43f6-97f1-72b1248b2638

   The subnet UUID comes from the Subnet column in the output of openstack network list for the "internal network" (the one that is not ext-net.

3. map the floating IP address to the server:

   openstack server add floating ip build-arm-10.torproject.org 213.146.141.28

References

• How to launch an instance