... | ... | @@ -2477,7 +2477,7 @@ address blocks reserved in the cluster. |
|
|
|
|
|
rm /etc/no_modules_disabled
|
|
|
|
|
|
9. run puppet across the ganeti cluster to firewalls are correctly
|
|
|
9. run puppet across the Ganeti cluster so firewalls are correctly
|
|
|
configured:
|
|
|
|
|
|
cumin -p 0 'C:roles::ganeti::chi' 'puppet agent -t'
|
... | ... | @@ -2575,6 +2575,163 @@ The following IPs were reserved: |
|
|
The first two are for the gateway, but the rest is temporary and might
|
|
|
be reclaimed eventually.
|
|
|
|
|
|
### New gnt-dal node
|
|
|
|
|
|
1. To create a new box, follow [howto/quintex](howto/quintex) tutorial
|
|
|
|
|
|
2. follow the [howto/new-machine](howto/new-machine) post-install configuration
|
|
|
|
|
|
3. Allocate a private IP address for the node in the
|
|
|
`30.172.in-addr.arpa` zone and `torproject.org` zone, in the
|
|
|
`admin/dns/domains.git` repository
|
|
|
|
|
|
4. add the private IP address to the eth1 interface, for example in
|
|
|
`/etc/network/interfaces.d/eth1`:
|
|
|
|
|
|
auto eth2
|
|
|
iface eth2 inet static
|
|
|
address 172.30.131.101/24
|
|
|
|
|
|
Again, this IP must be allocated in the reverse DNS zone file
|
|
|
(`30.172.in-addr.arpa`) and the `torproject.org` zone file in the
|
|
|
`dns/domains.git` repository.
|
|
|
|
|
|
5. enable the interface:
|
|
|
|
|
|
ifup eth2
|
|
|
|
|
|
6. setup a bridge on the public interface, replacing the `eth0` blocks
|
|
|
with something like:
|
|
|
|
|
|
auto eth0
|
|
|
iface eth0 inet manual
|
|
|
|
|
|
auto br0
|
|
|
iface br0 inet static
|
|
|
address 204.8.99.101/24
|
|
|
gateway 204.8.99.254
|
|
|
bridge_ports eth0
|
|
|
bridge_stp off
|
|
|
bridge_fd 0
|
|
|
|
|
|
# IPv6 configuration
|
|
|
iface br0 inet6 static
|
|
|
accept_ra 0
|
|
|
address 2620:7:6002:0:3eec:efff:fed5:6b2a/64
|
|
|
gateway 2620:7:6002::1
|
|
|
|
|
|
6. allow modules to be loaded, cross your fingers that you didn't
|
|
|
screw up the network configuration above, and reboot:
|
|
|
|
|
|
touch /etc/no_modules_disabled
|
|
|
reboot
|
|
|
|
|
|
7. configure the node in Puppet by adding it to the
|
|
|
`roles::ganeti::chi` class, and run Puppet on the new node:
|
|
|
|
|
|
puppet agent -t
|
|
|
|
|
|
8. re-disable module loading:
|
|
|
|
|
|
rm /etc/no_modules_disabled
|
|
|
|
|
|
9. run puppet across the Ganeti cluster so firewalls are correctly
|
|
|
configured:
|
|
|
|
|
|
cumin -p 0 'C:roles::ganeti::chi' 'puppet agent -t'
|
|
|
|
|
|
10. Then the node is ready to be added to the cluster, by running
|
|
|
this on the master node:
|
|
|
|
|
|
gnt-node add \
|
|
|
--secondary-ip 172.30.131.101 \
|
|
|
--no-ssh-key-check \
|
|
|
--no-node-setup \
|
|
|
dal-node-01.torproject.org
|
|
|
|
|
|
If this is an entirely new cluster, you need a different
|
|
|
procedure, see [the cluster initialization procedure](#gnt-fsn-cluster-initialization) instead.
|
|
|
|
|
|
11. make sure everything is great in the cluster:
|
|
|
|
|
|
gnt-cluster verify
|
|
|
|
|
|
If the last step fails with SSH errors, you may need to re-synchronise
|
|
|
the SSH `known_hosts` file, see [SSH key verification failures](#ssh-key-verification-failures).
|
|
|
|
|
|
### gnt-dal cluster initialization
|
|
|
|
|
|
This procedure replaces the `gnt-node add` step in the initial setup
|
|
|
of the first Ganeti node when the `gnt-chi` cluster was setup:
|
|
|
|
|
|
gnt-cluster init \
|
|
|
--master-netdev eth2 \
|
|
|
--nic-parameters link=br0 \
|
|
|
--vg-name vg_ganeti \
|
|
|
--secondary-ip 172.30.131.101 \
|
|
|
--enabled-hypervisors kvm \
|
|
|
--mac-prefix 06:66:39 \
|
|
|
--no-ssh-init \
|
|
|
--no-etc-hosts \
|
|
|
dalgnt.torproject.org
|
|
|
|
|
|
The above assumes that `dalgnt` is already in DNS. See the [MAC
|
|
|
address prefix selection](#mac-address-prefix-selection) section for information on how the
|
|
|
`--mac-prefix` argument was selected.
|
|
|
|
|
|
Then the following extra configuration was performed:
|
|
|
|
|
|
```
|
|
|
gnt-cluster modify --reserved-lvs vg_system/root,vg_system/swap
|
|
|
gnt-cluster modify -H kvm:kernel_path=,initrd_path=
|
|
|
gnt-cluster modify -H kvm:security_model=pool
|
|
|
gnt-cluster modify -H kvm:kvm_extra='-device virtio-rng-pci\,bus=pci.0\,addr=0x1e\,max-bytes=1024\,period=1000'
|
|
|
gnt-cluster modify -H kvm:disk_cache=none
|
|
|
gnt-cluster modify -H kvm:disk_discard=unmap
|
|
|
gnt-cluster modify -H kvm:scsi_controller_type=virtio-scsi-pci
|
|
|
gnt-cluster modify -H kvm:disk_type=scsi-hd
|
|
|
gnt-cluster modify -H kvm:migration_bandwidth=950
|
|
|
gnt-cluster modify -H kvm:migration_downtime=500
|
|
|
gnt-cluster modify -H kvm:migration_caps=postcopy-ram
|
|
|
gnt-cluster modify -D drbd:c-plan-ahead=0,disk-custom='--c-plan-ahead 0'
|
|
|
gnt-cluster modify --uid-pool 4000-4019
|
|
|
```
|
|
|
|
|
|
The upper limit for CPU count and memory size were doubled, to 16 and
|
|
|
64G, respectively, with:
|
|
|
|
|
|
```
|
|
|
gnt-cluster modify --ipolicy-bounds-specs \
|
|
|
max:cpu-count=16,disk-count=16,disk-size=1048576,\
|
|
|
memory-size=65536,nic-count=8,spindle-use=12\
|
|
|
/min:cpu-count=1,disk-count=1,disk-size=1024,\
|
|
|
memory-size=128,nic-count=1,spindle-use=1
|
|
|
```
|
|
|
|
|
|
NOTE: watch out for whitespace here. The [original source](https://johnny85v.wordpress.com/2016/06/13/ganeti-commands/) for this
|
|
|
command had too much whitespace, which fails with:
|
|
|
|
|
|
Failure: unknown/wrong parameter name 'Missing value for key '' in option --ipolicy-bounds-specs'
|
|
|
|
|
|
The [network configuration](#network-configuration) (below) must also be performed for the
|
|
|
address blocks reserved in the cluster. This is the actual initial
|
|
|
configuration performed:
|
|
|
|
|
|
gnt-network add --network 204.8.99.128/25 --gateway 204.8.99.254 --network6 2620:7:6002::/64 --gateway6 2620:7:6002:1 gnt-dal-01
|
|
|
gnt-network connect --nic-parameters=link=br0 gnt-dal-01 default
|
|
|
|
|
|
Note that we reserve the first `/25` (209.44.8.99.0/25) for future
|
|
|
use. The above only uses the second half of the network in case we
|
|
|
need the rest of the network for other operations. A new network will
|
|
|
need to be added if we run out of IPs in the second half. This also
|
|
|
|
|
|
The following IPs were reserved:
|
|
|
|
|
|
gnt-network modify --add-reserved-ips=204.8.99.254 gnt-dal-01
|
|
|
|
|
|
This is just for the gateway. The node's public addresses are in the
|
|
|
other /25 and do not need to be reserved in this allocation.
|
|
|
|
|
|
### Network configuration
|
|
|
|
|
|
IP allocation is managed by Ganeti through the `gnt-network(8)`
|
... | ... | |