... | @@ -147,6 +147,111 @@ DRBD, in the FSN cluster: |
... | @@ -147,6 +147,111 @@ DRBD, in the FSN cluster: |
|
--backend-parameters memory=1g,vcpus=1 \
|
|
--backend-parameters memory=1g,vcpus=1 \
|
|
test-01.torproject.org
|
|
test-01.torproject.org
|
|
|
|
|
|
|
|
Do not forget to follow the [next steps](#next-steps), above.
|
|
|
|
|
|
|
|
### iSCSI integration
|
|
|
|
|
|
|
|
To create a VM with iSCSI backing, a disk must first be created on the
|
|
|
|
SAN, then adopted in a VM, which needs to be *reinstalled* on top of
|
|
|
|
that. This is typical how large disks are provisionned in the
|
|
|
|
`gnt-chi` cluster, in the [Cymru POP](howto/new-machine-cymru).
|
|
|
|
|
|
|
|
The following instructions assume you are on a node with an [iSCSI
|
|
|
|
initiator properly setup](howto/new-machine-cymru#iscsi-initiator-setup), and the [SAN cluster management tools
|
|
|
|
setup](howto/new-machine-cymru#san-management-tools-setup). It also assumes you are familiar with the `SMcli` tool, see
|
|
|
|
the [storage servers documentation](howto/new-machine-cymru#storage-servers) for an introduction on that.
|
|
|
|
|
|
|
|
This assumes you are creating a 500GB VM, partitioned on the Linux
|
|
|
|
host, *not* on the iSCSI volume. TODO: change those instructions to
|
|
|
|
create one volume per partition, so that those can be resized more
|
|
|
|
easily. This is how `web-chi-03` was setup.
|
|
|
|
|
|
|
|
1. create the disk on the SAN and assign it to the host group:
|
|
|
|
|
|
|
|
create virtualDisk physicalDiskCount=3 raidLevel=5 userLabel="web-chi-03" capacity=500GB;
|
|
|
|
set virtualDisk ["web-chi-03"] logicalUnitNumber=4 hostGroup="gnt-chi";
|
|
|
|
|
|
|
|
NOTE: the `logicalUnitNumber` here must be an increment from the
|
|
|
|
previous highest LUN. See also the [disk creation instructions](howto/new-machine-cymru#creating-a-disk)
|
|
|
|
for a discussion.
|
|
|
|
|
|
|
|
2. detect the new device on the Linux side:
|
|
|
|
|
|
|
|
iscsiadm -m node --rescan
|
|
|
|
ls -altr /dev/disk/by-path/*lun-4
|
|
|
|
|
|
|
|
TODO: is the `... --rescan` necessary?
|
|
|
|
|
|
|
|
3. find the associated WWID:
|
|
|
|
|
|
|
|
/lib/udev/scsi_id -g -u -d /dev/sdl
|
|
|
|
|
|
|
|
4. configure the disk on all Ganeti nodes, in Puppet's
|
|
|
|
`profile::ganeti::chi` class:
|
|
|
|
|
|
|
|
iscsi::multipath::alias { 'web-chi-03':
|
|
|
|
wwid => '36782bcb00063c6a500000d67603f7abf',
|
|
|
|
}
|
|
|
|
|
|
|
|
5. confirm that multipath works, it should look something like this":
|
|
|
|
|
|
|
|
root@chi-node-01:~# multipath -ll
|
|
|
|
web-chi-03-srv (36782bcb00063c6a500000d67603f7abf) dm-20 DELL,MD32xxi
|
|
|
|
size=500G features='5 queue_if_no_path pg_init_retries 50 queue_mode mq' hwhandler='1 rdac' wp=rw
|
|
|
|
|-+- policy='round-robin 0' prio=6 status=active
|
|
|
|
| |- 11:0:0:4 sdi 8:128 active ready running
|
|
|
|
| |- 12:0:0:4 sdj 8:144 active ready running
|
|
|
|
| `- 9:0:0:4 sdh 8:112 active ready running
|
|
|
|
`-+- policy='round-robin 0' prio=1 status=enabled
|
|
|
|
|- 10:0:0:4 sdk 8:160 active ghost running
|
|
|
|
|- 7:0:0:4 sdl 8:176 active ghost running
|
|
|
|
`- 8:0:0:4 sdm 8:192 active ghost running
|
|
|
|
root@chi-node-01:~#
|
|
|
|
|
|
|
|
and the device `/dev/mapper/web-chi-03` should exist.
|
|
|
|
|
|
|
|
5. partition the disk with `parted`:
|
|
|
|
|
|
|
|
parted --script --align optimal /dev/mapper/web-chi-03 \
|
|
|
|
mklabel gpt \
|
|
|
|
mkpart primary 0% 8MB \
|
|
|
|
set 1 bios_grub on \
|
|
|
|
mkpart primary 8MB 10008MB \
|
|
|
|
mkpart primary 10008MB 18008MB \
|
|
|
|
mkpart primary 18008MB 100%
|
|
|
|
|
|
|
|
TODO: this is one step that would be skipped if we have one iSCSI
|
|
|
|
volume per partition, obviously.
|
|
|
|
|
|
|
|
TODO: we probably do not need that `bios_grub` partition either.
|
|
|
|
|
|
|
|
6. adopt the disks in Ganeti:
|
|
|
|
|
|
|
|
gnt-instance add \
|
|
|
|
-o debootstrap+buster \
|
|
|
|
-t drbd --no-wait-for-sync \
|
|
|
|
--net 0:ip=pool,network=gnt-chi-01 \
|
|
|
|
--no-ip-check \
|
|
|
|
--no-name-check \
|
|
|
|
--disk 0:adopt=/dev/disk/by-id/dm-name-web-chi-03-part2 \
|
|
|
|
--disk 1:adopt=/dev/disk/by-id/dm-name-web-chi-03-part3,name=swap \
|
|
|
|
--disk 2:adopt=/dev/disk/by-id/dm-name-web-chi-03-part4
|
|
|
|
--backend-parameters memory=8g,vcpus=2 \
|
|
|
|
web-chi-03.torproject.org
|
|
|
|
|
|
|
|
7. at this point, the VM probably doesn't boot, because for some
|
|
|
|
reason the `gnt-instance-debootstrap` doesn't fire when disks are
|
|
|
|
adopted. so you need to reinstall the machine, which involves
|
|
|
|
stopping it first:
|
|
|
|
|
|
|
|
gnt-instance shutdown --timeout=0 web-chi-03
|
|
|
|
gnt-instance reinstall
|
|
|
|
|
|
|
|
From here on, follow the [next steps](#next-steps) above.
|
|
|
|
|
|
|
|
TODO: This would ideally be automated by an external storage provider,
|
|
|
|
see the [storage reference for more information](#storage).
|
|
|
|
|
|
## Modifying an instance
|
|
## Modifying an instance
|
|
|
|
|
|
### CPU, memory changes
|
|
### CPU, memory changes
|
... | @@ -1764,9 +1869,22 @@ particular. |
... | @@ -1764,9 +1869,22 @@ particular. |
|
|
|
|
|
See also the [DRBD documentation](howto/drbd).
|
|
See also the [DRBD documentation](howto/drbd).
|
|
|
|
|
|
NOTE: the Cymru PoP has an iSCSI cluster for large filesystem
|
|
The Cymru PoP has an iSCSI cluster for large filesystem
|
|
storage. See the [cymru documentation for
|
|
storage. Ideally, this would be automated inside Ganeti, some quick
|
|
details](howto/new-machine-cymru#ganeti-iscsi-integration) for details.
|
|
links:
|
|
|
|
|
|
|
|
* [search for iSCSI in the ganeti-devel mailing list](https://www.mail-archive.com/search?l=ganeti-devel@googlegroups.com&q=iscsi&submit.x=0&submit.y=0)
|
|
|
|
* in particular a [discussion of integrating SANs into ganeti](https://groups.google.com/forum/m/?_escaped_fragment_=topic/ganeti/P7JU_0YGn9s)
|
|
|
|
seems to say "just do it manually" (paraphrasing) and [this
|
|
|
|
discussion has an actual implementation](https://groups.google.com/forum/m/?_escaped_fragment_=topic/ganeti/kkXFDgvg2rY), [gnt-storage-eql](https://github.com/atta/gnt-storage-eql)
|
|
|
|
* it could be implemented as an [external storage provider](https://github.com/ganeti/ganeti/wiki/External-Storage-Providers), see
|
|
|
|
the [documentation](http://docs.ganeti.org/ganeti/2.10/html/design-shared-storage.html)
|
|
|
|
* the DSA docs are in two parts: [iscsi](https://dsa.debian.org/howto/iscsi/) and [export-iscsi](https://dsa.debian.org/howto/export-iscsi/)
|
|
|
|
* someone made a [Kubernetes provisionner](https://github.com/nmaupu/dell-provisioner) for our hardware which
|
|
|
|
could provide sample code
|
|
|
|
|
|
|
|
For now, iSCSI volumes are manually created and passed to new virtual
|
|
|
|
machines.
|
|
|
|
|
|
## Issues
|
|
## Issues
|
|
|
|
|
... | | ... | |