Analyse cluster expansion options for gnt-dal ganeti cluster
Storage is becoming a little tight on gnt-dal and we should look at what our options are to expand the available storage on that cluster, since that seems to be the main bottleneck right now.
We only have 2x 2.5" empty SSD drive bays in each of our three gnt-dal nodes. We can't add additional NVMe drives in these machines.
Designs
Activity
added lifecycle label
marked this issue as related to #41969 (closed)
assigned to @lavamind
added Backlog label
The discussion got started here in VM for metrics data processing (#41969 - closed) so we spun out a different ticket (this one) so as to not block that work and get the two mixed up.
mentioned in issue #41969 (closed)
changed milestone to %(next) cluster scaling
so to quote part of the discussion in that issue, so we can followup:
@lavamind wrote:
@anarcat wrote:
- a "sweet spot": say just adding more drives of the same size
We could add 2x 2TB SSD drives to each of the three nodes, that would give us 2TB extra useable storage space on each node if we use them to grow the existing RAID10 array.
How much would that be?
- a "pipe dream": new SSD (or NVMe?!) drives, as big as possible
First, we only have 2x 2.5" empty SSD drive bays in each of our three gnt-dal nodes. We can't add additional NVMe drives in these machines.
Second, Samsung makes 7.68TB versions of the current drives that we have, they're very expensive coming in at around $1,000 each drive. In addition, we'd probably need to investigate mdadm more closely to make sure we could actually grow our current RAID10 array with this new geometry. Otherwise, we could always just integrate them as a separate RAID1 array but it would make the setup somewhat less flexible.
Right. So that would be what, 6k$USD for the upgrade on the entire cluster? That's not completely out of this world, to be honest.
Please do investigate how it would integrate in the array.
There seems to be bigger drives, but prices become really prohibitive at 16TB (4k$/drive), so let's leave that aside for now.
- a "weird setup": add really huge HDD drives for archival (we have 16 and 24TB drives now!)
As far as I know those drives only exist in 3.5" spinning rust form factor, and we don't have any HDD drive bays that could house them in the gnt-dal machines. Also, I would seriously recommend against putting any sort of database workload on HDDs.
Of course, not for databases, but we could use it to archive things in a "cold" MinIO tier, for example, freeing up some space for other things.
I did a cursory exam of pcpartspicker on this, and it seems you are right that 2.5" HDDs are not available in those sizes: the biggest I found are 4TB, and their prices are not much better than SSDs anyways. e.g. there's the Seagate barracuda 4TB 5400rpm at 280USD(0.070USD(0.070 /GB):
but then right next to that there's a WD Blue 4TB SSD at 350USD(0.087USD(0.087 /GB):
Clearly not worth the rust.
-
Please do investigate how it would integrate in the array.
I'm growing an array at home so I looked into this.
It doesn't look like we can grow RAID-10 arrays with differently-sized disks. (interestingly, it's really hard to find authoritative data on this: there used to be a mediawiki for this which was already quite poor, but it's now archived and the thing it points to is an abysmal, md0-kernel-interface-only documentation that's utterly useless for our needs here.)
ANYWAYS.
It seems like RAID-1/4/5/6 arrays can be grown, according to those docs, but according to that, RAID-10 arrays are "very difficult to reshape and currently any attempt to do so will be rejected. It could be done if someone cares to code it."
So we can't resize the existing array with larger drives (but we could add similarly-sized drives to the array. I think. I'm not sure.)
In any case, what we can almost certainly do is add those as a new RAID-1 array, and pile that onto the VG, as a new PV. I think that would work. Essentially, that would be RAID-1+0, with LVM doing the RAID-0 layer.
(Perhaps, this is how we should have done this in the first place?)
So the question is whether we go with the current drive size or use bigger drives. Could you estimate the prices on both? Include spares, please. I think so far I have:
- 8x7.68TB drives (two spares, 23.04TB extra capacity, 8k$USD cost? 1k$/unit?)
- 6x2TB drives (no extra spares, 6TB extra capacity, 1.7k$USD cost, 285$/unit)
surely there's some math wrong there, as the 23TB option looks much more attractive there, even when taking the spares into account. And indeed, the Intel S4510 7.68TB drives are more like 1500USD on Newegg, which would bring the total to 12k$ for 23TB.
Edited by anarcat -
It seems like RAID-1/4/5/6 arrays can be grown, according to those docs, but according to that, RAID-10 arrays are "very difficult to reshape and currently any attempt to do so will be rejected. It could be done if someone cares to code it."
So we can't resize the existing array with larger drives (but we could add similarly-sized drives to the array. I think. I'm not sure.)
The documentation you're referring to is ancient. I don't think we should discount our ability to do this based only on that. I do think we can grow a RAID10 volume using larger disks, and this is based on experiments I did a while ago with loop devices, although we should verify this still holds with recent kernels of course.
In any case, I would advise against adding new storage capacity to the cluster by creating a new, separate volume. as opposed to extending the existing one. An extra storage boundary is going to hurt by reducing our flexibility and is likely to create situations in which we'll need to move virtual disks across volumes, which is a time consuming operation.
-
Jérôme Charaoui commented on a discussion: #41987 (comment 3174723)
It seems like RAID-1/4/5/6 arrays can be grown, according to those docs, but according to that, RAID-10 arrays are "very difficult to reshape and currently any attempt to do so will be rejected. It could be done if someone cares to code it."
So we can't resize the existing array with larger drives (but we could add similarly-sized drives to the array. I think. I'm not sure.)
The documentation you're referring to is ancient. I don't think we should discount our ability to do this based only on that. I do think we can grow a RAID10 volume using larger disks, and this is based on experiments I did a while ago with loop devices, although we should verify this still holds with recent kernels of course.
I couldn't find any newer documentation on this, I have done quite a bit of research!
In any case, I would advise against adding new storage capacity to the cluster by creating a new, separate volume. as opposed to extending the existing one. An extra storage boundary is going to hurt by reducing our flexibility and is likely to create situations in which we'll need to move virtual disks across volumes, which is a time consuming operation.
What do you mean by "volume" here? Volume Group (VG)? That's not my intention: my intention is to create a new RAID-1 Physical Volume (PV) that would get added to the the existing VG.
I don't think such a boundary (a new PV) will hurt our flexibility, as we won't need to move "virtual disks" (logical volumes, LV, i assume?) across PVs: it will just get allocated as is. and unless we need to rebuild an entire array, we'll never need to move LVs between PVs.
...
On 2025-03-17 13:56:18, Jérôme Charaoui (@lavamind) wrote:
marked this issue as related to #42077
marked this issue as related to #41449 (closed)
marked this issue as related to #41450
marked this issue as related to #41454
mentioned in incident #42077
-
this has become more urgent as the minio VM filled up, see #42077.
mentioned in issue #41454
-
So the question is whether we go with the current drive size or use bigger drives. Could you estimate the prices on both?
If anyone wants to pick this up, please go ahead. I don't expect to have time this week to look into it.
I'm going to pick this up because I think we really need to prioritize buying and shipping the hardware ASAP, but i will need to pick your brain on this:
Could you estimate the prices on both? Include spares, please. I think so far I have:
- 8x7.68TB drives (two spares, 23.04TB extra capacity, 8k$USD cost? 1k$/unit?)
- 6x2TB drives (no extra spares, 6TB extra capacity, 1.7k$USD cost, 285$/unit)
surely there's some math wrong there, as the 23TB option looks much more attractive there, even when taking the spares into account. And indeed, the Intel S4510 7.68TB drives are more like 1500USD on Newegg, which would bring the total to 12k$ for 23TB.
-
- shop for Samsung PM893 8TB drives or cheaper, but with similar write endurance, maybe 2x?
pcpartspicker says the samsung is pretty good, pricewise https://pcpartpicker.com/products/internal-hard-drive/#A=6000000000000,24000000000000&f=3&t=0&sort=ppgb&page=1
so it seems that's what we should go with. i'm thinking of going with two drives to reduce immediate costs and risks.
- test a RAID-10 growth with loop devices
- check with interpro to see if we can add drives
next steps
-
@lavamind according to interpro:
Please check the attached original invoice for those 3 servers. Each server already has 10 SSDs, with no more space for 2.5" hot-swap drives. You can only add 2 x M.2 SSDs on board.
Seems like we had a massive misunderstanding there, again.
One possibility would be to indeed add 2xM.2 drives and replace the Micro 2.5" 500GB drives with 8TB drives, but that's slightly more expensive, and a more involved operation as we need to do a two-stage migration. Ugh.
Edited by anarcat
- Resolved by Jérôme Charaoui
- test a RAID-10 growth with loop devices
so i made 4 1GB devices:
for dev in a b c d ; do truncate -s 1g $dev losetup -f $dev doneand two 10GB ones:
for dev in f g ; do truncate -s 10g $dev losetup -f $dev donei made a RAID-10 array with the former:
root@angela:/home/anarcat/raid-10-test# losetup -l NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 0 /home/anarcat/raid-10-test/b 0 512 /dev/loop4 0 0 0 0 /home/anarcat/raid-10-test/f 0 512 /dev/loop2 0 0 0 0 /home/anarcat/raid-10-test/c 0 512 /dev/loop0 0 0 0 0 /home/anarcat/raid-10-test/a 0 512 /dev/loop5 0 0 0 0 /home/anarcat/raid-10-test/g 0 512 /dev/loop3 0 0 0 0 /home/anarcat/raid-10-test/d 0 512 root@angela:/home/anarcat/raid-10-test# mdadm --create --verbose --level=10 --raid-devices=4 /dev/md0 /dev/loop[0-3] To optimalize recovery speed, it is recommended to enable write-indent bitmap, do you want to enable it now? [y/N]? y mdadm: layout defaults to n2 mdadm: layout defaults to n2 mdadm: chunk size defaults to 512K mdadm: size set to 1046528K mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started. root@angela:/home/anarcat/raid-10-test# cat /proc/mdstat Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid10 loop3[3] loop2[2] loop1[1] loop0[0] 2093056 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU] [=====>...............] resync = 29.2% (613760/2093056) finish=0.1min speed=204586K/sec bitmap: 1/1 pages [4KB], 65536KB chunk unused devices: <none> root@angela:/home/anarcat/raid-10-test# cat /proc/mdstat Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid10 loop3[3] loop2[2] loop1[1] loop0[0] 2093056 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU] bitmap: 0/1 pages [0KB], 65536KB chunk unused devices: <none> root@angela:/home/anarcat/raid-10-test# mkfs -t ext4 /dev/md0 mke2fs 1.47.2 (1-Jan-2025) Discarding device blocks: done Creating filesystem with 523264 4k blocks and 130816 inodes Filesystem UUID: 8e79a1d0-7e3d-464e-80f4-9f345e8d32fc Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done root@angela:/home/anarcat/raid-10-test# mount /dev/md0 /mnt root@angela:/home/anarcat/raid-10-test# df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/md0 2.0G 532K 1.9G 1% /mnt2GB, expected size.
then i tried to grow it by adding two 10GB drives. I expect this should grow the array by at least 10GB.
root@angela:/home/anarcat/raid-10-test# mdadm --grow /dev/md0 --raid-devices=6 -a /dev/loop[45] mdadm: added /dev/loop4 mdadm: added /dev/loop5 root@angela:/home/anarcat/raid-10-test# cat /proc/mdstat Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid10 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0] 2093056 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU] [=======>.............] reshape = 39.6% (829440/2093056) finish=0.0min speed=276480K/sec bitmap: 0/1 pages [0KB], 65536KB chunk unused devices: <none>after the reshape:
root@angela:/home/anarcat/raid-10-test# cat /proc/mdstat Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid10 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0] 3139584 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU] bitmap: 0/1 pages [0KB], 65536KB chunk unused devices: <none>resizing the filesystem:
root@angela:/home/anarcat/raid-10-test# resize2fs /dev/md0 resize2fs 1.47.2 (1-Jan-2025) Filesystem at /dev/md0 is mounted on /mnt; on-line resizing required old_desc_blocks = 1, new_desc_blocks = 1 The filesystem on /dev/md0 is now 784896 (4k) blocks long.only gives us an extra 1GB:
root@angela:/home/anarcat/raid-10-test# df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/md0 3.0G 532K 2.8G 1% /mntconclusion: we can't stick the new drives in a RAID-10 array and expect them to be fully used.
Edited by anarcat - Last reply by Jérôme Charaoui
added Roadmap::Future label and removed Doing label
unassigned @anarcat
-
note that i've analyzed expansion possibilities in https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-84-minio-backups-and-scaling
marked this issue as related to #41415