Storage configuration policy for the iSCSI SANs
Lately, while working on #40656 (closed) this week I noticed that the wiki procedure to provision storage on the iSCSI SANs is less than ideal: each create virtualDisk
command provisions a full RAID-5 array on 3 different physical disks (a "Disk Group" in MD3200i parlance) and creates a single LUN of the specified size on each of them. The rest of the space on the RAID-5 array is left unused.
At first I thought let's just create one big RAID-10 disk group per enclosure and provision all our LUNs on those, but it turns out this type of configuration is discouraged as explained in the Dell MD3200i performance tuning white paper, under Virtual Disk Location and Capacity:
Dell™ does not recommend using more than four virtual disks or repositories per disk group for peak performance. Additionally, where performance is critical, isolate virtual disks to separate disk groups when possible. When multiple high traffic virtual disks share a disk group, even with purely sequential usage models, the disk group I/O behavior becomes increasingly random, lowering overall performance.
The document has some additional interesting information about optimizing rotational-storage based SANs. For example, it mentions that virtual disks allocated beyond the first 1/3 of each disk group usually perform worse because of "short stroking".
So with this in mind:
First, I think we already agree that we should be using RAID-10 disk groups instead of parity-RAID. The performance and rebuild times are significantly better, at the cost of some loss in capacity. Since we have three of these SANs with two disk enclosure per SAN, I think we can totally afford this. This means that a "minimal" disk group would consume 4 physical disks and offer around 1.8TB of storage space, which seems to me pretty good.
Secondly, we should provision one dedicated disk group per host, with a minimum of one virtual disk (LUN) for the OS. The /srv
partition MAY be included in the same LUN but we should allow for /srv
to have its own, additional LUN, in the same disk group, in case we might benefit from setting different parameters on the virtual disk, for example to disable cache, configure a larger segment/stripe size, and so forth.
With such a setup we would have enough space to provision 20 new hosts with a max of 1.8TB each of dedicated storage on chi-node-02 and chi-node-03.
Obviously, some hosts have very modest storage requirements, so in these cases we should keep using the DRBD instance profile and not bother with iSCSI at all.
One last thing to keep in mind I think is that we should be careful about deploying database workloads on these HDD storage arrays. It's quite likely that even in optimal conditions such services wouldn't perform as well as the SSD+DRBD storage setup we have on gnt-fsn.