VM for metrics data processing

I need a VM for data processing of historical collector data. I have been trying to do that on metricsdb-01 but it has slowed down analysis and current ingestion.

The plan is to parse historical data from collector, from the beginning of times (tor speaking ofc). My idea is to ingest the data and export it into tabular format, probably parquet, to store on object storage, partitioned per month. Additionally some of this data would be aggregated into tables that would be copied over to metricsdb-01 and used to support graphs for the metrics website. Some data about relays would also be copied over to metricsdb-01. But the overall idea is to keep metricsdb-01 from growing exponentially and leverage object storage for historical analysis.

What is needed:

16 CPUs 128GB RAM 1TB SSD disk (2TB if possible)

Local postgresql A role/user that can write to postgresql on metricsdb-01 remotely

Yes, please!

added For Network Health Team Next lifecycle labels

assigned to @lavamind

I think we could allocate 1TB of fast NVMe replicated storage for this (ideal for database loads), however we wouldn't have much room for additional growth beyond that. Do you think it would be enough to handle "current" data before it's moved to object storage?

To have 2TB of replicated storage for a new VM we'd probably need to order extra hardware (SSDs).

added Needs Information label and removed Next label

added Next label and removed Needs Information label

marked this issue as related to #41987

mentioned in issue #41987

assigned to @lelutin and unassigned @lavamind

1TB of fast NVMe

important

@hiro I think that the only information I'm missing to move forward with the creation of the new VM is the hostname for it.

I've seen you mention metricsdb-02 in a previous post. would that hostname be OK for the final choice?

added Needs Information label and removed Next label

trying to launch VM creation on the dal cluster:

gnt-instance add -o debootstrap+bookworm -t drbd --no-wait-for-sync --net 0:ip=pool,network=gnt-dal-01 --no-ip-check --no-name-check --disk 0:size=10G --disk 1:size=1T,vg=vg_ganeti_nvme --backend-parameters memory=128g,vcpus=16 metricsdb-02.torproject.org

but that fails with the following error which I believe indicates that ganeti thinks there's not enough free disk, even though I'm requesting the volume on another VM which has enough space on all nodes:

Failure: prerequisites not met for this operation:
error type: insufficient_resources, error details:
Can't compute nodes using iallocator 'hail': Request failed: Group default (preferred): No valid allocation solutions, failure reasons: FailDisk: 6

I've tried adding --ignore-ipolicy to the command but it did nothing.

added Doing label and removed Needs Information label

the ipolicy currently has a maximum of 1Tb for disk-size. just to completely remove that limit from what is possibly blocking me, I've pushed the max disk-size of the dal cluster to 2Tb:

# before modifying:
root@dal-node-01:~# gnt-cluster info | grep -A20 "bounds specs:"
  bounds specs: 
    - min/0: 
        cpu-count: 1
        disk-count: 1
        disk-size: 512
        memory-size: 128
        nic-count: 1
        spindle-use: 1
      max/0: 
        cpu-count: 32
        disk-count: 16
        disk-size: 1048576
        memory-size: 307200
        nic-count: 8
        spindle-use: 12
  std: 
    cpu-count: 1
    disk-count: 1
    disk-size: 1024
    memory-size: 128
    nic-count: 1
# changing the limit:
root@dal-node-01:~# gnt-cluster modify --ipolicy-bounds-specs=max:cpu-count=32,disk-count=16,disk-size=2097152,memory-size=307200,nic-count=8,spindle-use=12/min:cpu-count=1,disk-count=1,disk-size=512,memory-size=128,nic-count=1,spindle-use=1

After that modification I'm still getting the same error

mentioned in commit wiki-replica@7d13341c

the VM is now online and provisioned by puppet.

ldap users are still not visible though. I'll check tomorrow morning to see if I missed something with regards to this or if it'll somehow sync up users overnight.

marked this issue as related to #42013 (closed)

The host is created and I've taken care of most initial alerts for it. closing this issue

closed