Skip to content

GitLab

Closed
Created by Arturo Filasto@art

Handle scaling issues with OONI pipeline

It looks like the new machine I acquired for hosting the OONI pipeline just 6 months ago has now run out of disk space.

585G	mongodb
68G	pipeline

It appears that since we have started compressing the reports now the majority of the disk usage is sucked up by mongodb.

I think this is a good opportunity to consider the migration over to elasticsearch as was suggested by @robertkeizer: https://github.com/TheTorProject/ooni-pipeline/commit/c20ad24e96736eb8dbebda6b23fe6ccde944b066#commitcomment-10450852

Support for elastic search in ooni-pipeline has been implemented inside of https://github.com/TheTorProject/ooni-pipeline/compare/feature/elasticsearch.

Once that is merged we need to consider the possibility of starting building an elastic search cluster for all the data.

I propose we use this venue to discuss.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information