Move collector storage from file based to object storage
We have been having performance issues with collector for a long time [1] [2] [3].
We should move away from the file based storage solution we are using now. There are different possibilities that we could explore. In a different discussion @irl suggested to look at an object store solution [4]. We could consider min.io [5] which seems easy to use and to deploy.
Another possibility would be to look at hadoop and hdfs but maybe that's overkill.
Which ever solution we end up deciding to use, I think the current issues we have show that it is time to move away from the file storage solution, probably even before we start putting recent data in a DB for querying [6].
The usual question is now, which storage solution would be easier for us to maintain and support over time? @anarcat @lavamind please speak your mind :)
Also CC other people that might be interested in case they have other ideas: @meskio, @dgoulet
[2] #40021 (closed)
[3] tpo/tpa/team#40650 (closed)
[5] https://min.io/
[6] #40012 (closed)