Skip to content

Speedup ingestion of data into the DB

I have been thinking of ways to speedup the ingestion process at the DB. I have been thinking of a way to make the status table smaller while also being a bit smarter in how and where we aggregate data.

This issue aims to address the following:

  • Make the router status table smaller to avoid passing a large data object in memory and saving a large table (large in number of columns)
  • Aggregate latest and first descriptors in materialized views to avoid sorting each time
  • Make sure the family digest use sha1 for easy of use across metrics ecosystem
Edited by Hiro