Add a heartbeat log message indicating progress and estimated time left
It would be really useful to know how long it still takes to read a bunch of descriptors, as this can range from minutes to hours or even days. In some cases this information would help to decide whether to continue or abort and try something different.
How about we start reading descriptors by walking files in the given directories, summing up their bytes, and using that total sum to periodically write a log message with progress and estimated time left?
Once per minute on debug level would be sufficient. And it's okay if it's not super precise and gets confused by compression or other factors. Just a rough number that gets more precise over time.
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information