Skip to content

Our zlib deflate objects require lots of memory per directory connection

Compressing descriptors on the fly requires a lot of memory with zlib's default settings. According to zconf.h:

/* The memory requirements for deflate are (in bytes):
            (1 << (windowBits+2)) +  (1 << (memLevel+9))
 that is: 128K for windowBits=15  +  128K for memLevel = 8  (default values)
 plus a few kilobytes for small objects.

We are using windowbits == 15 and memLevel == 8. This might explain in part why busy directories are so memory-hungry. In addition to our investigations for legacy/trac#11787 (moved), we should see whether we can safely reduce this to 32K or so with windowBits == 12 and memlevel == 5. It appears in preliminary investigation that this compression level would not hurt compression very much (0-5% for microdescs, 3-7% for server descriptors).

We could also choose our window size and memlevel based on current memory availability.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information