lzma needs too much Ram
This might be connected to #41159 (lzma: 292861977) #41152 (lzma: 292861977) #41147 (lzma: 292861977) where we can see a high lzma memory usage (see below lzma: 585723954).
The topic 'Memory limits for Tor process' was newly brought up in https://forum.torproject.org/t/memory-limits-for-tor-process/3961/7
OP Configuration:
Tor v0.4.8.18-1~d11.bullseye+1 running on Debian Bullseye 13.1
Nov 11 22:33:06.033 [notice] Tor 0.4.8.18 running on Linux with Libevent 2.1.12-stable, OpenSSL 1.1.1n, Zlib 1.3.1, Liblzma 5.8.1, Libzstd 1.5.7 and Glibc 2.41 as libc.OOM message:
Nov 12 19:01:16.000 [notice] We’re low on memory (cell queues total alloc: 528 buffer total alloc: 184320, tor compress total alloc: 597260954 (zlib: 395520, zstd: 11141352, lzma: 585723954), rendezvous cache total alloc: 0). Killing circuits withover-long queues. (This behavior is controlled by MaxMemInQueues.)
Nov 12 19:01:16.000 [notice] Removed 184848 bytes by killing 1 circuits; 0 circuits remain alive. Also killed 4 non-linked directory connections. Killed 0 edge connectionsIssue: The above OOM message shows 585,723,954 bytes for lzma what deems too high and might have triggered OOM.
So I did some testing on two pairs of relays on different dedicated servers. One pair of medium size relays (consensus 70000, guards) and one pair of smaller relays (consensus 500, middle). The relays own enough memory so no OoM trigger is expected.
Tor is build from source. Each pair of relays with one relay with lzma and the other lzma disabled.
The observation is the following: Small mid relays: After 8 hours the relay with lzma showed a 700MB higher memory usage. That holds over time. Medium guard relays: After 8 hours the relay with lzma showed a 200MB higher memory usage. That holds over time.
After the initial test I flipped the small relay with lzma to disabled lzma. And changed the one without lzma to enabled lzma. After 8 hours the relay with lzma showed a 700MB higher memory usage, again.
(The memory usage was taken from the RES column of 'top' https://man.freebsd.org/cgi/man.cgi?top(1) on Freebsd)
Obviously lzma seems to consume too much memory. Imho the behaviour is unexpected and can become a problem for relays with low memory. I have no idea why the medium size relays show a smaller difference.