Circuit cell queue can fill up memory
A relay operator just reported this on 0.3.3.2-alpha:
https://lists.torproject.org/pipermail/tor-relays/2018-February/014496.html
In a nutshell, the OOM fired up with these logs:
Feb 12 18:54:55 tornode2 Tor[6362]: We're low on memory (cell queues total alloc: 1602579792 buffer total alloc: 1388544, tor compress total alloc: 1586784 rendezvous cache total alloc: 489909). Killing circuits withover-long queues. (This behavior is controlled by MaxMemInQueues.)
Feb 12 18:54:56 tornode2 Tor[6362]: Removed 1599323088 bytes by killing 1 circuits; 39546 circuits remain alive. Also killed 0 non-linked directory connections.
Notice the ~1GB of cells for one single circuit? Somehow, there is an issue in tor that makes it possible to fill up the circuit cell queue while the scheduler is just not emptying that queue.
This really looks like the Sniper Attack: http://www.robgjansen.com/publications/sniper-ndss2014.pdf