Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-16T01:12:42Zhttps://gitlab.torproject.org/legacy/trac/-/issues/33966tba: Extension error: [Exception... "Component returned failure code: 0x80004...2020-06-16T01:12:42Ztraumschuletba: Extension error: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIDOMWindowUtils.removeSheetUsingURIString]"Experienced another OOM and ANR in combination with this trace which slipped through so far: shows up repeatedly (5 times each) and happens daily according to my logs, also unrelated to ANR:
```
04-22 01:11:58.547 3810 3840 I Gecko :...Experienced another OOM and ANR in combination with this trace which slipped through so far: shows up repeatedly (5 times each) and happens daily according to my logs, also unrelated to ANR:
```
04-22 01:11:58.547 3810 3840 I Gecko : Extension error: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIDOMWindowUtils.removeSheetUsingURIString]" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: re[/gre/modules/ExtensionCommon.jsm](/gre/modules/ExtensionCommon.jsm) :: runSafeSyncWithoutClone :: line 75" data: no] undefined 75
04-22 01:11:58.547 3810 3840 I Gecko : [[Exception stack
04-22 01:11:58.547 3810 3840 I Gecko : runSafeSyncWithoutClone@re[/gre/modules/ExtensionCommon.jsm:75:12](/gre/modules/ExtensionCommon.jsm:75:12)
04-22 01:11:58.547 3810 3840 I Gecko : cleanup@re[/gre/modules/ExtensionContent.jsm:403:11](/gre/modules/ExtensionContent.jsm:403:11)
04-22 01:11:58.547 3810 3840 I Gecko : close@re[/gre/modules/ExtensionContent.jsm:913:14](/gre/modules/ExtensionContent.jsm:913:14)
04-22 01:11:58.547 3810 3840 I Gecko : inner-window-destroyed@re[/gre/modules/ExtensionContent.jsm:998:19](/gre/modules/ExtensionContent.jsm:998:19)
04-22 01:11:58.547 3810 3840 I Gecko : observe@re[/gre/modules/ExtensionContent.jsm:1016:27](/gre/modules/ExtensionContent.jsm:1016:27)
04-22 01:11:58.547 3810 3840 I Gecko : Current stack
04-22 01:11:58.547 3810 3840 I Gecko : runSafeSyncWithoutClone@re[/gre/modules/ExtensionCommon.jsm:81:9](/gre/modules/ExtensionCommon.jsm:81:9)
04-22 01:11:58.547 3810 3840 I Gecko : cleanup@re[/gre/modules/ExtensionContent.jsm:403:11](/gre/modules/ExtensionContent.jsm:403:11)
04-22 01:11:58.547 3810 3840 I Gecko : close@re[/gre/modules/ExtensionContent.jsm:913:14](/gre/modules/ExtensionContent.jsm:913:14)
04-22 01:11:58.547 3810 3840 I Gecko : inner-window-destroyed@re[/gre/modules/ExtensionContent.jsm:998:19](/gre/modules/ExtensionContent.jsm:998:19)
04-22 01:11:58.547 3810 3840 I Gecko : observe@re[/gre/modules/ExtensionContent.jsm:1016:27](/gre/modules/ExtensionContent.jsm:1016:27)
04-22 01:11:58.547 3810 3840 I Gecko : ]]
```
Fixed in Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1550470
I have no additional addons installed.https://gitlab.torproject.org/legacy/trac/-/issues/27813Tor 0.3.4.8 is leaking memory2020-06-13T15:31:59ZTracTor 0.3.4.8 is leaking memoryI upgraded to Tor 0.3.4.8 yesterday and today the process has been killed by the oom killer.
Out of memory: Kill process 1369 (tor) score 338 or sacrifice child.
It seems Tor is leaking memory until there is nothing left.
**Trac**:
...I upgraded to Tor 0.3.4.8 yesterday and today the process has been killed by the oom killer.
Out of memory: Kill process 1369 (tor) score 338 or sacrifice child.
It seems Tor is leaking memory until there is nothing left.
**Trac**:
**Username**: anongTor: 0.3.5.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/27319remove buf_get_oldest_chunk_timestamp()2020-06-13T15:30:12Zcypherpunksremove buf_get_oldest_chunk_timestamp()buffers.c calls `monotime_coarse_get_stamp()` to store a timestamp per chunk, marking when it was allocated. This is weird layering for a simple bytes container.buffers.c calls `monotime_coarse_get_stamp()` to store a timestamp per chunk, marking when it was allocated. This is weird layering for a simple bytes container.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/25372relay: Allocation for compression goes very high2020-06-13T15:22:27ZDavid Gouletdgoulet@torproject.orgrelay: Allocation for compression goes very highMy relay just OOMed some circuits with filled up queue (#25226) but then a useful log was printed showing that the compress total allocation is huge.
```
Feb 27 20:02:55.718 [notice] We're low on memory (cell queues total alloc: 2322798...My relay just OOMed some circuits with filled up queue (#25226) but then a useful log was printed showing that the compress total allocation is huge.
```
Feb 27 20:02:55.718 [notice] We're low on memory (cell queues total alloc: 232279872 buffer total alloc: 1937408, tor compress total alloc: 878586075 rendezvous cache total alloc: 4684497). Killing circuits withover-long queues. (This behavior is controlled by MaxMemInQueues.)
```
That `878586075 = ~838MB`. My relay is hovering around 1.4GB of RAM right now which means ~60% of the RAM used is in the compression subsystem.
I'm not sure where it all comes, the relay is serving directory data but I have my doubt that *compressed*, it comes down to 800+ MB...
Datapoint:
```
$ du -sh diff-cache/
131M diff-cache/
```Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/24667OOM needs to consider the DESTROY queued cells2020-06-13T15:19:13ZDavid Gouletdgoulet@torproject.orgOOM needs to consider the DESTROY queued cellsOur OOM is only looking a the circuit queue cells and HS descriptors to free up memory.
We need to teach it to cleanup DESTROY cells in case cleaning up the circuits is not enough.
This isn't that trivial because while cleaning up circ...Our OOM is only looking a the circuit queue cells and HS descriptors to free up memory.
We need to teach it to cleanup DESTROY cells in case cleaning up the circuits is not enough.
This isn't that trivial because while cleaning up circuits in the OOM handler, we will also send DESTROY cells for those thus allocating memory. But also not sending those will affects other relays hanging on dead circuits.
All in all, this is an interesting challenge but there might be something smart to do even if not perfect.
The idea here is to avoid an attack that takes advantage of a bug in tor that can fill up the DESTROY cell queue and our OOM just can't do anything about it.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/18642Teach the OOM handler about the DNS cache2020-06-13T14:55:40ZNick MathewsonTeach the OOM handler about the DNS cacheTor: 0.3.5.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/18641Teach the OOM handler about uploaded descriptors on a dirauth.2020-06-13T14:55:39ZNick MathewsonTeach the OOM handler about uploaded descriptors on a dirauth.The OOM handler should know to do something with the descriptors that a dirauth has received via upload.The OOM handler should know to do something with the descriptors that a dirauth has received via upload.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/18637Have OOM handler look at all memory consumption, not just some2020-06-13T14:55:35ZNick MathewsonHave OOM handler look at all memory consumption, not just someJust because our OOM handler doesn't know how to free every kind of memory we allocate, doesn't mean we shouldn't teach it to consider our total allocation when deciding that we're low on memory.
For platforms where malloc() can return ...Just because our OOM handler doesn't know how to free every kind of memory we allocate, doesn't mean we shouldn't teach it to consider our total allocation when deciding that we're low on memory.
For platforms where malloc() can return NULL, we could have it look at that too.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/13856Make sure #9262 didn't introduce new OOM opportunities2020-06-13T14:40:46ZNick MathewsonMake sure #9262 didn't introduce new OOM opportunitiesIIUC, the #9262 changes don't introduce any big new queues or structures, so the OOM handler doesn't need any more changes to think about them. But I just merged #9262, and now I'm too tired to look for data structure changes. So, this...IIUC, the #9262 changes don't introduce any big new queues or structures, so the OOM handler doesn't need any more changes to think about them. But I just merged #9262, and now I'm too tired to look for data structure changes. So, this ticket should remind us/me to confirm that we don't need more OOM handler hacking for that, and then put it to bed.Tor: unspecifiedAndrea ShepardAndrea Shepardhttps://gitlab.torproject.org/legacy/trac/-/issues/13806Count HSDir storage towards MaxMemInQueues2020-06-13T14:40:29ZNick MathewsonCount HSDir storage towards MaxMemInQueuesSee branch "bug13806" in my public repository.See branch "bug13806" in my public repository.Tor: 0.2.6.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/11792Consider directory connections zlib buffers when handling OOM2020-06-13T14:36:06ZNick MathewsonConsider directory connections zlib buffers when handling OOMOur current OOM handler code looks at edge connections and circuit queues when deciding whether to declare OOM and when finding circuits to kill. It should also consider killing directory connections, and consider the total allocation f...Our current OOM handler code looks at edge connections and circuit queues when deciding whether to declare OOM and when finding circuits to kill. It should also consider killing directory connections, and consider the total allocation for zlib compression objects.
See also #11791Tor: 0.2.6.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/11396Detect maximum memory at runtime to allow lower default than 8GB2020-06-13T14:35:16ZNick MathewsonDetect maximum memory at runtime to allow lower default than 8GBThe default value for MaxMemInQueues is 8 GB, which means that for many users, there simply won't be any support for OOM prevention.
I think we can do better than that by detecting the physical memory and picking a default MaxMemInQueue...The default value for MaxMemInQueues is 8 GB, which means that for many users, there simply won't be any support for OOM prevention.
I think we can do better than that by detecting the physical memory and picking a default MaxMemInQueues based on a percentage of that. Of course, the user should still be able to override that value.Tor: 0.2.5.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/9093Better, fairer circuit OOM handling2020-06-13T14:34:06ZNick MathewsonBetter, fairer circuit OOM handlingWith our second attempt at a #9063 fix, merged into 0.2.4.14-alpha, I introduced an OOM handler that kills circuits if we're too low on memory. But the obvious algorithm I used ("Algorithm 1" as described on #9072) is not as good as it ...With our second attempt at a #9063 fix, merged into 0.2.4.14-alpha, I introduced an OOM handler that kills circuits if we're too low on memory. But the obvious algorithm I used ("Algorithm 1" as described on #9072) is not as good as it could be or should be.
Currently, I favor looking for a good estimator of "How long will this circuit take to drain all of its currently queued cells?" and using that for deciding which circuits to kill when low on RAM. There are other suggestions on #9072 too. And to those a suggestion from IRC that we look at the age of the oldest cell on the circuit.
If we find something that uses data that Tor currently captures (for example, with the ewma machinery), it will be much easier to deploy.
For whatever we pick, we need to analyze its security implications and look for ways to game it to provoke relays to do something stupid.
It would be good to have a default OOM threshold computed in some sane (though probably nonportable) way based on available RAM, with a reasonable cap. That might be ridiculously hard though.
It would also be good to see about taking more potentially-big things into account, not just circuit queues.Tor: 0.2.4.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/10169Extend OOM handler to cover channels/connection buffers2020-06-13T14:32:59ZNick MathewsonExtend OOM handler to cover channels/connection buffersRight now, the OOM handler looks at circuits but not streams. We should fix that. Any ideas for a good algorithm?Right now, the OOM handler looks at circuits but not streams. We should fix that. Any ideas for a good algorithm?Tor: 0.2.5.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/10116Avoid memory allocation in OOM handler2020-06-13T14:32:54ZNick MathewsonAvoid memory allocation in OOM handlerRight now, the OOM handler allocates an array of size(n_circs * sizeof(void*)) to figure out which circuits to kill. That's probably not a good idea. Instead, it would be better to select the select circuits from the circuitlist.
Opti...Right now, the OOM handler allocates an array of size(n_circs * sizeof(void*)) to figure out which circuits to kill. That's probably not a good idea. Instead, it would be better to select the select circuits from the circuitlist.
Options here include:
* Turn global_circuitlist into an array (smartlist) structure.
* Retain the linked-list structure, but do one of these:
* Use a merge sort to sort the circuitlist in place.
* Do an O(N^2) algorithm to walk the circuitlist to find the worst circuit, delete that, and then do it over and over until we have killed enough circuits.
* Do an O(Nk) algorithm to walk the circuitlist to find the k worst circuits; kill them; repeat until we have killed enough circuits. This still turns out to O(N^2)Tor: 0.2.6.x-final