Skip to content

proxy-go sometimes gets into a 100+% CPU state

proxy-go sometimes works itself into a state where it is still running and working, but using more than 100% CPU. I have had it happen locally a couple of times while testing turbotunnel stuff, and it's currently happening with proxy-go-restartless:

$ top
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
13844 snowfla+  20   0  551292 320692   8844 R 161.1 15.6 129356:18 proxy-go

Or looking at single threads:

$ top -H
  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
24628 snowfla+  20   0  551292 320692   8844 R 39.7 15.6  15219:01 proxy-go
13844 snowfla+  20   0  551292 320692   8844 R 35.4 15.6  15431:52 proxy-go
 1637 snowfla+  20   0  551292 320692   8844 R 34.8 15.6  16057:40 proxy-go
13848 snowfla+  20   0  551292 320692   8844 S 27.5 15.6  13669:02 proxy-go
13846 snowfla+  20   0  551292 320692   8844 S 22.5 15.6  17021:57 proxy-go

I caught it once and attached to the process with GDB, but didn't know what to make of it. thread apply all bt seemed to show all the threads being somewhere in the Go runtime; the thread that wasn't was not one of the threads using a lot of CPU. (Matching up the PID field from top -H with the LWP identifiers in gdb.)

I had the idea to make proxy-go emit profiling output, and then exmine the call chain that was resulting the in the most CPU using profiling tools. A patch to do that is proxy-go-profile.patch. But I haven't been able to reproduce the high CPU usage yet.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information