Fixing spiky CPU issues and unresponsiveness with Tomcat

2015-09-16

Once upon a time, our server was lagging after light usage of the applications, and the CPU usage would start spiking and rapidly cycle from many CPU cores (e.g. 2000% CPU usage) back to 0% CPU for no apparent reason.

After some debugging (https://cmdcolin.github.io/posts/2015-10-15) I found it was related to memory issues and garbage collection, but it was confusing because it wasn't strictly showing up as GC usage in JVisualVm (the GC usage, blue spikes on the left in fig 1, are small, but the orange spikes are large, even though the memory issues are the problem)

Here is what it looked like during spiking (obviously, pushing the memory limits here, a linked in article suggests having 6GB of "newgen" memory, so on top of the old gen, tomcat needs a bunch more for the newgen to make things happy.

Here is what it looks like when it is not spiking

See this follow up post for a little more info https://cmdcolin.github.io/posts/2015-10-15