Microservices, Tomcat, memory leak, migrating SpringBoot to 1.3.x

Situation:
I had to investigate an issue with one of our CI servers, which also is used for testing (don’t ask why).

This particular server is used to deploy multiple SpringBoot micro services, in a single tomcat. (again.. don’t bother).

This blog is just a log of investigation and resolution around that issue.

Our rest based SpringBoot micro services deployed within single tomcat were not responding to requests and thus timing out.

Task
Well my task was to bring them back up so testing could proceed.

Action
Like wise men say, always look at the logs, so after doing some quick system wide memory and disk checks, i decided to adhere to the wise men.

Wala! within logs i found this love note:

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread
Exception in thread java.lang.OutOfMemoryError: PermGen space

Every java programmer eventually realizes, PermGen actually means the java process has consumed all of it’s allocated memory and can’t do much anymore.

One quick and dirty solution is to restart the process, in this case Tomcat. Which i did, but after taking a backup of log file, just in case if i bump into a wise man.

Something must be causing this memory leak, and if unattended we would end up in same situation again. Thus root cause analysis was required. Further log analysis highlighted this warning message:

SEVERE: The web application [/users] created a ThreadLocal with key of type [org.jvnet.hk2.internal.Utilities$2] (value [org.jvnet.hk2.internal.Utilities$2@16d52580]) and a value of type [java.util.WeakHashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

This warning clearly suggests when SpringBoot based services are shutting down (by Tomcat Manager due to CI), the threads are not stopped properly thus they remain ‘Running’ and holding up any memory they acquired. After a few deployments and restarts this would make tomcat unresponsive for new requests.

Quick googling revealed hk2 is a Dependency Injection framework used by Jersey, which is a managed dependency within SpringBoot. Luckily this issue had already been fixed by developer community of hk2 and Jersey ver 2.21 is updated to use it. All I had to do was to update our SpringBoot version from 1.2.7 to 1.3.x.

Below are a few project specific unrelated issues that I faced when upgrading SpringBoot.

Problem:

error: cannot find symbol
symbol: method filter(List,Predicate)
location: class Iterables

Solution:
Turns out we were using guava classes that were originally packaged within Jersey.
Since Jersey project has extracted and packaged them seperately they were not in classpath anymore.
Instead of using jersey’s new package, just decided to include Google Guava directly in gradle like so:

compile ‘com.google.guava:guava:19.0’

Problem:

warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning

Solution:
JAVA_HOME for User was set to 1.8 while JAVA_HOME globally was set as 1.7

Problem:

C:\…..spring-context-4.2.7.RELEASE.jar(org/springframework/context/annotation/PropertySource.class):
warning: Cannot find annotation method ‘value()’ in type ‘Repeatable’: class file for java.lang.annotation.Repeatable not found
1 warning

Solution: (Didn’t Work for me)
Use @PropertySources to wrap @PropertySource and suggested here, here and here

This solution didn’t work for me, So went ahead and removed this annotation, all tests still passed, so this was an acceptable solution for now.

Result
Our CI server is running smoothly for few days now. Happy Days!

Leave a Reply

Your email address will not be published. Required fields are marked *

CAPTCHA
Change the CAPTCHA codeSpeak the CAPTCHA code
 

*