There is an annoying issue with a Jboss EAP 7 Docker container I am using. After 3 deploys of my ear package I hit the “java.lang.OutOfMemoryError: Metaspace” issue.
According to RedHat Solution 2038983 this is a known issue caused by:
Root Cause
The Metaspace is being exhausted. Either it is undersized, or there is a leak.
There are several reasons that can lead to the above:
The Metaspace is undersized
This is usually the case. The default configuration from “jboss-eap-7.0/bin/standalone.conf” is:
JAVA_OPTS="-Xms1303m -Xmx1303m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true"
As we can see a maximum of 256m is allocated which is way to small for anything bigger than a simple demo application.
If your application contains lots of classes, JSPs, dependency libraries 256m may be not enough even if we have no memory leaks.
So I had to change my docker-compose file accordingly:
services:
jboss:
build: ./app-server
environment:
- JAVA_OPTS=-Xmx4g -Xms2g -XX:MetaspaceSize=2g -XX:MaxMetaspaceSize=2g -XX:CompressedClassSpaceSize=1g
ports:
- "8080:8080"
- "8443:8443"
- "8787:8787"
- "9990:9990"
links:
- oracle:oracle
- activemq:activemq
stdin_open: true
tty: true
...
Where:
-XX:MetaspaceSize=2g -XX:MaxMetaspaceSize=2g control the MetaSpace memory space allocation.
-XX:CompressedClassSpaceSize=1g control the space for compressed class pointers.
Hot deployment or redeployment is exposing a classloader leak
There are multiple possible causes here:
CAUSE 1: some libraries are loaded twice
Things like database drivers or logging libraries (log4j for example) can be loaded multiple times. This is usually the case when they are loaded from a configuration file of the application container but are also referred by our deployed ear file.
Jboss EAP 7 when undeploying/redeploying the ear file of our application will remove only one copy of the loaded library classes. This behavior it seems is fixed in latter WildFly releases but we are not sure is 100% solid. So as an advice make sure you do not load something several times, for example do not refer or include the database driver in your ear if is already configured in the container.
CAUSE 2: some classes cannot be removed from MetaSpace because they are locked by other resources
This is the case when ehcache is used. As mentioned by Redhat 296803
Ehcache threads (such as the cacheManagerTimer) are still running, keeping the application classloader from collection.
The CacheManager was not shutdown. If it were, such references and threads should have likely been cleaned up.
The solution is to make sure ehcache is not properly shut down when the application is undeployed.
This can be done by including in the web.xml of your web application the following hook:
net.sf.ehcache.constructs.web.ShutdownListener