JVM OutOfMemory — Meta Space

I think as a programmer at least one time in your carrier you have come across the OutofMemory issue. In this tutorial, I will explain to you my experience with Heap dump analysis and what you have to check while doing heap dump analysis.

In this tutorial, I will focus on out-of-memory Metaspace. If you are not aware of what Metaspace is then I would suggest you read this.

There are 4 scenarios that I have come across which could cause Metaspace error

1. Classloader leak — Thread Local variables
2. Duplicate objects
3. OSGI Class loader leak
4. Metaspace size is less

Lets first understand how classloader leak occurs in the below code. Threadlocal variables is not removed after creating it. In a web application, instead of creating a new thread every time, it is followed to use the concept of Thread pool.

Main.java
--------------
public class Main {
public static void main(String...args) throws Exception {
loadClass();
while (true) {
System.gc();
Thread.sleep(1000);
}
}
private static void loadClass() throws Exception {
URL url = Main.class.getProtectionDomain().getCodeSource().getLocation();
MyCustomClassLoader cl = new MyCustomClassLoader(url);
Class<?> clazz = cl.loadClass("com.test.Foo");
clazz.newInstance();
cl = null;
}
}
class MyCustomClassLoader extends URLClassLoader {
public MyCustomClassLoader(URL... urls) {
super(urls, null);
}
@Override
protected void finalize() {
System.out.println("*** CustomClassLoader finalized!");
}
}
Foo.java
------------------------------
public class Foo {
private static final ThreadLocal<Bar> tl = new ThreadLocal<Bar>();
public Foo() {
Bar bar = new Bar();
tl.set(bar);
System.out.println("Test ClassLoader: " + this.getClass().getClassLoader());
}
@Override
protected void finalize() {
System.out.println( this + " finalized!");
}
}
Bar.java
----------------------------------
public class Bar {
public Bar() {
System.out.println(this + " created");
System.out.println("Bar ClassLoader: " + this.getClass().getClassLoader());
}
@Override
public void finalize() {
System.out.println(this + " finalized");
}
}

After running this code output shows :

com.test.Bar@71bc1ae4 created
Bar ClassLoader: com.test.MyCustomClassLoader@71dac704
Test ClassLoader: com.test.MyCustomClassLoader@71dac704
com.test.Foo@650de12 finalized!

Now if we change the ThreadLocal variable to String in Foo.java code as

public class Foo {
private static final ThreadLocal<String> tl = new ThreadLocal<String>();
public Foo() {
Bar bar = new Bar();
tl.set("Hello");
System.out.println("Test ClassLoader: " + this.getClass().getClassLoader());
}
@Override
protected void finalize() {
System.out.println( this + " finalized!");
}
}

The output shows all the classes are GCed

com.test.Bar@71bc1ae4 created
Bar ClassLoader: com.test.MyCustomClassLoader@71dac704
Test ClassLoader: com.test.MyCustomClassLoader@71dac704
com.test.Bar@71bc1ae4 finalized
com.test.Foo@5d783be3 finalized!
*** CustomClassLoader finalized!

The reason behind is that Bar.class is loaded by MyCustomClassLoaders which means it has a reference.

current thread -> Foo.tl -> Bar.class -> MyCustomClassLoader

But when we change to String then it uses Systemclassloader and has no reference to MyCustomClassLoader class

current thread -> Foo.tl -> String.class -> System Class Loader

Now if we modify the Main class like this:

public static void main(String...args) throws Exception {
//loadClass();
new Thread(new FutureTask(() -> { loadClass(); return null; })).start();
while (true) {
System.gc();
Thread.sleep(1000);
}
}

All classes including classloader will be GCed, this is because we are asking the main thread to loadClass to a different Thread rather than the current thread (main).

Now let's look at the Threadlocal variable as shown in the Heap dump:

In the above screenshots, it shows that the Bundle class loader is holding the reference of New Relic thread factory class which is using thread-local internally. And it is the same for most of the java.lang.ThreadLocal variable.

Current thread → New Relic Default Thread factory → OSGI Bundle class loader

In many enterprise applications, we use APM (Application performance management) agent which uses Java native library. In my case, we were using New Relic. As you can see in the above example, New Relic agent was holding the references and was not able to release the strong reference and hence caused an Out-of-Memory error in Metaspace.