JVM Crashes

Case Study - I gonna write a true story that happened on a company which I work for.

I got a task: Crash Hunting

Me: Who Crash?

QA: The system.

Me: What does it mean?

QA: The application stop working.

QA: before several changes, there wasn't any crash. after the changes, the system crash.

Me: OK, let me check.

Other developer: we look at the log files, we didn't find anything.

Me: The JVM crashed.


With that say, we need to make a research to understand 3 W's: Why there was a crash; Where the crash occurred; What causing the crash.

My first thought was: there are several reasons Why the JVM crashed, so we need to classify it.
In order to classify the reason we need an information. A log files is a pretty good place to get an information. 

But, as the other developer said: "we look at the log files, we didn't find anything".

I was wondering, what are the meaning of those log files? this is the application log files (The business logic log files).

If the JVM was crashed => application has crashed => there is no active process that can write to those log files. 

My next thought was, we need to look for the JVM log file in order to classify the crash, over there we will see a snapshot of the state of the machine and the JVM process at the time of the crash.

As stated at Oracle troubleshooting guide,

When a fatal error occurs, an error log is created with information and the state obtained at the time of the fatal error.

In order for this file to be created we need to configure when the JVM crash log will be created. we can configure it as part of the VM Options: 

-XX:ErrorFile=/var/log/java/java_error%p.log

We add it, reproduce the crash and the file created.

When we opened the file we noticed the following:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (sharedRuntime.cpp:834), pid=75034, tid=140148337530624
#  fatal error: exception happened outside interpreter, nmethods and vtable stubs at pc 0x00007f791eb2a80f
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f7728029800):  JavaThread "nioEventLoopGroup-2-10" [_thread_in_Java, id=77330, stack(0x00007f76d3cfd000,0x00007f76d3dfe000)]

Stack: [0x00007f76d3cfd000,0x00007f76d3dfe000],  sp=0x00007f76d3dfa4d0,  free space=1013k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xaac99a]  VMError::report_and_die()+0x2ba
V  [libjvm.so+0x4f2de9]  report_fatal(char const*, int, char const*)+0x59
V  [libjvm.so+0x9ab5ba]  SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned char*, SharedRuntime::ImplicitExceptionKind)+0x33a
V  [libjvm.so+0x914f1a]  JVM_handle_linux_signal+0x48a
V  [libjvm.so+0x90b493]  signalHandler(int, siginfo*, void*)+0x43
C  [libpthread.so.0+0xf5e0]
J 15905 C2 com.sun.crypto.provider.GCTR.update([BII[BI)I (158 bytes) @ 0x00007f79216aefea [0x00007f79216aeca0+0x34a]

At this file we can see the cpp stack trace which cause the JVM to throw an exception and crash.

It’s a bug that was reported on the JDK – they solve it on the next versions. The suggestion is to upgrade to the latest 8 version. 

To find a bug in Java it's like winning the lottery, it can happen but with very low probability