title: DTrace missing Java frames with ustack(). Running on Joyent SmartOS infrastructure container
body: <p>I cannot get any Java stack with dtrace in a Joyent SmartOS instance. </p>
  
  <p>I tried the <code>java:15.1.1</code> image and a plain SmartOS 'base64' image, where I installed openjdk 8.</p>
  
  <p>I most basic example:
  cat Loop.java </p>
  
  <pre><code>[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 /demo]# cat Loop.java 
  class Loop {
  
      public static void main(String[] args) throws InterruptedException {
          while (true) {
                  System.out.println("Sleepy");
                  Thread.sleep(2000);
          }
      }
  }
  [root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 /demo]# javac Loop.java 
  [root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 /demo]# java Loop
  </code></pre>
  
  <p>I added the <code>libdtrace_forceload.so</code> as <a href="https://stackoverflow.com/questions/36166805/how-do-i-use-the-hotspot-dtrace-probes-on-smartos/36167514#36167514">recommended</a>.</p>
  
  <pre><code>export LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so
  </code></pre>
  
  <p>It is a 64 bit JVM:</p>
  
  <pre><code>[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 /demo]# java -version
  openjdk version "1.7.0-internal"
  OpenJDK Runtime Environment (build 1.7.0-internal-pkgsrc_2015_05_29_19_05-b00)
  OpenJDK 64-Bit Server VM (build 24.76-b04, mixed mode)
  </code></pre>
  
  <p>When I run dtrace, and use jstack, I get the C-stacks. However, the JAVA frames are the raw addresses, quite useless:</p>
  
  <pre><code>[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 ~]# pgrep -fn "java Loop"
  32597
  [root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 ~]# dtrace -n 'syscall:::entry/pid == 32597/ { @num[ustack(20)] = count(); }'
  dtrace: description 'syscall:::entry' matched 237 probes
  ^C
  
  
                libc.so.1`__write+0xa
                libjvm.so`_ZN2os5writeEiPKvj+0x128
                libjvm.so`JVM_Write+0x34
                libjava.so`writeBytes+0x1b5
                libjava.so`Java_java_io_FileOutputStream_writeBytes+0x1f
                0xffffbf7ffa612d98
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa606058
                0xffffbf7ffa6004e7
                libjvm.so`_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread+0x31d
  
  *snip*
  </code></pre>
  
  <p>I do see hotspot probes are available:</p>
  
  <pre><code>[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 ~]# dtrace -l | grep hotspot | more
   6103 hotspot32597         libjvm.so     _ZN17VM_GenCollectFull4doitEv gc-begin
   6104 hotspot32597         libjvm.so _ZN15VM_GC_Operation13notify_gc_endEv gc-end
   6105 hotspot32597         libjvm.so _ZN26VM_GenCollectForAllocation4doitEv gc-end
   6106 hotspot32597         libjvm.so _ZN35VM_GenCollectForPermanentAllocation4doitEv gc-end
   6107 hotspot32597         libjvm.so     _ZN17VM_GenCollectFull4doitEv gc-end
   6132 hotspot32597         libjvm.so _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread class-initialization-end
   6133 hotspot32597         libjvm.so _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread class-initialization-erroneous
   6441 hotspot_jni32597         libjvm.so                jni_DeleteLocalRef DeleteLocalRef-entry
   6442 hotspot_jni32597         libjvm.so                jni_DeleteLocalRef DeleteLocalRef-return
   6443 hotspot_jni32597         libjvm.so           jni_DeleteWeakGlobalRef DeleteWeakGlobalRef-entry
   6444 hotspot_jni32597         libjvm.so           jni_DeleteWeakGlobalRef DeleteWeakGlobalRef-return
   6445 hotspot_jni32597         libjvm.so                 jni_DestroyJavaVM DestroyJavaVM-entry
   6446 hotspot_jni32597         libjvm.so                 jni_DestroyJavaVM DestroyJavaVM-return
  </code></pre>
  
  <p>Question: Is there a way to list ustack helpers and if they are loaded? Any way to get the Java stack?</p>