title: Dynamic Bytecode Instrumentation fails without any error
body: <h1><strong>Objective</strong></h1>
  
  <p>I'm doing dynamic bytecode instrumentation using a JVMTI agent. I have to instrument those methods which are "hot", that is, the methods which invoke JIT compiler. To do so I listen to a <code>CompiledLoadEvent</code> and inside its call back function, call <code>RetransformClasses</code>. This in turn invokes <code>ClassFileLoadHook</code> on the class containing "hot" function and actual instrumentation begins. </p>
  
  <h1><strong>Problem Premises</strong></h1>
  
  <p>Currently I'm instrumenting my class to spawn some threads. I also listen to thread starts and print them within my agent. With simple <code>ClassFileLoadHook</code> at class load time (without <code>RetransformClasses</code>), my instrumentation works perfectly and spawns new threads. I get following output when <code>ClassFileLoadHook</code> instruments at class load time:</p>
  
  <pre><code>Running Thread: Signal Dispatcher, Priority: 9, context class loader:Not Null
  Running Thread: main, Priority: 5, context class loader:Not Null
  Running Thread: Thread-0, Priority: 5, context class loader:Not Null
  Running Thread: Thread-1, Priority: 5, context class loader:Not Null
  Running Thread: Thread-2, Priority: 5, context class loader:Not Null
  Running Thread: Thread-3, Priority: 5, context class loader:Not Null
  Running Thread: Thread-4, Priority: 5, context class loader:Not Null
  Running Thread: Thread-6, Priority: 5, context class loader:Not Null
  Running Thread: Thread-5, Priority: 5, context class loader:Not Null
  Running Thread: Thread-7, Priority: 5, context class loader:Not Null
  Running Thread: DestroyJavaVM, Priority: 5, context class loader:: NULL
  </code></pre>
  
  <p>When I instrument the class file by invoking <code>RetransformClasses</code> and then <code>ClassFileLoadHook</code>, everything works fine but no threads are spawned and hence no effective instrumentation takes place. VM takes a long time even to execute the original code. </p>
  
  <p>I double checked both instrumentations using -XX:+TraceClassLoading. All the retransformed classes are loaded in both cases. Even the class I'm generating during runtime also gets loaded but no instrumentation happens. Below is the output of class loading trace:</p>
  
  <pre><code>[Loaded Test from __VM_RedefineClasses__]
  [Loaded Test_Worker_main_0 from file:/home/saqib/workspace/test/bin]
  </code></pre>
  
  <p>I'm generating second class during runtime and it loads into VM but I don't get any thread spawning.</p>
  
  <h1>Questions</h1>
  
  <ul>
  <li>Given my understanding of the problem (There is a high probability
  that I'd be wrong), why <code>ClassFileLoadHook</code> retransforms the class
  successfully during load time, but somehow doesn't behave correctly
  when JIT is invoked?</li>
  <li>Just writing the <code>RetransformClasses</code> function, with empty
  <code>ClassFileLoadHook</code> call back, also takes a lot of time without
  incurring any sort of error. What could be taking time? </li>
  </ul>
  
  <h1>Agent Code</h1>
  
  <h3>Compiled Load Event Call Back</h3>
  
  <pre><code>static int x = 1;
  void JNICALL
  compiled_method_load(jvmtiEnv *jvmti, jmethodID method, jint code_size,
          const void* code_addr, jint map_length, const jvmtiAddrLocationMap* map,
          const void* compile_info) {
      jvmtiError err;
      jclass klass;
  
      char* name = NULL;
      char* signature = NULL;
      char* generic_ptr = NULL;
  
      err = (*jvmti)-&gt;RawMonitorEnter(jvmti, lock);
      check_jvmti_error(jvmti, err, "raw monitor enter");
  
      err = (*jvmti)-&gt;GetMethodName(jvmti, method, &amp;name, &amp;signature,
              &amp;generic_ptr);
      check_jvmti_error(jvmti, err, "Get Method Name");
  
      printf("\nCompiled method load event\n");
      printf("Method name %s %s %s\n\n", name, signature,
              generic_ptr == NULL ? "" : generic_ptr);
  
      if (strstr(name, "main") != NULL &amp;&amp; x == 1) {
          x++;
          err = (*jvmti)-&gt;GetMethodDeclaringClass(jvmti, method, &amp;klass);
          check_jvmti_error(jvmti, err, "Get Declaring Class");
  
          err = (*jvmti)-&gt;RetransformClasses(jvmti, 1, &amp;klass);
          check_jvmti_error(jvmti, err, "Retransform class");
  
      }
  
      if (name != NULL) {
          err = (*jvmti)-&gt;Deallocate(jvmti, (unsigned char*) name);
          check_jvmti_error(jvmti, err, "deallocate name");
      }
      if (signature != NULL) {
          err = (*jvmti)-&gt;Deallocate(jvmti, (unsigned char*) signature);
          check_jvmti_error(jvmti, err, "deallocate signature");
      }
      if (generic_ptr != NULL) {
          err = (*jvmti)-&gt;Deallocate(jvmti, (unsigned char*) generic_ptr);
          check_jvmti_error(jvmti, err, "deallocate generic_ptr");
      }
  
      err = (*jvmti)-&gt;RawMonitorExit(jvmti, lock);
      check_jvmti_error(jvmti, err, "raw monitor exit");
  }
  </code></pre>
  
  <h3>Class File Load Hook</h3>
  
  <pre><code>void JNICALL
  Class_File_Load_Hook(jvmtiEnv *jvmti_env, JNIEnv* jni_env,
          jclass class_being_redefined, jobject loader, const char* name,
          jobject protection_domain, jint class_data_len,
          const unsigned char* class_data, jint* new_class_data_len,
          unsigned char** new_class_data) {
      jvmtiError err;
      unsigned char* jvmti_space = NULL;
  
      if (strstr(name, "Test") != NULL &amp;&amp; x == 2) {
          char* args = "op";
  
          javab_main(2, args, class_data, class_data_len);
  
          err = (*jvmti_env)-&gt;Allocate(jvmti_env, (jlong)global_pos, &amp;jvmti_space);
          check_jvmti_error(jvmti_env, err, "Allocate new class Buffer.");
  
          (void)memcpy((void*)jvmti_space, (void*)new_class_ptr, (int)global_pos);
  
          *new_class_data_len = (jint)global_pos;
          *new_class_data = jvmti_space;
  
          if ( new_class_ptr != NULL ) {
              (void)free((void*)new_class_ptr);
          }
  
  #if DEBUG
          printf("Size of the class is: %d\n", class_data_len);
          for (int i = 0; i &lt; class_data_len; i += 4) {
              if (i % 16 == 0)
                  printf("\n");
  
              printf("%02x%02x  %02x%02x  ", new_class_data[i],
                      new_class_data[i + 1], new_class_data[i + 2],
                      new_class_data[i + 3]);
          }
          printf("\n");
          system("javap -c -v Test_debug");
  #endif
          x++;
      }
  }
  </code></pre>
  
  <p>Here <code>javab_main</code> returns the instrumented <code>char *</code> array which is correct. The instrumented array is stored in a global variable <code>new_class_ptr</code> which is copied into <code>new_class_data</code>. To debug the output of the instrumentation, I also printed the instrumented class in a file called Test_debug and invoking <code>javap</code> on it produces desired result.</p>
  
  <p>The complete agent file is given here:
  <a href="https://gist.github.com/saqibahmed515/9864ec018edeca14e14ce518131c753c" rel="nofollow noreferrer">Agent.c</a></p>
  
  <h1>Original Code:</h1>
  
  <pre><code>            for (int i = 0; i &lt; s; i++)
                  for (int j = 0; j &lt; s; j++) {
                      c2[i][j] = 0;
                      for (int k = 0; k &lt; s; k++)
                          c2[i][j] += a[i][k] * b[k][j];
                  }
  </code></pre>
  
  <h1>Instrumented Code: (Equivalent)</h1>
  
  <pre><code>Thread[] threads = new Thread[NTHREADS];    
              for (int i = 0; i &lt; NTHREADS ; i++) {
                  final int lb = i * SIZE/NTHREADS;
                  final int ub = (i+1) * SIZE/NTHREADS;
                  threads[i] = new Thread(new Runnable() {
  
                      public void run() {
                          for (int i = lb; i &lt; ub; i++)
                              for (int j = 0; j &lt; SIZE; j++) {
                                  c2[i][j] = 0;
                                  for (int k = 0; k &lt; SIZE; k++)
                                      c2[i][j] += a[i][k] * b[k][j];
                              }
                      }
                  });
                  threads[i].start();
              }
  
              // wait for completion
              for (int i = 0; i &lt; NTHREADS; i++) {
                  try {
                      threads[i].join();
                  } catch (InterruptedException ignore) {
                  }
              }
  </code></pre>
  
  <h1>Java Version</h1>
  
  <pre><code>openjdk version "1.8.0-internal-debug"
  OpenJDK Runtime Environment (build 1.8.0-internal-debug-saqib_2016_12_26_10_52-b00)
  OpenJDK 64-Bit Server VM (build 25.71-b00-debug, mixed mode)
  </code></pre>