title: How to integrate Darknet Yolo inside Unity/Hololens?
body: <p>I am currently trying to integrate the Darknet YOLO (<a href="https://pjreddie.com/darknet/yolo/" rel="nofollow noreferrer">https://pjreddie.com/darknet/yolo/</a>) computer vision package into Unity in order to investigate for a research project how fast real-time object detection on it performs in a Hololens device. So far, I have been able to export the YOLO package as a DLL file, and have Unity communicate with it through function calls. Here are the Marshalled functions that I have created inside the yolo_dll.cpp file that is a part of the YOLO package. These are the functions that I invoke from the Unity end:</p>
  
  <pre><code>extern "C" _declspec(dllexport) Detector* _stdcall Init() {
      return new Detector("yolo_write_into.txt", "yolo.weights");
  }
  
  extern "C" _declspec(dllexport) void _stdcall DeleteDetector(Detector* detector) {
      delete detector;
  }
  
  extern "C" _declspec(dllexport) int _stdcall Test(int s) {
      return s;
  }
  
  extern "C" _declspec(dllexport) int _stdcall OutsideDetect(Detector* 
  mDetect, bbox_t** results, int* resultSize, int h, int w, float threshold = 
  0.2f, bool use_mean = false) {
      /*image_t newImg;
      newImg.data = data;
      newImg.h = h;
      newImg.w = w;
      newImg.c = 3;*/
  
      std::vector&lt;bbox_t&gt; vec = mDetect-&gt;detect("Assets/yolo_write_image.jpg", 
      threshold, use_mean);
      //TODO: clean up boxArray
      bbox_t* boxArray = new bbox_t[vec.size()];
      for (int i = 0; i &lt; vec.size(); i++) {
          boxArray[i] = vec[i];
      }
      *resultSize = vec.size();
      *results = boxArray;
      return 1;
  }
  
  extern "C" _declspec(dllexport) void _stdcall DeleteBArray(bbox_t** results) 
  {
      delete[] results;
  }
  </code></pre>
  
  <p>And then here are the marshalled c# functions that I call that match the signatures of the ones on the c++ side:</p>
  
  <pre><code>internal static class YoloDLL
  {
      [DllImport("yolo_cpp_dll", EntryPoint = "Test")]
      public static extern int DLLTest(int s);
  
      [DllImport("yolo_cpp_dll", EntryPoint = "OutsideDetect")]
      public static extern int OutsideDetect(IntPtr mDetect, out IntPtr results, out int resultSize, int h, int w, float threshold, bool use_mean);
  
      [DllImport("yolo_cpp_dll", EntryPoint = "Init", CharSet = CharSet.Unicode)]
      public static extern IntPtr Init();
  
      [DllImport("yolo_cpp_dll", EntryPoint = "DeleteDetector", CharSet = CharSet.Unicode)]
      public static extern void DestroyDetector(IntPtr detector);
  
      [DllImport("yolo_cpp_dll", EntryPoint = "SetDebugFunction", CharSet = CharSet.Unicode)]
      public static extern void SetDebugFunction(IntPtr fp);
  
      [DllImport("yolo_cpp_dll", EntryPoint = "DebugLogDLL", CharSet = CharSet.Unicode)]
      public static extern void DebugLogDLL(string s);
  
  
      [DllImport("yolo_cpp_dll", EntryPoint = "DeleteBArray", CharSet = CharSet.Unicode)]
      public static extern void DeleteBArray(IntPtr results);
  }
  </code></pre>
  
  <p>I have a Unity script that uses the PhotoCapture class to take a picture with my computer's webcam, and once it saves the pic in memory, it invokes the external c++ detect() function to retrieve the recently-taken picture from memory, detect objects in it, and return an array of bounding boxes that locate where the objects are in the picture. Once the array of bounding boxes have been returned, I simply debug log all of them on the Unity console for now, and repeat the process of taking a picture and using YOLO detect() on it as soon as the information is output to the console. I have been able to have Unity and YOLO successfully communicate with each other. However, on average, it takes 10 seconds from the time one image's bounding boxes have been displayed to the time that the next image's bounding boxes have been displayed on the Unity debug console. I need the YOLO package to be able to process images and output feedback in real-time (30-60fps), and it needs to work on the Hololens rather than just on Unity. I tried exporting this as a build into the Hololens, and when I tried to open the application inside the Hololens, it simply wouldn't open. Therefore, I have some questions:</p>
  
  <ul>
  <li>Am I able to somehow integrate Unity's video capture features into YOLO rather than constantly taking pictures and processing them individually? Will this work on Hololens?</li>
  <li>In the YOLO package, I had to comment out all sections that had #ifdef GPU because if any of them were active, they would merely crash Unity right when I press "Play". Am I able to make these sections work with Unity somehow, and would they speed up processing time?</li>
  <li>Is invoking marshalled functions from YOLO through a DLL playing a role in the huge lag in between image analysis cycles? Is there a way to compensate for this?</li>
  <li>Is the fact that I am using an outside DLL the reason why the application is not opening up inside the Hololens?</li>
  </ul>
  
  <p>I would greatly appreciate any steps in the right direction in being able to integrate this package into the Hololens so that I can implement YOLO's real-time object detection features!</p>