title: SetWindowsHookEx is injecting 32-bit DLL into 64-bit process and vice versa
body: <p>I've been working on an application that requires monitoring <em><strong>thread specific</strong></em> mouse activity (<code>WH_MOUSE</code>) on another process and encountered something very curious.</p>
  
  <p>After finding out that <a href="https://support.microsoft.com/en-us/kb/318804" rel="nofollow">this is not possible via exclusively managed code</a> if I don't want to use <code>WH_MOUSE_LL</code> and that I'd need a native DLL export to inject itself in the target process, I set out and created it in C++ according to what scattered documentation I could find on the subject and then tried using it to hook into Notepad.</p>
  
  <p>Although according to <a href="https://msdn.microsoft.com/en-us/library/system.runtime.interopservices.marshal.getlastwin32error(v=vs.110).aspx" rel="nofollow"><code>GetLastWin32Error</code></a> the injection succeeded, I was not getting notified of mouse events. After nearly giving up and going for the low level global hook option, I re-read the "Remarks" section of <a href="https://msdn.microsoft.com/en-us/library/ms644990(v=vs.85).aspx" rel="nofollow">this article</a> which made me suspect that the problem may be because of the "bitness" of my code vs notepad:</p>
  
  <blockquote>
    <p>A 32-bit DLL cannot be injected into a 64-bit process, and a 64-bit
    DLL cannot be injected into a 32-bit process. If an application
    requires the use of hooks in other processes, it is required that a
    32-bit application call SetWindowsHookEx to inject a 32-bit DLL into
    32-bit processes, and a 64-bit application call SetWindowsHookEx to
    inject a 64-bit DLL into 64-bit processes.</p>
  </blockquote>
  
  <p>However, both my native DLL and managed application were compiled as x64, and I was trying to hook into the 64-bit version of notepad, so it should've worked fine, but I took a shot in the dark anyway and went into the <code>SysWOW64</code> folder and opened the 32-bit Notepad from there, tried hooking in again and this time the hook worked beautifully!</p>
  
  <p>Curiously, I then recompiled both my native DLL and managed app as x86 and tested it it against the 32-bit Notepad and it didn't work, but it worked on my normal 64-bit Notepad!</p>
  
  <p>How am I possibly seem to be able to inject a 32-bit DLL into a 64-bit process and vice versa!</p>
  
  <p>Although my original problem has been solved and I can continue with my app's development, the curiosity as to <em>why</em> I'm observing this strange inverse behavior from <code>SetWindowsHookEx</code> is driving me insane, so I really hope someone will be able to shed some light on this.</p>
  
  <p>I know this a lot of talk and no code, but the code for even a sample app is rather large and comes in both managed and unmanaged flavors, however I'll promptly post any piece of the code you think might be relevant.</p>
  
  <p>I've also created a sample app so you can test this behavior yourself. It's a simple WinForms app that tries to hook into Notepad and displays its mouse events:</p>
  
  <p><a href="http://saebamini.com/HookTest.zip" rel="nofollow">http://saebamini.com/HookTest.zip</a></p>
  
  <p>It contains both an x86 version and an x64 version. On my machine (I'm on a 64-bit Windows 7), the x86 version only works with 64-bit Notepad, and the x64 version only works with 32-bit Notepad (from SysWOW64).</p>
  
  <p><strong>UPDATE - Relevant Bits of Code:</strong></p>
  
  <p>C# call to the unmanaged library:</p>
  
  <pre><code>public SetCallback(HookTypes type)
  {
      _type = type;
  
      _processHandler = new HookProcessedHandler(InternalHookCallback);
      SetCallBackResults result = SetUserHookCallback(_processHandler, _type);
      if (result != SetCallBackResults.Success)
      {
          this.Dispose();
          GenerateCallBackException(type, result);
      }
  }
  
  public void InstallHook()
  {
      Process[] bsProcesses = Process.GetProcessesByName("notepad");
      if(bsProcesses.Length == 0)
      {
          throw new ArgumentException("No open Notepad instance found.");
      }
      ProcessThread tmp = GetUIThread(bsProcesses[0]);
  
      if (!InitializeHook(_type, tmp.Id))
      {
          throw new ManagedHooksException("Hook initialization failed.");
      }
      _isHooked = true;
  }
  
  [System.Runtime.InteropServices.DllImport("user32.dll")]
  private static extern int GetWindowThreadProcessId(IntPtr hWnd, IntPtr procid);
  
  // 64-bit version
  [DllImport("SystemHookCore64.dll", EntryPoint = "InitializeHook", SetLastError = true,
          CharSet = CharSet.Unicode, ExactSpelling = true,
          CallingConvention = CallingConvention.Cdecl)]
  private static extern bool InitializeHook(HookTypes hookType, int threadID);
  
  
  [DllImport("SystemHookCore64.dll", EntryPoint = "SetUserHookCallback", SetLastError = true,
          CharSet = CharSet.Unicode, ExactSpelling = true,
          CallingConvention = CallingConvention.Cdecl)]
  private static extern SetCallBackResults SetUserHookCallback(HookProcessedHandler hookCallback, HookTypes hookType);
  </code></pre>
  
  <p>C++:</p>
  
  <pre><code>HookProc UserMouseHookCallback = NULL;
  HHOOK hookMouse = NULL;
  
  HINSTANCE g_appInstance = NULL;
  
  MessageFilter mouseFilter;
  
  bool InitializeHook(UINT hookID, int threadID)
  {
      if (g_appInstance == NULL)
      {
          return false;
      }
  
      if (hookID == WH_MOUSE)
      {
          if (UserMouseHookCallback == NULL)
          {
              return false;
          }
  
          hookMouse = SetWindowsHookEx(hookID, (HOOKPROC)InternalMouseHookCallback, g_appInstance, threadID);
          return hookMouse != NULL;
      }
  }
  
  int SetUserHookCallback(HookProc userProc, UINT hookID)
  {   
      if (userProc == NULL)
      {
          return HookCoreErrors::SetCallBack::ARGUMENT_ERROR;
      }
  
      if (hookID == WH_MOUSE)
      {
          if (UserMouseHookCallback != NULL)
          {
              return HookCoreErrors::SetCallBack::ALREADY_SET;
          }
  
          UserMouseHookCallback = userProc;
          mouseFilter.Clear();
          return HookCoreErrors::SetCallBack::SUCCESS;
      }
  
      return HookCoreErrors::SetCallBack::NOT_IMPLEMENTED;
  }
  
  int FilterMessage(UINT hookID, int message)
  {
      if (hookID == WH_MOUSE)
      {
          if(mouseFilter.AddMessage(message))
          {
              return HookCoreErrors::FilterMessage::SUCCESS;
          }
          else
          {
              return HookCoreErrors::FilterMessage::FAILED;
          }
      }
  
      return HookCoreErrors::FilterMessage::NOT_IMPLEMENTED;
  }
  
  static LRESULT CALLBACK InternalMouseHookCallback(int code, WPARAM wparam, LPARAM lparam)
  {
      if (code &lt; 0)
      {
          return CallNextHookEx(hookMouse, code, wparam, lparam);
      }
  
      if (UserMouseHookCallback != NULL &amp;&amp; !mouseFilter.IsFiltered((int)wparam))
      {
          UserMouseHookCallback(code, wparam, lparam);
      }
  
      return CallNextHookEx(hookMouse, code, wparam, lparam);
  }
  </code></pre>