title: Hooking CreateFile in notepad.exe does not catch API calls
body: <p>My ultimate goal is to track file operations done by explorer.exe via hooking file api in kernel32.dll, however I have yet to get that working (either explorer.exe is not calling the APIs, or something is wrong on my end).  In order to figure out what is going on, I've set a goal to track whenever notepad.exe creates a file, however this has failed for some reason as well!</p>
  
  <p>I have 3 Visual Studio 2012 C++ projects: my DLL, a DLL injector that will force any executable to load my dll (although it may fail if Unicode/Multibyte and 32/64bit settings don't match up), and a test program that calls API.  I have a batch file that will start my test program, and then use the injector program to load my DLL into the test program.  The weird part is that the outputs <em>do</em> show that the APIs are being tracked on the test program (spawns the console and prints everything!) but nothing happens for notepad.exe (no console = no operations caught).</p>
  
  <p>Here is the DLL that hooks the API (uses the mhook library).  The format and concepts are taken from <a href="http://www.codeproject.com/Articles/49319/Easy-way-to-set-up-global-API-hooks" rel="nofollow noreferrer">this</a> guide.  The important things to note are that I hook CreateFile(A/W), and spawn a console to print file I/O logs when the first operation occurs.</p>
  
  <pre><code>#include "stdafx.h"
  #include "mhook/mhook-lib/mhook.h"
  
  //////////////////////////////////////////////////////////////////////////
  // Defines and typedefs
  typedef HANDLE (WINAPI *CreateFileWFP)(
      _In_ LPCWSTR lpFileName,
      _In_ DWORD dwDesiredAccess,
      _In_ DWORD dwShareMode,
      _In_opt_ LPSECURITY_ATTRIBUTES lpSecurityAttributes,
      _In_ DWORD dwCreationDisposition,
      _In_ DWORD dwFlagsAndAttributes,
      _In_opt_ HANDLE hTemplateFile
      );
  typedef HANDLE (WINAPI *CreateFileAFP)(
      _In_ LPCSTR lpFileName,
      _In_ DWORD dwDesiredAccess,
      _In_ DWORD dwShareMode,
      _In_opt_ LPSECURITY_ATTRIBUTES lpSecurityAttributes,
      _In_ DWORD dwCreationDisposition,
      _In_ DWORD dwFlagsAndAttributes,
      _In_opt_ HANDLE hTemplateFile
      );
  
  //////////////////////////////////////////////////////////////////////////
  // Original function
  CreateFileWFP OriginalCreateFileW = (CreateFileWFP)::GetProcAddress(::GetModuleHandle(TEXT("kernel32")), "CreateFileW");
  CreateFileAFP OriginalCreateFileA = (CreateFileAFP)::GetProcAddress(::GetModuleHandle(TEXT("kernel32")), "CreateFileA");
  
  //////////////////////////////////////////////////////////////////////////
  // Some Helper Stuff
  
  struct Console{
      HANDLE handle;
  
      Console(){ handle = INVALID_HANDLE_VALUE; }
  
      void write(LPCWSTR text){
          if(handle==INVALID_HANDLE_VALUE){
              AllocConsole();
              handle = GetStdHandle(STD_OUTPUT_HANDLE);
          }
          DWORD numCharsWritten = 0;
          WriteConsoleW(handle, text, (DWORD)wcslen(text), &amp;numCharsWritten,NULL);
      }
      void write(LPCSTR text){
          if(handle==INVALID_HANDLE_VALUE){
              AllocConsole();
              handle = GetStdHandle(STD_OUTPUT_HANDLE);
          }
          DWORD numCharsWritten = 0;
          WriteConsoleA(handle, text, (DWORD)strlen(text), &amp;numCharsWritten,NULL);
      }
  } console;
  
  void operationPrint(LPCWSTR left, LPCWSTR middle, LPCWSTR right){
      console.write(left);
      console.write(middle);
      console.write(right);
      console.write(L"\n");
  }
  void operationPrint(LPCSTR left, LPCSTR middle, LPCSTR right){
      console.write(left);
      console.write(middle);
      console.write(right);
      console.write(L"\n");
  }
  
  //////////////////////////////////////////////////////////////////////////
  // Hooked function
  
  HANDLE HookedCreateFileW(
      _In_ LPCWSTR lpFileName,
      _In_ DWORD dwDesiredAccess,
      _In_ DWORD dwShareMode,
      _In_opt_ LPSECURITY_ATTRIBUTES lpSecurityAttributes,
      _In_ DWORD dwCreationDisposition,
      _In_ DWORD dwFlagsAndAttributes,
      _In_opt_ HANDLE hTemplateFile
      ){
          HANDLE out = OriginalCreateFileW(
                          lpFileName,
                          dwDesiredAccess,
                          dwShareMode,
                          lpSecurityAttributes,
                          dwCreationDisposition,
                          dwFlagsAndAttributes,
                          hTemplateFile);
          if(out == INVALID_HANDLE_VALUE) return out; //ignore failiures
          operationPrint(L"CreatedW file",L" at ",lpFileName);
          return out;
  }
  HANDLE HookedCreateFileA(
      _In_ LPCSTR lpFileName,
      _In_ DWORD dwDesiredAccess,
      _In_ DWORD dwShareMode,
      _In_opt_ LPSECURITY_ATTRIBUTES lpSecurityAttributes,
      _In_ DWORD dwCreationDisposition,
      _In_ DWORD dwFlagsAndAttributes,
      _In_opt_ HANDLE hTemplateFile
      ){
          HANDLE out = OriginalCreateFileA(
                          lpFileName,
                          dwDesiredAccess,
                          dwShareMode,
                          lpSecurityAttributes,
                          dwCreationDisposition,
                          dwFlagsAndAttributes,
                          hTemplateFile);
          if(out == INVALID_HANDLE_VALUE) return out; //ignore failiures
          operationPrint("CreatedA file"," at ",lpFileName);
          return out;
  }
  
  //////////////////////////////////////////////////////////////////////////
  // Entry point
  
  BOOL WINAPI DllMain(
      __in HINSTANCE  hInstance,
      __in DWORD      Reason,
      __in LPVOID     Reserved
      )
  {        
      switch (Reason)
      {
      case DLL_PROCESS_ATTACH:
          Mhook_SetHook((PVOID*)&amp;OriginalCreateFileW, HookedCreateFileW);
          Mhook_SetHook((PVOID*)&amp;OriginalCreateFileA, HookedCreateFileA);
          break;
  
      case DLL_PROCESS_DETACH:
          FreeConsole();
          Mhook_Unhook((PVOID*)&amp;OriginalCreateFileW);
          Mhook_Unhook((PVOID*)&amp;OriginalCreateFileA);
          break;
      }
  
      return TRUE;
  }
  </code></pre>
  
  <p>I can't find exactly where I found the injector program, but it is <em>nearly</em> identical to <a href="http://www.mpgh.net/forum/290-crossfire-spammers-injectors-multi-tools/332194-how-make-dll-injection-c.html" rel="nofollow noreferrer">this</a> guide.  Some of the comments are even the same, so I'm pretty sure one took from the other (not sure which from whom though).  In any case, I just made minor alterations so that it works when compiled either under Unicode or Multibyte.  I won't post the entire code here unless requested because I think it's a waste of space, but here is the important part.</p>
  
  <pre><code>#include "Injector.h"
  #include &lt;windows.h&gt;
  #include &lt;tlhelp32.h&gt;
  #include &lt;shlwapi.h&gt;
  #include &lt;conio.h&gt;
  #include &lt;stdio.h&gt; 
  #include "DebugPrint.h"
  #include &lt;atlbase.h&gt;
  
  using namespace std;
  
  Injector::Injector(void)
  {
  }
  
  
  Injector::~Injector(void)
  {
  } 
  
  #define CREATE_THREAD_ACCESS (PROCESS_CREATE_THREAD | PROCESS_QUERY_INFORMATION | PROCESS_VM_OPERATION | PROCESS_VM_WRITE | PROCESS_VM_READ) 
  
  bool Injector::Inject(string procName, string dllName){
     DWORD pID = GetTargetThreadIDFromProcName(procName.c_str()); 
     return Inject(pID,dllName);
  }
  
  bool Injector::Inject(DWORD pID, string dllName){
     const char* DLL_NAME = dllName.c_str();
  
     HANDLE Proc = 0; 
     HMODULE hLib = 0; 
     LPVOID RemoteString, LoadLibAddy; 
  
     if(!pID) 
        return false; 
  
     Proc = OpenProcess(PROCESS_ALL_ACCESS, FALSE, pID); 
     if(!Proc) 
     { 
        DEBUG_PRINT("OpenProcess() failed: %d", GetLastError()); 
        return false; 
     } 
  
     LoadLibAddy = (LPVOID)GetProcAddress(GetModuleHandle(TEXT("kernel32.dll")), "LoadLibraryA"); 
  
     // Allocate space in the process for our &lt;strong class="highlight"&gt;DLL&lt;/strong&gt; 
     RemoteString = (LPVOID)VirtualAllocEx(Proc, NULL, strlen(DLL_NAME), MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE); 
  
     // Write the string name of our &lt;strong class="highlight"&gt;DLL&lt;/strong&gt; in the memory allocated 
     WriteProcessMemory(Proc, (LPVOID)RemoteString, DLL_NAME, strlen(DLL_NAME), NULL); 
  
     // Load our &lt;strong class="highlight"&gt;DLL&lt;/strong&gt; 
     CreateRemoteThread(Proc, NULL, NULL, (LPTHREAD_START_ROUTINE)LoadLibAddy, (LPVOID)RemoteString, NULL, NULL); 
  
     CloseHandle(Proc); 
     return true; 
  }
  
  DWORD Injector::GetTargetThreadIDFromProcName(const char* ProcName)
  {
      PROCESSENTRY32 pe;
      HANDLE thSnapShot;
      BOOL retval, ProcFound = false;
  
      thSnapShot = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
      if(thSnapShot == INVALID_HANDLE_VALUE)
      {
          //MessageBox(NULL, "Error: Unable to create toolhelp snapshot!", "2MLoader", MB_OK);
          DEBUG_PRINT("Error: Unable to create toolhelp snapshot!");
          return false;
      }
  
      pe.dwSize = sizeof(PROCESSENTRY32);
  
      retval = Process32First(thSnapShot, &amp;pe);
      while(retval)
      {
  #ifdef _UNICODE
          char peSzExeFile[MAX_PATH];
          wcstombs_s(NULL,peSzExeFile,MAX_PATH,pe.szExeFile,MAX_PATH);
  #else
          const char* peSzExeFile = pe.szExeFile;
  #endif
          DEBUG_PRINT("\nSearching for process: %s ",peSzExeFile);
          if(!strcmp(peSzExeFile, ProcName))
          {
              DEBUG_PRINT(" Found!\n\n");
              return pe.th32ProcessID;
          }
          retval = Process32Next(thSnapShot, &amp;pe);
      }
      return 0;
  }
  </code></pre>
  
  <p>Finally, my test program just calls some file APIs to see if the injected DLL can catch them.  As mentioned before, it does catch the calls quite successfully.</p>
  
  <pre><code>#include &lt;windows.h&gt;
  #include &lt;tchar.h&gt;
  
  using namespace std;
  
  #ifdef _UNICODE
  #define printf(X,...) wprintf(TEXT(X),__VA_ARGS__); //I don't want to have to keep replacing printf whenever I switch to Unicode or Multibyte
  #endif
  
  #ifdef _DEBUG
  int _tmain(int argc, TCHAR* argv[]){ //makes a console.  printf() will have a place to go in this case
  #else
  int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow){ //no console
  #endif
      Sleep(2000); //let DLL finish loading
  
      LPCTSTR fileSrc = TEXT("C:\\Users\\jajoach\\Desktop\\hi.txt");
      LPCTSTR fileDst = TEXT("C:\\Users\\jajoach\\Desktop\\hi\\hi.txt");
  
      printf("Moving file from %s to %s\n",fileSrc,fileDst);
      MoveFile(fileSrc,fileDst);
      Sleep(1000);
  
      printf("Moving file from %s to %s\n",fileSrc,fileDst);
      MoveFile(fileDst,fileSrc);
      Sleep(1000);
  
      printf("Copying file from %s to %s\n",fileSrc,fileDst);
      CopyFile(fileSrc,fileDst,true);
      Sleep(1000);
  
      printf("Deleting file %s\n",fileDst);
      DeleteFile(fileDst);
      Sleep(1000);
  
      printf("Creating file %s\n",fileDst);
      HANDLE h=CreateFile(fileDst,0,0,NULL,CREATE_NEW,FILE_ATTRIBUTE_NORMAL,NULL);
      Sleep(1000);
  
      printf("Deleting file %s\n",fileDst);
      CloseHandle(h);
      DeleteFile(fileDst);
  
      Sleep(5000);
  
      return 0;
  }
  </code></pre>
  
  <p>Here is the confirmed output for Unicode (as noted by the 'W's) and Release mode, and what I was expecting (but didn't get) from both notepad.exe and explorer.exe.  For the record, the test also works for Multibyte, but instead gives 'A's as expected.
  <img src="https://i.stack.imgur.com/9I3Os.png" alt="enter image description here"></p>
  
  <p>I had thought that maybe explorer.exe and notepad.exe don't use these API for their file I/O, but my research says otherwise.  <a href="https://stackoverflow.com/questions/14235482/ms-detours-express-3-0-is-not-hooking-createfile-win32-api-function-properly">This</a> post hooks CreateFile in notepad.exe using Detours and reports success for that application.  Additionally, ProgramMonitor clearly shows notepad.exe calling CreateFile during a <code>Saveas</code> operation (after many failed requests with different parameters...):
  <img src="https://i.stack.imgur.com/DJQVD.png" alt="enter image description here"></p>
  
  <p>Nevermind about explorer.exe for now; <strong>why isn't my hook working for notepad.exe when I do Saveas?</strong></p>
  
  <p><strong>EDIT:</strong> I forgot to mention that I also hook MoveFile(A/W) and CopyFile(A/W) with the test program, but I removed that code from the DLL on this post for brevity.</p>