I do malware analysis on Windows. I run hundreds of Windows PEs per day and it is actually relatively common for a file to not run (or sometimes not run on just one specific version of Windows) and I get an error message such as This is not a valid Win32 application. However, when I open the file in a hex editor, it does have the MZ, PE signature, and even the sections intact. Also, programs such as PE Explorer can open the file fine and claim that it's a valid PE file, without even having to open it in safemode for example. Note that I am NOT talking about error messages concerning 64 bit on a 32 bit version of Windows, as that is self-explanatory.Below is a screenshot of one such file. This file, however, has no DOS stub at all and the DOS header data structures are all set to 0 except for e_lfanew which does point to a PE sig. However, the Windows loader says it's not a valid Win32 app at least on my version of Win7 64bit.I do know that the loader essentially reads the data structures and from that, it does things such as allocate stack and heap memory, determine which symbols are needed and from which DLL files, as well as a few other tasks. So my assumption based off of that would be that if for example, one of these header data structures told the loader to do something that made no sense such as allocate negative space, too much space, or contained nonsense in a structure that was critical for the loader to work, it could crash it. However, this is just speculation on my part.
What PE anomalies can crash the Windows Loader or cause a file to not load?
windows;malware;pe
Based on my experience, these type of errors could happen when MajorOperatingSystem/MinorOperatingSystem and/or MajorSubsystemVersion/MinorSubsystemVersion are not compatible with the system you are trying to run the PE file. You can find more information about that in Corkami page regarding PE file format. Also, you can look at the Peering inside the PE... paper on Microsoft website which says for these fields:WORD MajorOperatingSystemVersion  WORD MinorOperatingSystemVersion  The minimum version of the operating system required to use this executable. This field is somewhat ambiguous since the subsystem fields (a few fields later) appear to serve a similar purpose. This field defaults to 1.0 in all Win32 EXEs to date.andWORD MajorSubsystemVersion  WORD MinorSubsystemVersion  Contains the minimum subsystem version required to run the executable. A typical value for this field is 3.10 (meaning Windows NT 3.1).Also, if you are interested in anomalies and undocumented stuff of the PE file format, I suggest this very good paper which deals with anomalies in PE which can break poorly implemented parsers. Also, this site of Corkami contains POCs of various working PE samples (both source code and binaries), which test the format to its limits and can be used for testing.Hope this helps out a bit.