Is there any kind of software or research or paper which discusses replacement of frequent x86 instructions with ones which are less common and thus less understandable to the attacker (floating point/SSE/Virtualization/undocumented) while still maintaining the functionality?For example, I wan to replace this    PUSH EBP    MOV EBP,ESP    ...    PUSH DWORD [0x0BEE]    PUSH 3    CALL <check>    TEST EAX,EAX    JE <0xabcd>    PUSH <text1>    PUSH [EBP+5]    CALL <MessageBox>0xabcd:    PUSH <text2>    PUSH [EBP+5]    CALL <MessageBox>with this    AESKEYGENASSIST    VFMSUBADDPD    MOVLPS    PMADDUBSW    RET    FLDL2T    CMPXCHG8B    AESKEYGENASSIST    VFMSUBADDPD    MOVLPS    CMPXCHG8B    STOSW    VMLAUNCH    etc etcwhile still performing the same operation.
Replacing common x86 instructions with less known ones
obfuscation;x86
I do not think code morphism is the or an answer to this question. What the question was about is obfuscating the algorithm implementation by using less common or undocumented assembly instructions. This can actually be done by some compilers when extensive optimizations are turned on. For example compilers like the Intel C Compiler, GCC, or PGI can autovectorize loops when matched to some internal patterns (reductions, matrix multiplications, ...) and when the target architecture supports vectorization. Other optimizations can lead to extremely tricky assembly code but still, it can always be reversed since the compiler performs no explicit obfuscation and because most of what the compiler does is pattern matching. Of course if you associate a high level pattern to a low level one, well, you lose the obfuscation and your code can easily be reversed. Thus techniques as the one you're looking for can only be performed by hand either by writing high level code using compiler intrinsics and alternative constructs or at the assembly level.   If you are really interested in obfuscation techniques I recommend you going over Jan CAPPAERT's PhD thesis : https://www.cosic.esat.kuleuven.be/publications/thesis-199.pdf, it covers some nice techniques used not only on malware but on industrial software too. The bibliography is quite rich.You can also check this talk given by Sean Taylor at Defcon on how to make the compiler do the obfuscation : https://www.defcon.org/images/defcon-17/dc-17-presentations/defcon-17-sean_taylor-binary_obfuscation.pdf.About polymorphism, it is a nice obfuscation technique though it is rarely used in malware nowadays, and for many reasons. One of them is that few, if none, malware authors write code in assembly anymore, and most use frameworks and engines. You have to keep in mind that writing obfuscated assembly code is an art ... and that now it is used to harden the reverse engineering process for profit not for the challenge.I've been working on a GCC plugin that adds an optimization pass which performs code obfuscation on the IR - internal representation (GIMPLE) - of a code before applying another obfuscation pass at the assembly level. The interesting thing about this approach is that you have the CFG (Control Flow Graph) of the program at compile time, and you can apply many obfuscation algorithms and techniques in order to break it into other equivalent CFGs and then assess which suites best and use it throughout the remaining compilation phases.Hope my post helps.