How can I build a fool proof security system?
It is true that there is no general algorithm $D$ which can take as input any program $M$ and decide whether $M$ is malicious or not. Rice's theorem applies, here.

However, Rice's theorem does not imply that there is no $D$ that can do its job on a useful subset of programs $M$. This subset can not be the whole set of programs, according to Rice, but it could still be large enough to be useful in practice.

I'd say that most of the research in formal verification is about finding such a $D$ for specific cases. Usually the typical verifier $D$ outputs results meaning "I established that $M$ is malicious", "I established that $M$ is not malicious", or "my analysis was inconclusive, $M$ is too complex for me".

Essentially, we allow $D$ to throw the sponge, but hope that it won't.
We can't have a guaranteed YES or NO without also having, in infinitely many cases, the "inconclusive" result.

A naive technique to implement a $D$ could be to run $M$ for 1000 steps, and sees is $M$ does something bad. If it does, output "$M$ is malicious", otherwise output "inconclusive".

Static analyzers and model checkers (to mention two common approaches) only succeed on a subset of cases, but they can still be quite useful.