Logical characterization of P versus NP problem (and references for least fixed point logic)
Your second question first. The context of your question is called descriptive complexity theory. In this context, I would suggest the book Finite Model Theory by Heinz-Dieter Ebbinghaus and Jörg Flum. As far as I recall it thoroughly covers both logics and their connection to P, respectively NP.

Now to your first question. Of course, as soon as both complexity classes had been characterized by logics, there were hopes to settle the P=NP question this way. Yet by now the problem has eluded this approach for several decades. One would need to pick some problem from NP and show that it is not expressible in FO(LFP) (first-order logic with least-fixed point operators). Essentially the only tool for proving non-expressibility are Ehrenfeucht–Fraïssé games. These games are notoriously hard to analyze in the presence of non-monadic second-order variables, which are needed in the case of FO(LFP). See also the paper https://eccc.weizmann.ac.il/report/2013/065/ by Yijia Chen and Jörg Flum.