Go back to Stock Kernel
Here is a list with stock kernels for nexus 4. As said before, simply download it and flash it like any other kernel. 

http://forum.xda-developers.com/nexus-4/development/kernel-stock-flashable-kernels-t2381128