My Arduino Fingerprint Sensor FPM10A is not working with an Arduino Nano with Atmega168 processor
Did you buy the arduino board Nano Atmega168 on Aliexpress? Sometimes the cheapest arduino compatible boards are okay, but sometimes they are stripped of decoupling capacitors or the polyfuse is missing (on a uno board). Sometimes they put a Atmega168 on the board, it is a few cents cheaper.

If you buy the cheapest nano compatible board, look at the photo if the quality is reasonable. The newer arduino boards have a ARM Cortex processors. The Atmega328p is still okay for small projects. I suggest to use the Atmega328p as the minimum.
A Arduino nano Atmega328p clone at Aliexpress costs about 2 dollars, so there is no reason to buy an Atmega168.

When the compiler tells you that "Low memory available, stability problems may occur" for the ram, then you can almost be sure that it will not work.  

I try to keep about 300 bytes ram available for local variables, stack and the heap. It depends how much memory the libraries require. I also don't use the String objects for the Arduino uno or nano.
For a beginner, you can try to keep at least 500 bytes ram available.