Possible to shift image pixel-wise with FFT?
The point that with "pixel-wise" you mean "differently for each pixel" indicates that a frequency-domain transform is a really bad approach for this. The DFT of a single element is that element (scaled, if at all modified), so you can't win with this.

You can use the FFT and similar algorithms to do elegant shifts, but the fact that it's used in light-field reconstruction stems from the fact that there's a certain, mathematically elegant model of what should happen to many of those pixels that relates nicely to frequency domain.

If you just want to shift/interpolate/distort your image pixel-wise, the normal distortion matrices are the way to go – ask google about that. 

For example, OpenCV has a very short tutorial on camera calibration; maybe the theory contained in there is something for you?