How to perform 2D image rotation by FFT based image translation using matlab?
The following is the completed code that yields the expected result:

close all; clear all; clc;

I = double(imread('Cameraman.tif'))/255;
figure,imshow(I)

[M, N, s] = size (I); 

temp=ones(2*M,2*N);
temp(floor(M/2):floor(M/2)+M-1,floor(N/2):floor(N/2)+N-1)=I;
I=temp; clear temp;

figure,imshow(I)
[M N s]=size(I)
teta = pi/3;
a = tan(teta/2)
b = -sin(teta)

 Nx = ifftshift(-fix(M/2):ceil(M/2)-1); 
 Ny = ifftshift(-fix(N/2):ceil(N/2)-1);
 Ix=zeros(length(Nx),length(Ny));
 for k=1:M
   Ix(k,:)=real(ifft(fft(I(k,:)).*exp(-2*i*pi*(k-floor(M/2))*Ny*a/N)));
 end

 Iy = zeros(size(Ix));
 for k=1:M
   Iy(:,k)=real(ifft(fft(Ix(:,k)).*exp(2*i*pi*(k-floor(M/2))*Ny*b/N)'));
 end

 If = zeros(size(Iy));
 for k=1:M
   If(k,:)=real(ifft(fft(Iy(k,:)).*exp(-2*i*pi*(k-floor(M/2))*Ny*a/N)));
 end

 figure,imshow(Ix);
 figure,imshow(Iy);
 figure,imshow(If);


So this works as expected from a 2D rotation about z-axis. The paper argues that this is a fast and accurate method, but I doubt that, as a simple 2D rotation could easily be implemented using raw trigonometric computations and a bilinear interpolation (or bicubic if a sharper result is desired) other than forward and inverse FFTs.

I can say that the method is slow (on matlab, it can be further vectorized however). On the other hand the perceptual quality of the resulting rotated image is fairly good except some little ringing artefacts towards the edges.

I suggest you the ch-5 of Principles of Computer Graphics 2E by FOLEY for further understanding of the mathematics used in these algorithms and applications.