nearest neighbor implementation matlab
$D$ is $m \times m$, where $m$ is the total number of time frames in the spectrogram, it represents the squared euclidean distance of each frame to each other frame. In Matlab you could calculate it like this:

% Equation 10
for k = 1:m
    for l = 1:m
      D(k,l) = sum((X(:,k)-X(:,l)).^2);
    end
end


This is clearer from the text than the equation itself. 

The $P$ matrix is calculated for each frame from the $p$ the nearest frames to it. The $Y$ matrix is then the median of $P$ at each frame. In matlab you could calculate it like this. 

[~,I] = sort(D,2);
for k = 1:m
   pNearest = I(k,1:p);
   P = X(:,pNearest);
   Y(k,:) = median(P,2); 
end