Computing the sum of squared differences
Here's a way to do it using pdist to generate both the differences in value, plus logical indices that you can use to select which distances you wish to look at/use further.

D1 = [1 2 3;4 5 6;7 8 9];
D2 = [9 8 7;6 5 4;3 2 1];
D = cat(3,D1,D2);

D(:,:,1) =

 1     2     3
 4     5     6
 7     8     9

D(:,:,2) =

 9     8     7
 6     5     4
 3     2     1

Y = repmat([1 2 3]', [1 3 2]); %# (' in comment to fix SO highlighting)
X = repmat([1 2 3],[3 1 2]);
T = repmat(cat(3,1,2),[3 3 1]);
D_diffsqrd = pdist(D(:),'euclidean').^2;
X_dist = pdist(X(:),'euclidean');
Y_dist = pdist(Y(:),'euclidean');
T_dist = pdist(T(:),'euclidean');
Angle_dist = pdist([X(:) Y(:)],'euclidean');

D_diffsqrd(X_dist==2 &amp; T_dist==0 &amp; Y_dist==0)
    4     4     4     4     4     4

D_diffsqrd(X_dist==0 &amp; T_dist==0 &amp; Y_dist==2)
    36    36    36    36    36    36

D_diffsqrd(X_dist==0 &amp; T_dist==1 &amp; Y_dist==0)
    64     4    16    36     0    36    16     4    64