Determine type of filtering applied between two images I am not too familiar with DSP, and would be grateful for a pointer in the right direction.

Given two images A and B, is it possible to determine the filtering applied, and thereby apply it again to other candidate images?

For example, if you have an image of a triangle, and the second image is of that triangle rotated 90 degrees. Or the first image is a square, and the second image is a pentagon. Is it possible to determine the filter applied?
You can use Discrete Fourier transform. One of it's interesting qualities is that convolution in the time domain is translated into multiplication in the frequency domain. 

Applying a filter, represented by the function $h(t)$, on a signal\image $x(t)$ is equivalent, in fact, to convolving the two in the time domain. You can learn more about convolution here. We will denote the output of the operation $y(t)$, and it will be represented by $y(t)=x(t)*h(t)$. Note that $*$ in this context is a convolution operation.

If we will apply a Discrete Fourier transform $\mathcal F$ on both sides we get: 
$$\mathcal F[y(t)]=\mathcal F[x(t)*h(t)]=\mathcal F[x(t)]\cdot \mathcal F[h(t)]$$
Where the $\cdot$ denotes multiplication. This is again derived from the fact that convolution in the time domain is translated into multiplication in the frequency domain.

You can now use simple math to get:
$$\mathcal F[h(t)]=\frac{\mathcal F[y(t)]}{\mathcal F[x(t)]}$$
And hence:
$$h(t)=\mathcal F^{-1}\left[\frac{\mathcal F[y(t)]}{\mathcal F[x(t)]}\right]$$ where $\mathcal F^{-1}$ represent the inverse of DFT.

You can now use $h(t)$ for any new $x'(t)$ by applying:
$$y'(t)=x'(t)*h(t)\implies \mathcal F[y'(t)]=\mathcal F[x'(t)]\cdot \mathcal F[h(t)]$$

Sure DFT is a bit more complex on an image and you will have to learn how to apply it, but the principle is the same.