Spatial and temporal frequency of a wave
The two frequencies you are referring to are the spatial frequency and temporal frequency of the wave, and you are correct in your reasoning on converting one to the other. The spatial frequency refers to how many complete periods the signal goes through for a given unit of distance (eg. cylcles/m) while the temporal frequency refers to how many complete periods the signal goes through for a given unit of time (eg. cyles/s or Hz).

The physical meaning of the two frequencies should be fairly easy to understand if you imagine the wave you are sampling is a simple sine wave. Then it is easy to see that for any constant point in time, your signal will be a frozen sin wave which is periodic in space, and thus has a "spatial frequency". Conversely, for any constant point in space, the value of the signal will oscillate over a regular period in time, and thus have a "temporal frequency" component. You can also imagine that the faster the wave was travelling through space, the faster the signal would change at any one point in space, giving rise to a higher temporal frequency and underlining the relationship between the two properties via the wave's speed.

Regarding using all the sample points to estimate the wave frequency, I believe you should be able to use the known speed to scale one of your dimensions to match the other, since $distance=velocity*time$. Then a 2D FFT of the matrix should give you information based on all of the available samples in whichever dimension you choose.

This article gives a brief description of the relationship between the two in the context of radar imaging and this article talks about the use of the 2D FFT for analyzing higher dimensional signals.