What is the physical interpretation of Lowpass/Highpass filtering?
The example you gave of taking 4 samples and taking the average of it is sort of a poor-man's low-pass filter. Generally things aren't as simple as that. But for understanding sake there is some value in using these simple examples.

A low pass filter is indeed like taking 4 samples and taking an average of it. Ex:

samples = [6 1 -10 -4];
avg_value = mean(samples) = -1.75


The high pass filter is removing the "DC" term. Or more generally, it is removing the data that isn't changing. A simple way of thinking of this is to subtract your avg_value from every sample. Ex:

samples = [6 1 -10 -4];
avg_value = mean(samples) = -1.75;
high_pass = samples-avg_value;
high_pass: [7.75 2.75 -8.25 -2.25]


Now if you take the average of the "high passed" signal you end up getting 0.



These two 'filters' that I mention are both to the extreme in that you have one filter that only gives you DC and another filter that only removes DC. Basically what you end up getting is this ideal filter where the low-pass filter gives you the green and the high pass filter gives you the yellow.



Most of the filters you will use will have a response that looks more like this for a low pass:



and this for a high pass: