ALL PASS Filter-Why the name filter
It is designed using filter design techniques and implemented using filter components. Though it has ideally no effect on the signal amplitudes (hence all-pass) it affects phase, or group delay. 

Combining an all-pass filtered signal with its original input will modify the amplitude response - e.g. where gain = 1 and phase = 180 degrees, adding input and output will produce a notch in the frequency response, so you could argue that the frequency selective aspect of a filter is latent, even if not directly visible.

More importantly, once you have designed a low pass filter with specific characteristics, there are techniques to transform it into a high-pass, bandpass, bandstop, or all-pass network with closely related properties. At which point it appears logical and consistent to label the all-pass network a "filter" like the others, even though it appears not to be frequency selective.

However if you are still not convinced, "all-pass network" is probably an acceptable alternative term in most uses.