Filtering the EEG signals
It is one way to do it, yes. You will get frequencies between 8 an 12 Hz less attenuated compared to others, but both the 8 and 12 Hz components will be attenuated 3 dB ($\frac{1}{\sqrt{2}}$ the original magnitude) and frequencies just outside the band will be attenuated only slightly more.

The way depends on how much you would like to attenuate and so on.

For a proper filter design you would like to specify a passband (between 8 and 12 in your case), a stopband (where the filter should attenuate according to your specification, there must be a transition band), maximum allowed attenuation in the passband and minimum required attenuation in the stopband. The exact numbers can be hard to determine, but once you have some you can rather straightforwardly input them into any appropriate filter design method.

However, you should obtain some filtering using the FFmpeg approach, although there may be artifacts which you did not expect/want.