Issue title: Inference error: shape mismatch
 Issue body: Hi, I was testing `live_asr.py` on my macOS Monterey (Python=3.8.11) under the following environment:
```
halo==0.0.31
numpy==1.21.4
PyAudio==0.2.11
Rx==3.2.0
SoundFile==0.10.3.post1
torch==1.10.0
torchaudio==0.10.0
transformers==4.8.2
webrtcvad==2.0.10
```

When I run `python live_asr.py`, I came across errors as below:
```bash
Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-large-960h-lv60-self and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

listening to your voice

/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/transformers/feature_extraction_utils.py:158: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
  tensor = as_tensor(value)
/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:986: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  return (input_length - kernel_size) // stride + 1
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/jkang/Desktop/test_asr/wav2vec2-live/live_asr.py", line 92, in asr_process
    text = wave2vec_asr.buffer_to_text(float64_buffer).lower()
  File "/Users/jkang/Desktop/test_asr/wav2vec2-live/wav2vec2_inference.py", line 22, in buffer_to_text
    logits = self.model(inputs.input_values, attention_mask=torch.ones(len(inputs.input_values[0]))).logits
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1120, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1528, in forward
    return_dict=return_dict,
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1120, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1171, in forward
    return_dict=return_dict,
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 791, in forward
    hidden_states[~attention_mask] = 0
IndexError: The shape of the mask [1920, 5] at index 0 does not match the shape of the indexed tensor [1, 5, 1024] at index 0

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/jkang/Desktop/test_asr/wav2vec2-live/live_asr.py", line 68, in vad_process
    frame = stream.read(CHUNK)
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/pyaudio.py", line 608, in read
    return pa.read_stream(self._stream, num_frames, exception_on_overflow)
OSError: [Errno -9981] Input overflowed
```

I think the critical issue is this:
```bash
IndexError: The shape of the mask [1920, 5] at index 0 does not match the shape of the indexed tensor [1, 5, 1024] at index 0
```

I installed transformers==4.8.2, but the error occurs which is probably not related to the transformers' version in my guess. 

Could you help me with what caused this error?

Thank you
 Comments: 
 Comment 0: Which model are you using?
 Comment 1: I am having the same issue. I am using the model "facebook/wav2vec2-large-960h-lv60-self".
 Comment 2: Thank you for the feedback. I fixed bug #2 and was able to run the model with the latest dependencies installed. 

Could you please check if it works for you?
 Comment 3: Yes, it works! Thank you for taking your time to fix it @oliverguhr.

<img width="969" alt="ex" src="https://user-images.githubusercontent.com/18749927/143400214-8dbb4df6-5b60-41f4-be9b-65c5a1fee228.png">
 Comment 4: Great - you are welcome.
{'source': 'GithubIssues', 'repo_name_with_owner': 'oliverguhr/wav2vec2-live', 'issue_id': 1060963193, 'repo_stars': 167.0}