BUG/API: Series[object].fillna ignores downcast=None, "infer"
### 

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the [latest version](https://pandas.pydata.org/docs/whatsnew/index.html) of pandas.

- [X] I have confirmed this bug exists on the master branch of pandas.


### Reproducible Example

```python
ser = pd.Series([True, None, False, True, None], dtype=object)
>>> ser.fillna(True, downcast=None)
0     True
1     True
2    False
3     True
4     True
dtype: bool

>>> ser.fillna(True, downcast="infer")
0     True
1     True
2    False
3     True
4     True
dtype: object
```


### Issue Description

Note that downcast=None is the default, so this also happens with just `ser.fillna(True)`.

Disabling the special-casing for object dtype in Block._maybe_downcast breaks 9 tests.  I need to check, think there may be some EA cases that aren't tested but where behavior would change if removed that special-casing.

### Expected Behavior

I'd expect it to go the other way, with object dtype retained with downcast=None and bool inferred with downcast="infer"

### Installed Versions

<details>

Replace this line with the output of pd.show_versions()

</details>
