BUG: Categorical data is not correctly plotted with its associated value when using plt.plot.
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

### Bug report

**Bug summary**

- If a column is set as categorical with [`pd.Categorical`][1], the categorical column is not correctly plotted with the associated value, when using `plt.plot`, as shown below.
- This issue was reported here [Why is a column of categorical data not correctly plotted with its associated values?][2]
- I tweeted the issue @matplotlib and the response was _Ack, please open an issue! This looks like it might be a bug in our **units/categorical machinery** Confused face_
- A workaround is to sort the `label` column with `df.sort_values('label', inplace=True)`, which leads to a correct plot when using `plt.plot(df.label, df.value, "o")`.
- The bug was originally posted to [matplotlib][3].


**Code for reproduction**

```python
import pandas as pd
import matplotlib.pyplot as plt

labels = ["b", "b", "a", "b", "a"]
values = [10, 11, 1, 12, 2]
df = pd.DataFrame([{"label": label, "value": v} for label, v in zip(labels, values)])
df.label = pd.Categorical(df.label, categories=["a", "b"], ordered=True)

# display(df)
  label  value
0     b     10
1     b     11
2     a      1
3     b     12
4     a      2

# plot
plt.plot(df.label, df.value, "o")
```
**Actual outcome**

![image](https://user-images.githubusercontent.com/12875240/88105677-21557d00-cb59-11ea-8344-d145ee9c091e.png)

**Expected outcome**

```python
plt.scatter(df.label, df.value)
```
![image](https://user-images.githubusercontent.com/12875240/88105856-6aa5cc80-cb59-11ea-82db-579b799cdc66.png)

**Matplotlib version**
<!--Please specify your platform and versions of the relevant libraries you are using:-->
  * Operating system: Windows 10 64-bit
  * Matplotlib version: 3.2.2
  * Matplotlib backend (`print(matplotlib.get_backend())`): `module://ipykernel.pylab.backend_inline`
  * Python version: 3.7.7
  * Jupyter Lab version (if applicable): 2.1.5
- matplotlib was installed with Anaconda and updated way `conda update conda` & `conda update --all`
- I don't know these details for the person who reported the issue on stack overflow


  [1]: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Categorical.html#pandas-categorical
  [2]: https://stackoverflow.com/questions/63013321
  [3]: https://github.com/matplotlib/matplotlib/issues/18005
