BUG: byteorder (endian) error for int64 and int32
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.

#### Code Sample, a copy-pastable example

```python
import pandas as pd

values = {'int64': pd.Series([0], dtype='>i8'),
          'int32': pd.Series([1], dtype='>i4'),
          'int64_2': pd.Series([2], dtype='>i8')}
df = pd.DataFrame(values)
print("series", [(v.dtype, v.dtype.byteorder) for k, v in values.items()])
print("df    ", [(df[c].dtype, df[c].dtype.byteorder) for c in df.columns])

```

- The OS is Macos Catalina, 10.15.7.
- python_version(): 3.9.5
- pandas version:  1.3.1

#### Problem description

We specify the `byteorder`(endian) as `>` of `Series`, but when we construct `DataFrame`  by `Series`, the `DataFrame`'s `byteorder` gets changed.

#### Expected Output

The expected output is, the byteorders of three columns are both '>'. 
But the actual output is as follows:
```python
series [(dtype('>i8'), '>'), (dtype('>i4'), '>'), (dtype('>i8'), '>')]
df     [(dtype('int64'), '='), (dtype('>i4'), '>'), (dtype('int64'), '=')]
```
