API: inconsistency in casting back to its EA dtype (try_cast_to_ea)
With the current logic we have in `try_cast_to_ea`:

https://github.com/pandas-dev/pandas/blob/a72eef5a86979232337149961d2122f0b555c876/pandas/core/arrays/base.py#L49-L54

it's easy to get inconsistencies, and it depends critically on what `_from_sequence` accepts as valid scalar. 

For example, we now have this

```
>>> s = pd.Series([0, 1, 2], dtype="Int64") 
>>> s.combine(0, lambda x, y: x == y)      
0     True
1    False
2    False
dtype: bool
```

However, if the IntegerArray constructor gets changed slightly to be more willing to accept all kinds of boolean like values (see https://github.com/pandas-dev/pandas/issues/31104 for a current inconsistency on this aspect, and this is what happens in https://github.com/pandas-dev/pandas/pull/30282), you can get:

```
>>> s = pd.Series([0, 1, 2], dtype="Int64")  
>>> s.combine(0, lambda x, y: x == y)       
0    1
1    0
2    0
dtype: Int64
```

So how "forgiving" the EA constructor is, will determine the output type. 

Here the example is with `combine`, but the `try_cast_to_ea` function is also used in a part of `groupby`.