# Version 0.21.0 (October 27, 2017)¶

This is a major release from 0.20.3 and includes a number of API changes, deprecations, new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version.

Highlights include:

• Integration with Apache Parquet, including a new top-level `read_parquet()` function and `DataFrame.to_parquet()` method, see here.

• New user-facing `pandas.api.types.CategoricalDtype` for specifying categoricals independent of the data, see here.

• The behavior of `sum` and `prod` on all-NaN Series/DataFrames is now consistent and no longer depends on whether bottleneck is installed, and `sum` and `prod` on empty Series now return NaN instead of 0, see here.

• Compatibility fixes for pypy, see here.

• Additions to the `drop`, `reindex` and `rename` API to make them more consistent, see here.

• Addition of the new methods `DataFrame.infer_objects` (see here) and `GroupBy.pipe` (see here).

• Indexing with a list of labels, where one or more of the labels is missing, is deprecated and will raise a KeyError in a future version, see here.

Check the API Changes and deprecations before updating.

## New features¶

### Integration with Apache Parquet file format¶

Integration with Apache Parquet, including a new top-level `read_parquet()` and `DataFrame.to_parquet()` method, see here (GH15838, GH17438).

Apache Parquet provides a cross-language, binary file format for reading and writing data frames efficiently. Parquet is designed to faithfully serialize and de-serialize `DataFrame` s, supporting all of the pandas dtypes, including extension dtypes such as datetime with timezones.

This functionality depends on either the pyarrow or fastparquet library. For more details, see the IO docs on Parquet.

### Method `infer_objects` type conversion¶

The `DataFrame.infer_objects()` and `Series.infer_objects()` methods have been added to perform dtype inference on object columns, replacing some of the functionality of the deprecated `convert_objects` method. See the documentation here for more details. (GH11221)

This method only performs soft conversions on object columns, converting Python objects to native types, but not any coercive conversions. For example:

```In : df = pd.DataFrame({'A': [1, 2, 3],
...:                    'B': np.array([1, 2, 3], dtype='object'),
...:                    'C': ['1', '2', '3']})
...:

In : df.dtypes
Out:
A     int64
B    object
C    object
Length: 3, dtype: object

In : df.infer_objects().dtypes
Out:
A     int64
B     int64
C    object
Length: 3, dtype: object
```

Note that column `'C'` was not converted - only scalar numeric types will be converted to a new type. Other types of conversion should be accomplished using the `to_numeric()` function (or `to_datetime()`, `to_timedelta()`).

```In : df = df.infer_objects()

In : df['C'] = pd.to_numeric(df['C'], errors='coerce')

In : df.dtypes
Out:
A    int64
B    int64
C    int64
Length: 3, dtype: object
```

### Improved warnings when attempting to create columns¶

New users are often puzzled by the relationship between column operations and attribute access on `DataFrame` instances (GH7175). One specific instance of this confusion is attempting to create a new column by setting an attribute on the `DataFrame`:

```In : df = pd.DataFrame({'one': [1., 2., 3.]})
In : df.two = [4, 5, 6]
```

This does not raise any obvious exceptions, but also does not create a new column:

```In : df
Out:
one
0  1.0
1  2.0
2  3.0
```

Setting a list-like data structure into a new attribute now raises a `UserWarning` about the potential for unexpected behavior. See Attribute Access.

### Method `drop` now also accepts index/columns keywords¶

The `drop()` method has gained `index`/`columns` keywords as an alternative to specifying the `axis`. This is similar to the behavior of `reindex` (GH12392).

For example:

```In : df = pd.DataFrame(np.arange(8).reshape(2, 4),
...:                   columns=['A', 'B', 'C', 'D'])
...:

In : df
Out:
A  B  C  D
0  0  1  2  3
1  4  5  6  7

[2 rows x 4 columns]

In : df.drop(['B', 'C'], axis=1)
Out:
A  D
0  0  3
1  4  7

[2 rows x 2 columns]

# the following is now equivalent
In : df.drop(columns=['B', 'C'])
Out:
A  D
0  0  3
1  4  7

[2 rows x 2 columns]
```

### Methods `rename`, `reindex` now also accept axis keyword¶

The `DataFrame.rename()` and `DataFrame.reindex()` methods have gained the `axis` keyword to specify the axis to target with the operation (GH12392).

Here’s `rename`:

```In : df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})

In : df.rename(str.lower, axis='columns')
Out:
a  b
0  1  4
1  2  5
2  3  6

[3 rows x 2 columns]

In : df.rename(id, axis='index')
Out:
A  B
94636016352480  1  4
94636016352512  2  5
94636016352544  3  6

[3 rows x 2 columns]
```

And `reindex`:

```In : df.reindex(['A', 'B', 'C'], axis='columns')
Out:
A  B   C
0  1  4 NaN
1  2  5 NaN
2  3  6 NaN

[3 rows x 3 columns]

In : df.reindex([0, 1, 3], axis='index')
Out:
A    B
0  1.0  4.0
1  2.0  5.0
3  NaN  NaN

[3 rows x 2 columns]
```

The “index, columns” style continues to work as before.

```In : df.rename(index=id, columns=str.lower)
Out:
a  b
94636016352480  1  4
94636016352512  2  5
94636016352544  3  6

[3 rows x 2 columns]

In : df.reindex(index=[0, 1, 3], columns=['A', 'B', 'C'])
Out:
A    B   C
0  1.0  4.0 NaN
1  2.0  5.0 NaN
3  NaN  NaN NaN

[3 rows x 3 columns]
```

We highly encourage using named arguments to avoid confusion when using either style.

### `CategoricalDtype` for specifying categoricals¶

`pandas.api.types.CategoricalDtype` has been added to the public API and expanded to include the `categories` and `ordered` attributes. A `CategoricalDtype` can be used to specify the set of categories and orderedness of an array, independent of the data. This can be useful for example, when converting string data to a `Categorical` (GH14711, GH15078, GH16015, GH17643):

```In : from pandas.api.types import CategoricalDtype

In : s = pd.Series(['a', 'b', 'c', 'a'])  # strings

In : dtype = CategoricalDtype(categories=['a', 'b', 'c', 'd'], ordered=True)

In : s.astype(dtype)
Out:
0    a
1    b
2    c
3    a
Length: 4, dtype: category
Categories (4, object): ['a' < 'b' < 'c' < 'd']
```

One place that deserves special mention is in `read_csv()`. Previously, with `dtype={'col': 'category'}`, the returned values and categories would always be strings.

```In : data = 'A,B\na,1\nb,2\nc,3'

Out: Index(['1', '2', '3'], dtype='object')
```

Notice the “object” dtype.

With a `CategoricalDtype` of all numerics, datetimes, or timedeltas, we can automatically convert to the correct type

```In : dtype = {'B': CategoricalDtype([1, 2, 3])}

Out: Int64Index([1, 2, 3], dtype='int64')
```

The values have been correctly interpreted as integers.

The `.dtype` property of a `Categorical`, `CategoricalIndex` or a `Series` with categorical type will now return an instance of `CategoricalDtype`. While the repr has changed, `str(CategoricalDtype())` is still the string `'category'`. We’ll take this moment to remind users that the preferred way to detect categorical data is to use `pandas.api.types.is_categorical_dtype()`, and not `str(dtype) == 'category'`.

See the CategoricalDtype docs for more.

### `GroupBy` objects now have a `pipe` method¶

`GroupBy` objects now have a `pipe` method, similar to the one on `DataFrame` and `Series`, that allow for functions that take a `GroupBy` to be composed in a clean, readable syntax. (GH17871)

For a concrete example on combining `.groupby` and `.pipe` , imagine having a DataFrame with columns for stores, products, revenue and sold quantity. We’d like to do a groupwise calculation of prices (i.e. revenue/quantity) per store and per product. We could do this in a multi-step operation, but expressing it in terms of piping can make the code more readable.

First we set the data:

```In : import numpy as np

In : n = 1000

In : df = pd.DataFrame({'Store': np.random.choice(['Store_1', 'Store_2'], n),
....:                    'Product': np.random.choice(['Product_1',
....:                                                 'Product_2',
....:                                                 'Product_3'
....:                                                 ], n),
....:                    'Revenue': (np.random.random(n) * 50 + 10).round(2),
....:                    'Quantity': np.random.randint(1, 10, size=n)})
....:

Out:
Store    Product  Revenue  Quantity
0  Store_2  Product_2    32.09         7
1  Store_1  Product_3    14.20         1

[2 rows x 4 columns]
```

Now, to find prices per store/product, we can simply do:

```In : (df.groupby(['Store', 'Product'])
....:    .pipe(lambda grp: grp.Revenue.sum() / grp.Quantity.sum())
....:    .unstack().round(2))
....:
Out:
Product  Product_1  Product_2  Product_3
Store
Store_1       6.73       6.72       7.14
Store_2       7.59       6.98       7.23

[2 rows x 3 columns]
```

See the documentation for more.

### `Categorical.rename_categories` accepts a dict-like¶

`rename_categories()` now accepts a dict-like argument for `new_categories`. The previous categories are looked up in the dictionary’s keys and replaced if found. The behavior of missing and extra keys is the same as in `DataFrame.rename()`.

```In : c = pd.Categorical(['a', 'a', 'b'])

In : c.rename_categories({"a": "eh", "b": "bee"})
Out:
['eh', 'eh', 'bee']
Categories (2, object): ['eh', 'bee']
```

Warning

To assist with upgrading pandas, `rename_categories` treats `Series` as list-like. Typically, Series are considered to be dict-like (e.g. in `.rename`, `.map`). In a future version of pandas `rename_categories` will change to treat them as dict-like. Follow the warning message’s recommendations for writing future-proof code.

```In : c.rename_categories(pd.Series([0, 1], index=['a', 'c']))
FutureWarning: Treating Series 'new_categories' as a list-like and using the values.
In a future version, 'rename_categories' will treat Series like a dictionary.
For dict-like, use 'new_categories.to_dict()'
For list-like, use 'new_categories.values'.
Out:
[0, 0, 1]
Categories (2, int64): [0, 1]
```

## Backwards incompatible API changes¶

### Dependencies have increased minimum versions¶

We have updated our minimum supported versions of dependencies (GH15206, GH15543, GH15214). If installed, we now require:

Package

Minimum Version

Required

Numpy

1.9.0

X

Matplotlib

1.4.3

Scipy

0.14.0

Bottleneck

1.0.0

Additionally, support has been dropped for Python 3.4 (GH15251).

### Sum/prod of all-NaN or empty Series/DataFrames is now consistently NaN¶

Note

The changes described here have been partially reverted. See the v0.22.0 Whatsnew for more.

The behavior of `sum` and `prod` on all-NaN Series/DataFrames no longer depends on whether bottleneck is installed, and return value of `sum` and `prod` on an empty Series has changed (GH9422, GH15507).

Calling `sum` or `prod` on an empty or all-`NaN` `Series`, or columns of a `DataFrame`, will result in `NaN`. See the docs.

```In : s = pd.Series([np.nan])
```

Previously WITHOUT `bottleneck` installed:

```In : s.sum()
Out: np.nan
```

Previously WITH `bottleneck`:

```In : s.sum()
Out: 0.0
```

New behavior, without regard to the bottleneck installation:

```In : s.sum()
Out: 0.0
```

Note that this also changes the sum of an empty `Series`. Previously this always returned 0 regardless of a `bottleneck` installation:

```In : pd.Series([]).sum()
Out: 0
```

but for consistency with the all-NaN case, this was changed to return NaN as well:

```In : pd.Series([]).sum()
Out: 0.0
```

### Indexing with a list with missing labels is deprecated¶

Previously, selecting with a list of labels, where one or more labels were missing would always succeed, returning `NaN` for missing labels. This will now show a `FutureWarning`. In the future this will raise a `KeyError` (GH15747). This warning will trigger on a `DataFrame` or a `Series` for using `.loc[]` or `[[]]` when passing a list-of-labels with at least 1 missing label. See the deprecation docs.

```In : s = pd.Series([1, 2, 3])

In : s
Out:
0    1
1    2
2    3
Length: 3, dtype: int64
```

Previous behavior

```In : s.loc[[1, 2, 3]]
Out:
1    2.0
2    3.0
3    NaN
dtype: float64
```

Current behavior

```In : s.loc[[1, 2, 3]]
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike

Out:
1    2.0
2    3.0
3    NaN
dtype: float64
```

The idiomatic way to achieve selecting potentially not-found elements is via `.reindex()`

```In : s.reindex([1, 2, 3])
Out:
1    2.0
2    3.0
3    NaN
Length: 3, dtype: float64
```

Selection with all keys found is unchanged.

```In : s.loc[[1, 2]]
Out:
1    2
2    3
Length: 2, dtype: int64
```

### NA naming changes¶

In order to promote more consistency among the pandas API, we have added additional top-level functions `isna()` and `notna()` that are aliases for `isnull()` and `notnull()`. The naming scheme is now more consistent with methods like `.dropna()` and `.fillna()`. Furthermore in all cases where `.isnull()` and `.notnull()` methods are defined, these have additional methods named `.isna()` and `.notna()`, these are included for classes `Categorical`, `Index`, `Series`, and `DataFrame`. (GH15001).

The configuration option `pd.options.mode.use_inf_as_null` is deprecated, and `pd.options.mode.use_inf_as_na` is added as a replacement.

### Iteration of Series/Index will now return Python scalars¶

Previously, when using certain iteration methods for a `Series` with dtype `int` or `float`, you would receive a `numpy` scalar, e.g. a `np.int64`, rather than a Python `int`. Issue (GH10904) corrected this for `Series.tolist()` and `list(Series)`. This change makes all iteration methods consistent, in particular, for `__iter__()` and `.map()`; note that this only affects int/float dtypes. (GH13236, GH13258, GH14216).

```In : s = pd.Series([1, 2, 3])

In : s
Out:
0    1
1    2
2    3
Length: 3, dtype: int64
```

Previously:

```In : type(list(s))
Out: numpy.int64
```

New behavior:

```In : type(list(s))
Out: int
```

Furthermore this will now correctly box the results of iteration for `DataFrame.to_dict()` as well.

```In : d = {'a': , 'b': ['b']}

In : df = pd.DataFrame(d)
```

Previously:

```In : type(df.to_dict()['a'])
Out: numpy.int64
```

New behavior:

```In : type(df.to_dict()['a'])
Out: int
```

### Indexing with a Boolean Index¶

Previously when passing a boolean `Index` to `.loc`, if the index of the `Series/DataFrame` had `boolean` labels, you would get a label based selection, potentially duplicating result labels, rather than a boolean indexing selection (where `True` selects elements), this was inconsistent how a boolean numpy array indexed. The new behavior is to act like a boolean numpy array indexer. (GH17738)

Previous behavior:

```In : s = pd.Series([1, 2, 3], index=[False, True, False])

In : s
Out:
False    1
True     2
False    3
Length: 3, dtype: int64
```
```In : s.loc[pd.Index([True, False, True])]
Out:
True     2
False    1
False    3
True     2
dtype: int64
```

Current behavior

```In : s.loc[pd.Index([True, False, True])]
Out:
False    1
False    3
Length: 2, dtype: int64
```

Furthermore, previously if you had an index that was non-numeric (e.g. strings), then a boolean Index would raise a `KeyError`. This will now be treated as a boolean indexer.

Previously behavior:

```In : s = pd.Series([1, 2, 3], index=['a', 'b', 'c'])

In : s
Out:
a    1
b    2
c    3
Length: 3, dtype: int64
```
```In : s.loc[pd.Index([True, False, True])]
KeyError: "None of [Index([True, False, True], dtype='object')] are in the [index]"
```

Current behavior

```In : s.loc[pd.Index([True, False, True])]
Out:
a    1
c    3
Length: 2, dtype: int64
```

### `PeriodIndex` resampling¶

In previous versions of pandas, resampling a `Series`/`DataFrame` indexed by a `PeriodIndex` returned a `DatetimeIndex` in some cases (GH12884). Resampling to a multiplied frequency now returns a `PeriodIndex` (GH15944). As a minor enhancement, resampling a `PeriodIndex` can now handle `NaT` values (GH13224)

Previous behavior:

```In : pi = pd.period_range('2017-01', periods=12, freq='M')

In : s = pd.Series(np.arange(12), index=pi)

In : resampled = s.resample('2Q').mean()

In : resampled
Out:
2017-03-31     1.0
2017-09-30     5.5
2018-03-31    10.0
Freq: 2Q-DEC, dtype: float64

In : resampled.index
Out: DatetimeIndex(['2017-03-31', '2017-09-30', '2018-03-31'], dtype='datetime64[ns]', freq='2Q-DEC')
```

New behavior:

```In : pi = pd.period_range('2017-01', periods=12, freq='M')

In : s = pd.Series(np.arange(12), index=pi)

In : resampled = s.resample('2Q').mean()

In : resampled
Out:
2017Q1    2.5
2017Q3    8.5
Freq: 2Q-DEC, Length: 2, dtype: float64

In : resampled.index
Out: PeriodIndex(['2017Q1', '2017Q3'], dtype='period[2Q-DEC]')
```

Upsampling and calling `.ohlc()` previously returned a `Series`, basically identical to calling `.asfreq()`. OHLC upsampling now returns a DataFrame with columns `open`, `high`, `low` and `close` (GH13083). This is consistent with downsampling and `DatetimeIndex` behavior.

Previous behavior:

```In : pi = pd.period_range(start='2000-01-01', freq='D', periods=10)

In : s = pd.Series(np.arange(10), index=pi)

In : s.resample('H').ohlc()
Out:
2000-01-01 00:00    0.0
...
2000-01-10 23:00    NaN
Freq: H, Length: 240, dtype: float64

In : s.resample('M').ohlc()
Out:
open  high  low  close
2000-01     0     9    0      9
```

New behavior:

```In : pi = pd.period_range(start='2000-01-01', freq='D', periods=10)

In : s = pd.Series(np.arange(10), index=pi)

In : s.resample('H').ohlc()
Out:
open  high  low  close
2000-01-01 00:00   0.0   0.0  0.0    0.0
2000-01-01 01:00   NaN   NaN  NaN    NaN
2000-01-01 02:00   NaN   NaN  NaN    NaN
2000-01-01 03:00   NaN   NaN  NaN    NaN
2000-01-01 04:00   NaN   NaN  NaN    NaN
...                ...   ...  ...    ...
2000-01-10 19:00   NaN   NaN  NaN    NaN
2000-01-10 20:00   NaN   NaN  NaN    NaN
2000-01-10 21:00   NaN   NaN  NaN    NaN
2000-01-10 22:00   NaN   NaN  NaN    NaN
2000-01-10 23:00   NaN   NaN  NaN    NaN

[240 rows x 4 columns]

In : s.resample('M').ohlc()
Out:
open  high  low  close
2000-01     0     9    0      9

[1 rows x 4 columns]
```

### Improved error handling during item assignment in pd.eval¶

`eval()` will now raise a `ValueError` when item assignment malfunctions, or inplace operations are specified, but there is no item assignment in the expression (GH16732)

```In : arr = np.array([1, 2, 3])
```

Previously, if you attempted the following expression, you would get a not very helpful error message:

```In : pd.eval("a = 1 + 2", target=arr, inplace=True)
...
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`)
and integer or boolean arrays are valid indices
```

This is a very long way of saying numpy arrays don’t support string-item indexing. With this change, the error message is now this:

```In : pd.eval("a = 1 + 2", target=arr, inplace=True)
...
ValueError: Cannot assign expression output to target
```

It also used to be possible to evaluate expressions inplace, even if there was no item assignment:

```In : pd.eval("1 + 2", target=arr, inplace=True)
Out: 3
```

However, this input does not make much sense because the output is not being assigned to the target. Now, a `ValueError` will be raised when such an input is passed in:

```In : pd.eval("1 + 2", target=arr, inplace=True)
...
ValueError: Cannot operate inplace if there is no assignment
```

### Dtype conversions¶

Previously assignments, `.where()` and `.fillna()` with a `bool` assignment, would coerce to same the type (e.g. int / float), or raise for datetimelikes. These will now preserve the bools with `object` dtypes. (GH16821).

```In : s = pd.Series([1, 2, 3])
```
```In : s = True

In : s
Out:
0    1
1    1
2    3
dtype: int64
```

New behavior

```In : s = True

In : s
Out:
0       1
1    True
2       3
Length: 3, dtype: object
```

Previously, as assignment to a datetimelike with a non-datetimelike would coerce the non-datetime-like item being assigned (GH14145).

```In : s = pd.Series([pd.Timestamp('2011-01-01'), pd.Timestamp('2012-01-01')])
```
```In : s = 1

In : s
Out:
0   2011-01-01 00:00:00.000000000
1   1970-01-01 00:00:00.000000001
dtype: datetime64[ns]
```

These now coerce to `object` dtype.

```In : s = 1

In : s
Out:
0    2011-01-01 00:00:00
1                      1
Length: 2, dtype: object
```
• Inconsistent behavior in `.where()` with datetimelikes which would raise rather than coerce to `object` (GH16402)

• Bug in assignment against `int64` data with `np.ndarray` with `float64` dtype may keep `int64` dtype (GH14001)

### MultiIndex constructor with a single level¶

The `MultiIndex` constructors no longer squeezes a MultiIndex with all length-one levels down to a regular `Index`. This affects all the `MultiIndex` constructors. (GH17178)

Previous behavior:

```In : pd.MultiIndex.from_tuples([('a',), ('b',)])
Out: Index(['a', 'b'], dtype='object')
```

Length 1 levels are no longer special-cased. They behave exactly as if you had length 2+ levels, so a `MultiIndex` is always returned from all of the `MultiIndex` constructors:

```In : pd.MultiIndex.from_tuples([('a',), ('b',)])
Out:
MultiIndex([('a',),
('b',)],
)
```

### UTC localization with Series¶

Previously, `to_datetime()` did not localize datetime `Series` data when `utc=True` was passed. Now, `to_datetime()` will correctly localize `Series` with a `datetime64[ns, UTC]` dtype to be consistent with how list-like and `Index` data are handled. (GH6415).

Previous behavior

```In : s = pd.Series(['20130101 00:00:00'] * 3)
```
```In : pd.to_datetime(s, utc=True)
Out:
0   2013-01-01
1   2013-01-01
2   2013-01-01
dtype: datetime64[ns]
```

New behavior

```In : pd.to_datetime(s, utc=True)
Out:
0   2013-01-01 00:00:00+00:00
1   2013-01-01 00:00:00+00:00
2   2013-01-01 00:00:00+00:00
Length: 3, dtype: datetime64[ns, UTC]
```

Additionally, DataFrames with datetime columns that were parsed by `read_sql_table()` and `read_sql_query()` will also be localized to UTC only if the original SQL columns were timezone aware datetime columns.

### Consistency of range functions¶

In previous versions, there were some inconsistencies between the various range functions: `date_range()`, `bdate_range()`, `period_range()`, `timedelta_range()`, and `interval_range()`. (GH17471).

One of the inconsistent behaviors occurred when the `start`, `end` and `period` parameters were all specified, potentially leading to ambiguous ranges. When all three parameters were passed, `interval_range` ignored the `period` parameter, `period_range` ignored the `end` parameter, and the other range functions raised. To promote consistency among the range functions, and avoid potentially ambiguous ranges, `interval_range` and `period_range` will now raise when all three parameters are passed.

Previous behavior:

``` In : pd.interval_range(start=0, end=4, periods=6)
Out:
IntervalIndex([(0, 1], (1, 2], (2, 3]]
closed='right',
dtype='interval[int64]')

In : pd.period_range(start='2017Q1', end='2017Q4', periods=6, freq='Q')
Out: PeriodIndex(['2017Q1', '2017Q2', '2017Q3', '2017Q4', '2018Q1', '2018Q2'], dtype='period[Q-DEC]', freq='Q-DEC')
```

New behavior:

```In : pd.interval_range(start=0, end=4, periods=6)
---------------------------------------------------------------------------
ValueError: Of the three parameters: start, end, and periods, exactly two must be specified

In : pd.period_range(start='2017Q1', end='2017Q4', periods=6, freq='Q')
---------------------------------------------------------------------------
ValueError: Of the three parameters: start, end, and periods, exactly two must be specified
```

Additionally, the endpoint parameter `end` was not included in the intervals produced by `interval_range`. However, all other range functions include `end` in their output. To promote consistency among the range functions, `interval_range` will now include `end` as the right endpoint of the final interval, except if `freq` is specified in a way which skips `end`.

Previous behavior:

```In : pd.interval_range(start=0, end=4)
Out:
IntervalIndex([(0, 1], (1, 2], (2, 3]]
closed='right',
dtype='interval[int64]')
```

New behavior:

```In : pd.interval_range(start=0, end=4)
Out: IntervalIndex([(0, 1], (1, 2], (2, 3], (3, 4]], dtype='interval[int64, right]')
```

### No automatic Matplotlib converters¶

pandas no longer registers our `date`, `time`, `datetime`, `datetime64`, and `Period` converters with matplotlib when pandas is imported. Matplotlib plot methods (`plt.plot`, `ax.plot`, …), will not nicely format the x-axis for `DatetimeIndex` or `PeriodIndex` values. You must explicitly register these methods:

pandas built-in `Series.plot` and `DataFrame.plot` will register these converters on first-use (GH17710).

Note

This change has been temporarily reverted in pandas 0.21.1, for more details see here.

### Other API changes¶

• The Categorical constructor no longer accepts a scalar for the `categories` keyword. (GH16022)

• Accessing a non-existent attribute on a closed `HDFStore` will now raise an `AttributeError` rather than a `ClosedFileError` (GH16301)

• `read_csv()` now issues a `UserWarning` if the `names` parameter contains duplicates (GH17095)

• `read_csv()` now treats `'null'` and `'n/a'` strings as missing values by default (GH16471, GH16078)

• `pandas.HDFStore`’s string representation is now faster and less detailed. For the previous behavior, use `pandas.HDFStore.info()`. (GH16503).

• Compression defaults in HDF stores now follow pytables standards. Default is no compression and if `complib` is missing and `complevel` > 0 `zlib` is used (GH15943)

• `Index.get_indexer_non_unique()` now returns a ndarray indexer rather than an `Index`; this is consistent with `Index.get_indexer()` (GH16819)

• Removed the `@slow` decorator from `pandas._testing`, which caused issues for some downstream packages’ test suites. Use `@pytest.mark.slow` instead, which achieves the same thing (GH16850)

• Moved definition of `MergeError` to the `pandas.errors` module.

• The signature of `Series.set_axis()` and `DataFrame.set_axis()` has been changed from `set_axis(axis, labels)` to `set_axis(labels, axis=0)`, for consistency with the rest of the API. The old signature is deprecated and will show a `FutureWarning` (GH14636)

• `Series.argmin()` and `Series.argmax()` will now raise a `TypeError` when used with `object` dtypes, instead of a `ValueError` (GH13595)

• `Period` is now immutable, and will now raise an `AttributeError` when a user tries to assign a new value to the `ordinal` or `freq` attributes (GH17116).

• `to_datetime()` when passed a tz-aware `origin=` kwarg will now raise a more informative `ValueError` rather than a `TypeError` (GH16842)

• `to_datetime()` now raises a `ValueError` when format includes `%W` or `%U` without also including day of the week and calendar year (GH16774)

• Renamed non-functional `index` to `index_col` in `read_stata()` to improve API consistency (GH16342)

• Bug in `DataFrame.drop()` caused boolean labels `False` and `True` to be treated as labels 0 and 1 respectively when dropping indices from a numeric index. This will now raise a ValueError (GH16877)

• Restricted DateOffset keyword arguments. Previously, `DateOffset` subclasses allowed arbitrary keyword arguments which could lead to unexpected behavior. Now, only valid arguments will be accepted. (GH17176).

## Deprecations¶

### Series.select and DataFrame.select¶

The `Series.select()` and `DataFrame.select()` methods are deprecated in favor of using `df.loc[labels.map(crit)]` (GH12401)

```In : df = pd.DataFrame({'A': [1, 2, 3]}, index=['foo', 'bar', 'baz'])
```
```In : df.select(lambda x: x in ['bar', 'baz'])
FutureWarning: select is deprecated and will be removed in a future release. You can use .loc[crit] as a replacement
Out:
A
bar  2
baz  3
```
```In : df.loc[df.index.map(lambda x: x in ['bar', 'baz'])]
Out:
A
bar  2
baz  3

[2 rows x 1 columns]
```

### Series.argmax and Series.argmin¶

The behavior of `Series.argmax()` and `Series.argmin()` have been deprecated in favor of `Series.idxmax()` and `Series.idxmin()`, respectively (GH16830).

For compatibility with NumPy arrays, `pd.Series` implements `argmax` and `argmin`. Since pandas 0.13.0, `argmax` has been an alias for `pandas.Series.idxmax()`, and `argmin` has been an alias for `pandas.Series.idxmin()`. They return the label of the maximum or minimum, rather than the position.

We’ve deprecated the current behavior of `Series.argmax` and `Series.argmin`. Using either of these will emit a `FutureWarning`. Use `Series.idxmax()` if you want the label of the maximum. Use `Series.values.argmax()` if you want the position of the maximum. Likewise for the minimum. In a future release `Series.argmax` and `Series.argmin` will return the position of the maximum or minimum.

## Removal of prior version deprecations/changes¶

• `read_excel()` has dropped the `has_index_names` parameter (GH10967)

• The `pd.options.display.height` configuration has been dropped (GH3663)

• The `pd.options.display.line_width` configuration has been dropped (GH2881)

• The `pd.options.display.mpl_style` configuration has been dropped (GH12190)

• `Index` has dropped the `.sym_diff()` method in favor of `.symmetric_difference()` (GH12591)

• `Categorical` has dropped the `.order()` and `.sort()` methods in favor of `.sort_values()` (GH12882)

• `eval()` and `DataFrame.eval()` have changed the default of `inplace` from `None` to `False` (GH11149)

• The function `get_offset_name` has been dropped in favor of the `.freqstr` attribute for an offset (GH11834)

• pandas no longer tests for compatibility with hdf5-files created with pandas < 0.11 (GH17404).

## Documentation changes¶

• Several `NaT` method docstrings (e.g. `NaT.ctime()`) were incorrect (GH17327)

• The documentation has had references to versions < v0.17 removed and cleaned up (GH17442, GH17442, GH17404 & GH17504)

## Bug fixes¶

### Conversion¶

• Bug in assignment against datetime-like data with `int` may incorrectly convert to datetime-like (GH14145)

• Bug in assignment against `int64` data with `np.ndarray` with `float64` dtype may keep `int64` dtype (GH14001)

• Fixed the return type of `IntervalIndex.is_non_overlapping_monotonic` to be a Python `bool` for consistency with similar attributes/methods. Previously returned a `numpy.bool_`. (GH17237)

• Bug in `IntervalIndex.is_non_overlapping_monotonic` when intervals are closed on both sides and overlap at a point (GH16560)

• Bug in `Series.fillna()` returns frame when `inplace=True` and `value` is dict (GH16156)

• Bug in `Timestamp.weekday_name` returning a UTC-based weekday name when localized to a timezone (GH17354)

• Bug in `Timestamp.replace` when replacing `tzinfo` around DST changes (GH15683)

• Bug in `Timedelta` construction and arithmetic that would not propagate the `Overflow` exception (GH17367)

• Bug in `astype()` converting to object dtype when passed extension type classes (`DatetimeTZDtype`, `CategoricalDtype`) rather than instances. Now a `TypeError` is raised when a class is passed (GH17780).

• Bug in `to_numeric()` in which elements were not always being coerced to numeric when `errors='coerce'` (GH17007, GH17125)

• Bug in `DataFrame` and `Series` constructors where `range` objects are converted to `int32` dtype on Windows instead of `int64` (GH16804)

### Indexing¶

• When called with a null slice (e.g. `df.iloc[:]`), the `.iloc` and `.loc` indexers return a shallow copy of the original object. Previously they returned the original object. (GH13873).

• When called on an unsorted `MultiIndex`, the `loc` indexer now will raise `UnsortedIndexError` only if proper slicing is used on non-sorted levels (GH16734).

• Fixes regression in 0.20.3 when indexing with a string on a `TimedeltaIndex` (GH16896).

• Fixed `TimedeltaIndex.get_loc()` handling of `np.timedelta64` inputs (GH16909).

• Fix `MultiIndex.sort_index()` ordering when `ascending` argument is a list, but not all levels are specified, or are in a different order (GH16934).

• Fixes bug where indexing with `np.inf` caused an `OverflowError` to be raised (GH16957)

• Bug in reindexing on an empty `CategoricalIndex` (GH16770)

• Fixes `DataFrame.loc` for setting with alignment and tz-aware `DatetimeIndex` (GH16889)

• Avoids `IndexError` when passing an Index or Series to `.iloc` with older numpy (GH17193)

• Allow unicode empty strings as placeholders in multilevel columns in Python 2 (GH17099)

• Bug in `.iloc` when used with inplace addition or assignment and an int indexer on a `MultiIndex` causing the wrong indexes to be read from and written to (GH17148)

• Bug in `.isin()` in which checking membership in empty `Series` objects raised an error (GH16991)

• Bug in `CategoricalIndex` reindexing in which specified indices containing duplicates were not being respected (GH17323)

• Bug in intersection of `RangeIndex` with negative step (GH17296)

• Bug in `IntervalIndex` where performing a scalar lookup fails for included right endpoints of non-overlapping monotonic decreasing indexes (GH16417, GH17271)

• Bug in `DataFrame.first_valid_index()` and `DataFrame.last_valid_index()` when no valid entry (GH17400)

• Bug in `Series.rename()` when called with a callable, incorrectly alters the name of the `Series`, rather than the name of the `Index`. (GH17407)

• Bug in `String.str_get()` raises `IndexError` instead of inserting NaNs when using a negative index. (GH17704)

### Plotting¶

• Bug in plotting methods using `secondary_y` and `fontsize` not setting secondary axis font size (GH12565)

• Bug when plotting `timedelta` and `datetime` dtypes on y-axis (GH16953)

• Line plots no longer assume monotonic x data when calculating xlims, they show the entire lines now even for unsorted x data. (GH11310, GH11471)

• With matplotlib 2.0.0 and above, calculation of x limits for line plots is left to matplotlib, so that its new default settings are applied. (GH15495)

• Bug in `Series.plot.bar` or `DataFrame.plot.bar` with `y` not respecting user-passed `color` (GH16822)

• Bug causing `plotting.parallel_coordinates` to reset the random seed when using random colors (GH17525)

### GroupBy/resample/rolling¶

• Bug in `DataFrame.resample(...).size()` where an empty `DataFrame` did not return a `Series` (GH14962)

• Bug in `infer_freq()` causing indices with 2-day gaps during the working week to be wrongly inferred as business daily (GH16624)

• Bug in `.rolling(...).quantile()` which incorrectly used different defaults than `Series.quantile()` and `DataFrame.quantile()` (GH9413, GH16211)

• Bug in `groupby.transform()` that would coerce boolean dtypes back to float (GH16875)

• Bug in `Series.resample(...).apply()` where an empty `Series` modified the source index and did not return the name of a `Series` (GH14313)

• Bug in `.rolling(...).apply(...)` with a `DataFrame` with a `DatetimeIndex`, a `window` of a timedelta-convertible and `min_periods >= 1` (GH15305)

• Bug in `DataFrame.groupby` where index and column keys were not recognized correctly when the number of keys equaled the number of elements on the groupby axis (GH16859)

• Bug in `groupby.nunique()` with `TimeGrouper` which cannot handle `NaT` correctly (GH17575)

• Bug in `DataFrame.groupby` where a single level selection from a `MultiIndex` unexpectedly sorts (GH17537)

• Bug in `DataFrame.groupby` where spurious warning is raised when `Grouper` object is used to override ambiguous column name (GH17383)

• Bug in `TimeGrouper` differs when passes as a list and as a scalar (GH17530)

### Sparse¶

• Bug in `SparseSeries` raises `AttributeError` when a dictionary is passed in as data (GH16905)

• Bug in `SparseDataFrame.fillna()` not filling all NaNs when frame was instantiated from SciPy sparse matrix (GH16112)

• Bug in `SparseSeries.unstack()` and `SparseDataFrame.stack()` (GH16614, GH15045)

• Bug in `make_sparse()` treating two numeric/boolean data, which have same bits, as same when array `dtype` is `object` (GH17574)

• `SparseArray.all()` and `SparseArray.any()` are now implemented to handle `SparseArray`, these were used but not implemented (GH17570)

### Reshaping¶

• Joining/Merging with a non unique `PeriodIndex` raised a `TypeError` (GH16871)

• Bug in `crosstab()` where non-aligned series of integers were casted to float (GH17005)

• Bug in merging with categorical dtypes with datetimelikes incorrectly raised a `TypeError` (GH16900)

• Bug when using `isin()` on a large object series and large comparison array (GH16012)

• Fixes regression from 0.20, `Series.aggregate()` and `DataFrame.aggregate()` allow dictionaries as return values again (GH16741)

• Fixes dtype of result with integer dtype input, from `pivot_table()` when called with `margins=True` (GH17013)

• Bug in `crosstab()` where passing two `Series` with the same name raised a `KeyError` (GH13279)

• `Series.argmin()`, `Series.argmax()`, and their counterparts on `DataFrame` and groupby objects work correctly with floating point data that contains infinite values (GH13595).

• Bug in `unique()` where checking a tuple of strings raised a `TypeError` (GH17108)

• Bug in `concat()` where order of result index was unpredictable if it contained non-comparable elements (GH17344)

• Fixes regression when sorting by multiple columns on a `datetime64` dtype `Series` with `NaT` values (GH16836)

• Bug in `pivot_table()` where the result’s columns did not preserve the categorical dtype of `columns` when `dropna` was `False` (GH17842)

• Bug in `DataFrame.drop_duplicates` where dropping with non-unique column names raised a `ValueError` (GH17836)

• Bug in `unstack()` which, when called on a list of levels, would discard the `fillna` argument (GH13971)

• Bug in the alignment of `range` objects and other list-likes with `DataFrame` leading to operations being performed row-wise instead of column-wise (GH17901)

### Numeric¶

• Bug in `.clip()` with `axis=1` and a list-like for `threshold` is passed; previously this raised `ValueError` (GH15390)

• `Series.clip()` and `DataFrame.clip()` now treat NA values for upper and lower arguments as `None` instead of raising `ValueError` (GH17276).

### Categorical¶

• Bug in `Series.isin()` when called with a categorical (GH16639)

• Bug in the categorical constructor with empty values and categories causing the `.categories` to be an empty `Float64Index` rather than an empty `Index` with object dtype (GH17248)

• Bug in categorical operations with Series.cat not preserving the original Series’ name (GH17509)

• Bug in `DataFrame.merge()` failing for categorical columns with boolean/int data types (GH17187)

• Bug in constructing a `Categorical`/`CategoricalDtype` when the specified `categories` are of categorical type (GH17884).

### PyPy¶

• Compatibility with PyPy in `read_csv()` with `usecols=[<unsorted ints>]` and `read_json()` (GH17351)

• Split tests into cases for CPython and PyPy where needed, which highlights the fragility of index matching with `float('nan')`, `np.nan` and `NAT` (GH17351)

• Fix `DataFrame.memory_usage()` to support PyPy. Objects on PyPy do not have a fixed size, so an approximation is used instead (GH17228)

### Other¶

• Bug where some inplace operators were not being wrapped and produced a copy when invoked (GH12962)

• Bug in `eval()` where the `inplace` parameter was being incorrectly handled (GH16732)

## Contributors¶

A total of 206 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.

• 3553x +

• Aaron Barber

• Alan Velasco +

• Alan Yee +

• Alex B +

• Alex Lubbock +

• Alex Marchenko +

• Alex Rychyk +

• Amol K +

• Andreas Winkler

• Andrew +

• Andrew 亮

• André Jonasson +

• Becky Sweger

• Berkay +

• Bob Haffner +

• Bran Yang

• Brian Tu +

• Brock Mendel +

• Carol Willing +

• Carter Green +

• Chankey Pathak +

• Chris

• Chris Billington

• Chris Filo Gorgolewski +

• Chris Kerr

• Chris M +

• Chris Mazzullo +

• Christian Prinoth

• Christoph Moehl +

• DSM

• Daniel Chen +

• Daniel Himmelstein

• Dave Willmer

• David Cook

• David Gwynne

• Dillon Niederhut +

• Douglas Rudd

• Eric Stein +

• Eric Wieser +

• Erik Fredriksen

• Florian Wilhelm +

• Floris Kint +

• Forbidden Donut

• Gabe F +

• Giulio Pepe +

• Guilherme Beltramini

• Guillem Borrell +

• Hanmin Qin +

• Hendrik Makait +

• Hugues Valois

• Hussain Tamboli +

• Iva Miholic +

• Jan Novotný +

• Jan Rudolph

• Jean Helie +

• Jean-Baptiste Schiratti +

• Jean-Mathieu Deschenes

• Jeff Knupp +

• Jeff Reback

• Jeff Tratner

• JennaVergeynst

• JimStearns206

• Joel Nothman

• John W. O’Brien

• Jon Crall +

• Jon Mease

• Jonathan J. Helmus +

• Joris Van den Bossche

• JosephWagner

• Juarez Bochi

• Julian Kuhlmann +

• Karel De Brabandere

• Kassandra Keeton +

• Keiron Pizzey +

• Keith Webber

• Kernc

• Kevin Sheppard

• Kirk Hansen +

• Licht Takeuchi +

• Lucas Kushner +

• Mahdi Ben Jelloul +

• Makarov Andrey +

• Malgorzata Turzanska +

• Marc Garcia +

• Margaret Sy +

• MarsGuy +

• Matt Bark +

• Matthew Roeschke

• Matti Picus

• Mehmet Ali “Mali” Akmanalp

• Michael Gasvoda +

• Michael Penkov +

• Milo +

• Morgan Stuart +

• Morgan243 +

• Nathan Ford +

• Nick Eubank

• Nick Garvey +

• Oleg Shteynbuk +

• P-Tillmann +

• Pankaj Pandey

• Patrick Luo

• Patrick O’Melveny

• Paul Reidy +

• Paula +

• Peter Quackenbush

• Peter Yanovich +

• Phillip Cloud

• Pierre Haessig

• Pietro Battiston

• Prasanjit Prakash

• RobinFiveWords

• Ryan Hendrickson

• Sam Foo

• Sangwoong Yoon +

• Simon Gibbons +

• SimonBaron

• Steven Cutting +

• Sudeep +

• Sylvia +

• T N +

• Telt

• Thomas A Caswell

• Tim Swast +

• Tom Augspurger

• Tong SHEN

• Tuan +

• Vincent La +

• Vivek +

• WANG Aiyong

• WBare

• Wes McKinney

• XF +

• Yi Liu +

• Yosuke Nakabayashi +

• aaron315 +

• abarber4gh +

• aernlund +

• agustín méndez +

• andymaheshw +

• ante328 +

• aviolov +

• bpraggastis

• cbertinato +

• cclauss +

• chernrick

• chris-b1

• dkamm +

• dwkenefick

• economy

• faic +

• fding253 +

• gfyoung

• guygoldberg +

• hhuuggoo +

• huashuai +

• ian

• iulia +

• jaredsnyder

• jbrockmendel +

• jdeschenes

• jebob +

• jschendel +

• keitakurita

• kernc +

• kiwirob +

• kjford

• linebp

• lloydkirk

• louispotok +

• majiang +

• manikbhandari +

• matthiashuschle +

• mattip

• maxwasserman +

• mjlove12 +

• nmartensen +

• pandas-docs-bot +

• parchd-1 +

• philipphanemann +

• rdk1024 +

• reidy-p +

• ri938

• ruiann +

• rvernica +

• s-weigand +

• scotthavard92 +

• skwbc +

• step4me +

• tobycheese +

• topper-123 +

• tsdlovell

• ysau +

• zzgao +
{"ft_lang_label":"__label__en","ft_lang_prob":0.6025145,"math_prob":0.76503026,"size":47936,"snap":"2022-27-2022-33","text_gpt3_token_len":13475,"char_repetition_ratio":0.12786865,"word_repetition_ratio":0.04165409,"special_character_ratio":0.28867656,"punctuation_ratio":0.17466915,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.95122164,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2022-06-26T03:22:52Z\",\"WARC-Record-ID\":\"<urn:uuid:61595093-70c2-44ca-87c6-c8cf16e5a80d>\",\"Content-Length\":\"245123\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:5d97c0e3-2aae-427c-86ec-d306fd39fc5b>\",\"WARC-Concurrent-To\":\"<urn:uuid:edc72441-1ae0-4598-8157-1d563a08ac5c>\",\"WARC-IP-Address\":\"104.26.0.204\",\"WARC-Target-URI\":\"https://pandas.pydata.org/pandas-docs/version/1.3.0/whatsnew/v0.21.0.html\",\"WARC-Payload-Digest\":\"sha1:SIJ5ILRMFSJJUKTR7ALOPUXKS7Q33MU7\",\"WARC-Block-Digest\":\"sha1:NXBITCPBFTAGII4U4AQ7MATYTLR4EFRR\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2022/CC-MAIN-2022-27/CC-MAIN-2022-27_segments_1656103036363.5_warc_CC-MAIN-20220626010644-20220626040644-00531.warc.gz\"}"}