text: written by Eric J. Ma on 2020-08-30 | tags: data science statistics data validation data engineering pandera software tools tips and tricks
  I have test-driven the
  pandera package over the past few days,
  and I'm extremely happy that this package exists!
  Let me tell you why.
  "Data testing" is the colloquial term that I have used in place of the more formal term, "schema validation", or "data validation". Ever since I was burned once by CSV files with invalid column names, I put together a data testing tutorial to kickstart the idea, and have been keeping an eye on the space ever since.
  But let’s step back a bit.
  What do we mean by schema/data validation? It’s merely a fancy way of checking, “are my data as I expect them to:
  df = pd.DataFrame( { "non_negative": [1.1, 0.34, 2.5], "two_values": ["val1", "val2", "val2"], } )
  If I were implementing these checks in a test suite in vanilla Python, I would write them as follows:
  from project_source.data import load_data import numpy as np def test_load_data(): data = load_data(**kwargs) assert np.all(data["non_negative"] > 0) assert set(data["two_values"].unique()) == set(["val1", "val2"])
  Having the checks in the test suite is nice,
  as it means if the checks are run as part of a CI pipeline,
  they’ll continually be checked.
  However,
  they cannot be checked whenever I call on
  load_data(),
  load_data() function.
  And we know that duplication of code increases risk of stale code.
  Enter
  pandera.
  Instead of writing the checks in two separate places,
  I can start by writing the following declarative schema in a Python module,
  that serves as a defined source of truth:
  # project_source/schemas.py from pandera import Column, DataFrameSchema, Check import pandera as pa schema = DataFrameSchema( columns={ "non_zero_column": Column(pa.Float, Check.greater_than(0)), "categorical_column": Column(pa.String, Check.isin(["value_1", "value_2"])), } )
  Now, the flexibility with which this schema gets used is simply superb! Because I've declared it as a Python object in code, I can import it anywhere I want validation to happen.
  For example, I can validate the inputs to a function at runtime:
  # project_source/data.py from pandera import check_input, check_output from .schemas import dataframe_schema, cleaned_df_schema @check_input(dataframe_schema) @check_output(cleaned_df_schema) def clean(dataframe): # stuff happens return cleaned_df
  And because the validation happens as soon as the function is executed, I can also validate the schema as part of an automatically-executed test suite that continuously runs and checks data alongside code.
  from project_src.data import load_some_data def test_load_some_data(): data = load_some_data()
  The data test suite becomes simpler to maintain, as it gets reduced to a simple execution test. Validating data at every step that it could go wrong is pretty important!
  Additionally,
  if there is a new data file that needs to be processed in the same way,
  when we pass in the dataframe version into the
  clean_data function,
  it is automatically validated by
  pandera’s
  check_input.
  If a dataframe that fails
  check_input,
  then
  pandera will give us an error message straight away!
  These are just the basics. Pandera comes with even more:
  Index/
  MultiIndex.
  I'm excited to see how this goes.
  Congratulations to the package author Niels Bantilan,
  it's a package that has all of the
  pandas DataFrames,
  an idiomatic data structure in the Python data science world,
  which is always a plus.
  Having incorporated
  pandera into a project just recently,
  I'm excited to see where its future is headed!
  I can see that data validation actually has its roots in statistical evaluation of data.
  Humour me for a moment.
  Say we have a column that can only take two values, such as “yes” and “no”.
  From a statistical viewpoint, that data are generated from Bernoulli distribution, with “yes”/1 and “no”/0 being the outcomes.
  If you get a “maybe” in that column, that value is outside of the support of the distribution. It’s like trying to evaluate the likelihood of observing a 2 from a Bernoulli. It won’t compute.
  So every time we run a
  pandera check,
  we are effectively expressing a statistical check of some kind.
  The byline of the package,
  “Statistical Data Validation for Pandas”,
  is even more apt once we consider this viewpoint!
  I hope this post encourages you to give
  pandera a test-drive!
  Its implementation of "runtime data validation" is very handy,
  allowing me to declare the assumptions I have about my data up-front,
  and catch any violations of those assumptions as early as possible.