# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Integrate Data Validation Into Your Pipeline
#
#

# +
# Prep environment and logging

import json
import great_expectations as ge
import great_expectations.jupyter_ux
from great_expectations.datasource.types import BatchKwargs
from datetime import datetime

# -

# **Watch** a [short tutorial video](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#video) or **read** [the written tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation)
#
# We'd love it if you **reach out for help on** the [**Great Expectations Slack Channel**](https://greatexpectations.io/slack)

# ## 1. Get a DataContext
# This represents your project that you set up using `great_expectations init`. [Read more in the tutorial](https://great-expectations.readthedocs.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#get-datacontext-object)
#

context = ge.data_context.DataContext()

# ## 2. Get a pipeline run id
#
# Generate a run id, a timestamp, or any other string that is meaningful to you that will help you refer to validation results. We recommend they be chronologically sortable.
# [Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#set-a-run-id)
#

# Let's make a simple sortable timestamp. Note this could come from your pipeline runner.
run_id = datetime.utcnow().strftime("%Y%m%dT%H%M%S.%fZ")
run_id

# ## 3. Pick a data asset & expectation suite name
#
# [Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#choose-data-asset-and-expectation-suite)
#

great_expectations.jupyter_ux.list_available_data_asset_names(context)

data_asset_name = "REPLACE_ME" # TODO: replace with your value!
data_asset_name = context.normalize_data_asset_name(data_asset_name)
expectation_suite_name = "warning" # TODO: replace with your value!

# ## 4. Load a batch of data from the data asset you want to validate
#
# Learn about `get_batch` in [this tutorial](https://docs.greatexpectations.io/en/latest/getting_started/create_expectations.html?utm_source=notebook&utm_medium=create_expectations#get-batch)
#
# How you get a batch of data will depend on your datasource (pandas, SQL, Spark, etc). `BatchKwargs` tell your specific datasource how to fetch a batch of data and might include a filepath, table name, SQL query, or even an existing DataFrame loaded outside of Great Expectations. Choose the best batch_kwargs for your situation.

# + pycharm={"name": "#%%\n"}
# If you're working with data loaded from files, and GE listed and profiled your files correctly:
batch_kwargs = context.yield_batch_kwargs(data_asset_name)

# If you would like to validate data in a database, using an entire table or view:
# batch_kwargs = {'table': 'name_of_table_to_validate'}  # Add a 'schema' key if you need to specify that explicitly

# If you would like to validate data in a database, using a query to construct a temporary table:
# batch_kwargs = {'query': 'SELECT YOUR_ROWS FROM YOUR_TABLE'}

# If you would like to control reading of data outside of Great Expectations, and provide a pre-built Dataframe:
# df = spark.read.csv(...)
# df = pd.readcsv(...)
# batch_kwargs = {'dataset': df}
# -

batch = context.get_batch(
    data_asset_name=data_asset_name,
    expectation_suite_name=expectation_suite_name,
    batch_kwargs=batch_kwargs
)
batch.head()

# ## 5. Validate the batch
#
# This is the "workhorse" of Great Expectations. Call it in your pipeline code after loading the file and just before passing it to your computation.
#
# [Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#validate)
#
#

# +
validation_result = batch.validate(run_id=run_id)

if validation_result["success"]:
    print("This file meets all expectations from a valid batch of {0:s}".format(str(data_asset_name)))
else:
    print("This file is not a valid batch of {0:s}".format(str(data_asset_name)))
# -

# ## 6. Review the validation results
#
# [Read more in the tutorial](https://docs.greatexpectations.io/en/latest/getting_started/pipeline_integration.html?utm_source=notebook&utm_medium=integrate_validation#review-validation-results)
#

print(json.dumps(validation_result, indent=4))

# ## 7. Validation Operators
#
# The `validate` method evaluates one batch of data against one expectation suite and returns a dictionary of validation results. This is sufficient when you explore your data and get to know Great Expectations.
# When deploying Great Expectations in a **real data pipeline, you will typically discover additional needs**:
#
# * validating a group of batches that are logically related
# * validating a batch against several expectation suites such as using a tiered pattern like `warning` and `failure`
# * doing something with the validation results (e.g., saving them for a later review, sending notifications in case of failures, etc.).
#
# `Validation Operators` provide a convenient abstraction for both bundling the validation of multiple expectation suites and the actions that should be taken after the validation.
#
# [Read more about Validation Operators](https://docs.greatexpectations.io/en/latest/features/validation_operators_and_actions.html?utm_source=notebook&utm_medium=integrate_validation)
#

# +
# This is an example of invoking a validation operator that is configured by default in the great_expectations.yml file

results = context.run_validation_operator(
    assets_to_validate=[batch],
    run_id=run_id,
    validation_operator_name="action_list_operator",
)

print(results["success"])
# -

# ## Congratulations! You used your Expectations to validate one of your data assets!
#
# ## Next steps:
#
# ### 1. Data Docs
# Jump back to the command line and run `great_expectations build-docs` to see your Data Docs. These will now include a **data quality report** built from the validations that helps you understand and communicate about your data.
# ### 2. Explore the documentation & community
# You are now among the elite data professionals who know how to build robust protection for pipelines and machine learning models. Join the [**Great Expectations Slack Channel**](https://greatexpectations.io/slack) to see how others are wielding these superpowers.

StarcoderdataJupyterScriptsDedupFiltered