%load_ext yamlmagic

%%yaml basic_config
blobstores:
    disk:
        type: disk
        cachedir: /tmp/provenance-intro-artifacts
        read: True
        write: True
        delete: True
artifact_repos:
    local:
        type: postgres
        db: postgresql://localhost/provenance-intro
        store: 'disk'
        read: True
        write: True
        delete: True
        # this option will create the database if it doesn't exist
        create_db: True
default_repo: local

import provenance as p


p.load_config(basic_config)

"""
Explanation: Introductory Guide
provenance is a Python library for function-level caching and provenance that aids in creating parsimonious pythonic pipelines&trade;. By wrapping functions in the provenance decorator computed results are cached across various stores (disk, S3, SFTP) and provenance (i.e. lineage) information is tracked and stored in an artifact repository. A central artifact repostiory can be used to enable production pipelines, team collaboration, and reproducible results.
What that means in practice is that you can easily keep track of how artifacts (models, features, or any object or file) are created, where they are used, and have a central place to store and share these artifacts. This basic plumbing is required (or at least desired!) in any machine learning pipeline or project. provenance can be used standalone along with a build server to run pipelines or in conjunction with more advanced workflow systems (e.g. Airflow, Luigi).
Configuration
Before you can use provenance you need to configure it. The suggested way to do so is via the load_config function that takes a dictionary of settings. (ProTip: For a team environment take a look at ymlconf which does merging of configs, e.g. a shared config in a repo and a local one that isn't committed.)
We'll define our configuration map in YAML below. The config specifies a blobstore, where the cached values are stored, and an artifact repository, where the metadata is stored. In this example the cached values are stored on disk and the metadata is stored in a Postgres database.
End of explanation
"""


import time

@p.provenance()
def expensive_add(a, b):
    time.sleep(2)
    return a + b


@p.provenance()
def expensive_mult(a, b):
    time.sleep(2)
    return a * b

%%time
result = expensive_add(4, 3)
print(result)

"""
Explanation: Basic Example
To introduce the concepts we'll take a look at a small function that has been decorated with provenance.
Now lets define some decorated functions...
End of explanation
"""


%%time
result = expensive_add(4, 3)
print(result)

"""
Explanation: As expected, we have a slow addition function. To see the effect of the caching we can repeat the same function invocation:
End of explanation
"""


import provenance.vis as vis

vis.visualize_lineage(result)

"""
Explanation: Visualization
If you have used any caching/memoization decorator or library (e.g. joblib) then this is old hat to you. What is different is how the provenance of this result is recorded and how it can be accessed. For example, with the same result we can visualize the associated lineage:
End of explanation
"""


result

"""
Explanation: What is an artifact?
In the above visualization the artifact is outlined in red with the artifact id on the left and the value (result) on the right. How is this possible and what exactly is an artifact? Well, the result is not a raw 7 but rather is an ArtifactProxy which wraps the 7.
End of explanation
"""


result + 3

"""
Explanation: The hash in the parenthesis is the id of the artifact which is a function of the name of the function used and the inputs used to produce the value. 
All properties, methods, and operations called against this object will be proxied to the underlying value, 7 in this case. You can treat the result as you normally would:
End of explanation
"""


artifact = result.artifact
artifact

"""
Explanation: The one exception to this is the artifact property which returns the Artifact.
End of explanation
"""


# What function what used to create artifact?
artifact.fn_module, artifact.fn_name

# What inputs were used?
artifact.inputs

"""
Explanation: On this artifact is where you will find the provenance information for the value.
End of explanation
"""


artifact.run_info

"""
Explanation: This information is what powered the visualization above.
Each artifact has additional information attached to it, such as the id, value_id, value, and run_info. run_info captures information about the environment when the artifact was created:
End of explanation
"""


# get the id of the artifact above
artifact_id = artifact.id

loaded_artifact = p.load_artifact(artifact_id)

loaded_artifact

"""
Explanation: You typically also want the run_info to include the git ref or build server job ID that was used to produce the artifact. This is easily done by using the provided provenance.set_run_info_fn hook. Please refer to the API documenation for details.
Loading Artifacts
Aside from calling decorating functions and having cached artifacts returned you can also explicitly load an artifact by it's id. This becomes useful in a team setting since you can have a shared store on S3, e.g. "Check out this amazing model I just built, load artifact 349ded4f...!". The other main usecase for this is in production settings. See the Machine Learning Pipeline guide for more details on that.
End of explanation
"""


loaded_artifact.proxy()

"""
Explanation: You can inspect the loaded_artifact as done above or load the value into a proxy:
End of explanation
"""


p.load_proxy(artifact_id)

"""
Explanation: Alternatively, if you can load the proxy directly with load_proxy. load_artifact is still useful when you want the provenance and other metadata about an artifact but not the actual value.
End of explanation
"""


a1 = expensive_add(4, 3)
a2 = expensive_add(1, 1)

result = expensive_mult(a1, a2)

vis.visualize_lineage(result)

"""
Explanation: Multiple Functions Example
In any advanced workflow you will have a series of functions all pipelined together. For true provenance you will need to be able to track all artifacts to their source.  This is why the computed results of functions are returned as ArtifactProxys, to enable the flow of artifacts to be tracked.
End of explanation
"""


repo = p.get_default_repo()

db = repo._db_engine

import pandas as pd

pandas.read_sql("select * from artifacts", db)

"""
Explanation: Note how the final artifact can be traced all the way back to the original inputs! As mentioned above this is enabled by the passing of the ArtifactProxy results, which means that for you to take advantage of this you must be passing the proxies throughout your entire pipeline. Best practice is to have your outer functions take basic Python data structures and then pass all resulting complex objects (e.g. scikit learn models) as ArtifactProxys.
Underlying Storage
Under the hood all of the artifacts are being recorded to the artifact repository we configured above. A repository is comprised of a store that records provenance and metadata and a blobstore that stores the serialized artifact (joblib serialization is the default). While you rarely would need to access the blobstore files directly, querying the DB directly is useful and is a hallmark of the library.
End of explanation
"""


pandas.read_sql("""select * from artifacts 
                   where fn_name = 'expensive_add'
                   and (inputs_json ->> 'a')::int IN (1, 4)
                """, db)

# the blobs are written to files with the names matching the hash of the content
!ls -l /tmp/provenance-intro-artifacts | head

"""
Explanation: A few things to note here...


The input_artifact_ids is an array of all of the artifact ids that were used in the input (even if nested in another data structure). This allows you to efficiently query for the progeny of a particular artifact.


The inputs_json and custom_fields columns are stored as JSONB in Postgres allowing you to query the nested structure via SQL. For example, you could find all the addition artifacts that were created with a particular argument:
End of explanation
"""


@p.provenance()
def expensive_add(a, b):
    time.sleep(2)
    return a + b + a + b + a

%%time
expensive_add(4, 3)

"""
Explanation: Function Versioning
A potential pitfall of using provenance is getting stale cache results after updating your function. For example lets say we update our function definition:
End of explanation
"""


# the default version is 0 so lets set it to 1
@p.provenance(version=1)
def expensive_add(a, b):
    time.sleep(2)
    return a + b + a + b + a

%%time
expensive_add(4, 3)

"""
Explanation: Noting the quick return time and incorrect result we see that we got an old (stale) cached result from our initial implementation. Rather than trying to determine if a function has been updated as joblib does the provenance library requires that you explicity version your functions to force new results to be computed. The rationale behind this is that it is quite hard (impossible in Python?) to tell when a funtion's definition or that of a helper function, which may be in another library, has semantically changed. For this reason the user of provenance must increment the version number of a function in the decorator:
End of explanation
"""


