# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/SubstraFoundation/substra-examples/blob/master/mnist-dp/Mnist_DP_Substra_Example_Notebook.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="-4DF18EyM3rb" colab_type="text"
# In this Notebook, you can test the example's assets and potentially run ML tasks on a public distant VM.
# On Google Colab, you can see and modify the assets with the "Files" button on the left.

# + id="0X1CGFsjTxFp" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="532a7e45-07d3-436e-8cf0-326525044473"
try:
  # # %tensorflow_version only exists in Colab.
  # %tensorflow_version 1.x
except Exception:
  pass

# + [markdown] id="1SJi6ACtKP0B" colab_type="text"
# # Mnist with Differential Privacy
#
# *This example is a Substra implementation of on the [Classification_Privacy tutorial](https://github.com/tensorflow/privacy/blob/master/tutorials/Classification_Privacy.ipynb) from [Tensorflow_Privacy](https://github.com/tensorflow/privacy). The structure of this example is inspired from [Substra's Titanic Example](https://github.com/SubstraFoundation/substra/blob/master/examples/titanic/)*
#
# > [Differential privacy](https://en.wikipedia.org/wiki/Differential_privacy) (DP) is a framework for measuring the privacy guarantees provided by an algorithm. Through the lens of differential privacy, we can design machine learning algorithms that responsibly train models on private data. Learning with differential privacy provides provable guarantees of privacy, mitigating the risk of exposing sensitive training data in machine learning. Intuitively, a model trained with differential privacy should not be affected by any single training example, or small set of training examples, in its data set. This mitigates the risk of exposing sensitive training data in ML.
# >  
# > The basic idea of this approach, called differentially private stochastic gradient descent (DP-SGD), is to modify the gradients used in stochastic gradient descent (SGD), which lies at the core of almost all deep learning algorithms. Models trained with DP-SGD provide provable differential privacy guarantees for their input data. There are two modifications made to the vanilla SGD algorithm:
# >  
# > 1. First, the sensitivity of each gradient needs to be bounded. In other words, we need to limit how much each individual training point sampled in a minibatch can influence gradient computations and the resulting updates applied to model parameters. This can be done by *clipping* each gradient computed on each training point.
# > 2. *Random noise* is sampled and added to the clipped gradients to make it statistically impossible to know whether or not a particular data point was included in the training dataset by comparing the updates SGD applies when it operates with or without this particular data point in the training dataset.
# >  
# > This tutorial uses [tf.keras](https://www.tensorflow.org/guide/keras) to train a convolutional neural network (CNN) to recognize handwritten digits with the DP-SGD optimizer provided by the TensorFlow Privacy library. TensorFlow Privacy provides code that wraps an existing TensorFlow optimizer to create a variant that implements DP-SGD.  
# > &mdash; [The TensorFlow Authors][1]
#
# [1]: https://github.com/tensorflow/privacy/blob/master/tutorials/Classification_Privacy.ipynb
#

# + [markdown] id="jFjGg7WgOb1q" colab_type="text"
#
# ## Prerequisites
#
# In order to run this example, you'll need to:
#
# * use Python 3
# * have [Docker](https://www.docker.com/) installed
# * [install the `substra` cli](https://github.com/SubstraFoundation/substra#install) (supported version: 0.6.0)

# + id="0pnSX37XNth3" colab_type="code" colab={}
# ! pip3 install substra==0.6.0

# + [markdown] id="78Dg-ioqNqie" colab_type="text"
#
# * [install the `substratools` library](https://github.com/substrafoundation/substra-tools) (supported version: 0.6.0)
# * [pull the `substra-tools` docker images](https://github.com/substrafoundation/substra-tools#pull-from-private-docker-registry)
# * have access to a Substra installation ([configure your host to a public node ip](https://doc.substra.ai/getting_started/installation/local_install_skaffold.html#network) or [install Substra on your machine](https://doc.substra.ai/getting_started/installation/local_install_skaffold.html))

# + id="OXNWTXfLdNEr" colab_type="code" colab={}
#replace this ip by the ip of a distant VM running a substra node
public_node_ip = "127.0.0.1"

# + id="3Sj64EGcd2bG" colab_type="code" colab={}
# ! echo "{public_node_ip} substra-backend.node-1.com substra-frontend.node-1.com substra-backend.node-2.com substra-frontend.node-2.com" | sudo tee -a /etc/hosts
# Check if it's ok
# ! curl substra-backend.node-1.com/readiness

# + [markdown] id="WiNgyy4odKzR" colab_type="text"
#
# * create a substra profile to define the substra network to target, for instance:
#

# + id="njy5vYAvWoEE" colab_type="code" colab={}
# ! substra config --profile node-1 http://substra-backend.node-1.com
# ! substra login --profile node-1 --username node-1 --password '<PASSWORD>'

# + [markdown] id="laY7B4rs6Ik-" colab_type="text"
# * checkout this repository

# + id="NCaLsa5LOLTu" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 221} outputId="5cb0369a-7e6e-470a-f607-f8c4cbc2d95b"
# %cd /content
# ! git clone https://github.com/SubstraFoundation/substra-examples/
# %cd substra-examples/content/substra-examples/mnist-dp

# + [markdown] id="KUMkvMmyOHe5" colab_type="text"
#
# All commands in this example are run from the `mnist-dp` folder.
#

# + [markdown] id="bibVWArLPIoM" colab_type="text"
# ## Data preparation
#
# The first step will be to generate train and test data from keras.datasets.mnist
#
# To generate the data, run:
#
#

# + id="UBprYg-5PUvx" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 102} outputId="888d61d9-e26f-466d-b56f-484526de9f58"
# %cd /content/substra-examples/mnist-dp
# #! pip install --upgrade pip
# #! pip install -r scripts/requirements.txt
# requirements are already satisfied in Colab, except for substratools
# ! pip install substratools==0.6.0

# + id="3M648e_-QH4N" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 170} outputId="f429e716-b585-4512-b95e-c47af0f2faf7"
# %cd /content/substra-examples/mnist-dp
# ! python scripts/generate_data.py

# + [markdown] id="ccjTJmyMPTiX" colab_type="text"
#
#
# This will create two sub-folders in the `assets` folder:
#
# * `train_data` contains train data features and labels as numpy array files
# * `test_data` contains test data features and labels as numpy array files
#
# ## Writing the objective and data manager
#
# Both objective and data manager will need a proper markdown description, you can check them out in their respective
# folders. Notice that the data manager's description includes a formal description of the data structure.
#
# Notice also that the `metrics.py` and `opener.py` module both rely on classes imported from the `substratools` module.
# These classes provide a simple yet rigid structure that will make algorithms pretty easy to write.
#
# ## Writing a simple algorithm
#
# You'll find under `assets/algo_cnn_dp` an implementation of the cnn model in the [Classification_Privacy tutorial](https://github.com/tensorflow/privacy/blob/master/tutorials/Classification_Privacy.ipynb). Like the metrics and opener scripts, it relies on a
# class imported from `substratools` that greatly simplifies the writing process. You'll notice that it handles not only
# the train and predict tasks but also a lot of data preprocessing.
#
# This algorithm measure the differential privacy guarantee after training the model:
# You will see in the console the value Epsilon (ϵ) - This is the privacy budget. It measures the strength of the privacy guarantee by bounding how much the probability of a particular model output can vary by including (or excluding) a single training point. A smaller value for ϵ implies a better privacy guarantee. However, the ϵ value is only an upper bound and a large value could still mean good privacy in practice.
#
# This value depends on:
#
# 1. The total number of points in the training data, `n`.
# 2. The `batch_size`.
# 3. The `noise_multiplier`.
# 4. The number of `epochs` of training.
#
# You can find a description of the hyperparameters [here](https://github.com/tensorflow/privacy/tree/master/tutorials#parameters).
#
#

# + [markdown] id="tImaUQdlRDPv" colab_type="text"
# ## Testing our assets
#
# ### Using asset command line interfaces
#
# You can first test each assets with the `substratools` CLI, by running specific ML tasks in your local Python environment.
#
#

# + [markdown] id="2sKcqmkFUNvi" colab_type="text"
# #### Training task
#
#
#
#

# + id="yby0gIzZRQx7" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 1000} outputId="a6c4d866-b0de-4477-8198-219c676000b0"
# train your model with the train_data
# ! python assets/algo_cnn_dp/algo.py train \
#   --debug \
#   --opener-path assets/dataset/opener.py \
#   --data-samples-path assets/train_data \
#   --output-model-path assets/model/model \
#   --log-path assets/logs/train.log


# + id="5Pp7yN4YRYTy" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 938} outputId="dc1e0d1f-27f2-4e13-e7c9-9167802da87a"
#predict the labels of train_data with your previously trained model
# ! python assets/algo_cnn_dp/algo.py predict \
#   --debug \
#   --opener-path assets/dataset/opener.py \
#   --data-samples-path assets/train_data \
#   --output-predictions-path assets/pred-train.npy \
#   --models-path assets/model/ \
#   --log-path assets/logs/train_predict.log \
#   model

# + id="Nm6SnGQJRboG" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 136} outputId="357993af-b10e-419b-ef38-d58d0cc708fb"
#calculate the score of your model on train_data predictions
# ! python assets/objective/metrics.py \
#   --debug \
#   --opener-path assets/dataset/opener.py \
#   --data-samples-path assets/train_data \
#   --input-predictions-path assets/pred-train.npy \
#   --output-perf-path assets/perf-train.json \
#   --log-path assets/logs/train_metrics.log

# + [markdown] id="711Mg8smVlEf" colab_type="text"
# #### Testing task

# + id="vhoL2XIkVpjW" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 938} outputId="40a29a34-276b-4238-8350-4f868d4eb39e"
#predict the labels of test_data with your previously trained model
# ! python assets/algo_cnn_dp/algo.py predict \
#   --debug \
#   --opener-path assets/dataset/opener.py \
#   --data-samples-path assets/test_data \
#   --output-predictions-path assets/pred-test.npy \
#   --models-path assets/model/ \
#   --log-path assets/logs/test_predict.log \
#   model

# + id="pFXL-iIFVrkR" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 136} outputId="676009ba-9e10-49f3-e003-86b2364cc0a5"
#calculate the score of your model on test_data predictions
# ! python assets/objective/metrics.py \
#   --debug \
#   --opener-path assets/dataset/opener.py \
#   --data-samples-path assets/test_data \
#   --input-predictions-path assets/pred-test.npy \
#   --output-perf-path assets/perf-test.json \
#   --log-path assets/logs/test_metrics.log

# + [markdown] id="e56pBMUsRfHw" colab_type="text"
# ### Using substra cli
#
# Before pushing our assets to the platform, we need to make sure they work well. To do so, we can run them locally in a
# Docker container. This way, if the training fails, we can access the logs and debug our code.
#
# To test the assets, we'll use `substra run-local`, passing it paths to our algorithm of course, but also the opener,
# the metrics and to the data samples we want to use. It will launch a training task on the train data, a prediction task on the test data and return the accuracy score.
#

# + id="ZsVjUgrvYziC" colab_type="code" colab={}
#you will need Docker to run this (not available in Colab)
# ! substra run-local assets/algo_cnn_dp \
#   --train-opener=assets/dataset/opener.py \
#   --test-opener=assets/dataset/opener.py \
#   --metrics=assets/objective/ \
#   --train-data-samples=assets/train_data \
#   --test-data-samples=assets/test_data

# + [markdown] id="1nYflmdBWB_y" colab_type="text"
# At the end of this step, you'll find in the newly created `sandbox/model` folder a `model` file that contains your
# trained model. There is also a `sandbox/pred_train` folder that contains both the predictions made by the model on
# train data and the associated performance.
#

# + [markdown] id="Hljsxt1PWEAr" colab_type="text"
# #### Debugging
#
# It's more than probable that your code won't run perfectly the first time. Since runs happen in dockers, you can't
# debug using prints. Instead, you should use the `logging` module from python. All logs can then be consulted at the end
# of the run in  `sandbox/model/log_model.log`.
#
# ## Adding the assets to substra
#
# ### Adding the objective, dataset and data samples to substra
#
# A script has been written that adds objective, data manager and data samples to substra. It uses the `substra` python
# sdk to perform actions. It's main goal is to create assets, get their keys and use these keys in the creation of other
# assets.
#
# To run it:
#

# + id="jNSaDP9iWLyy" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 153} outputId="0a81aea5-cdb2-4293-8672-ecc445d1ea64"
# ! python scripts/add_dataset_objective.py

# + [markdown] id="BP7PUXTyWVt0" colab_type="text"
# This script just generated an `assets_keys.json` file in the `mnist` folder. This file contains the keys of all assets
# we've just created and organizes the keys of the train data samples in folds. This file will be used as input when
# adding an algorithm so that we can automatically launch all training and testing tasks.
#

# + [markdown] id="iF3ykfHVWYum" colab_type="text"
# ### Adding the algorithm and training it
#
# The script `add_train_algo_cnn.py` pushes our simple algo to substra and then uses the `assets_keys.json` file
# we just generated to train it against the dataset and objective we previously set up. It will then update the
# `assets_keys.json` file with the newly created assets keys (algo, traintuple and testtuple)
#
# To run it:

# + id="cpnh9e_yWVC_" colab_type="code" colab={}
# ! python scripts/add_train_algo_cnn_dp.py

# + [markdown] id="v9kc1CyBWaxJ" colab_type="text"
# It will end by providing a couple of commands you can use to track the progress of the train and test tuples as well
# as the associated scores. Alternatively, you can browse the frontend to look up progress and scores.

StarcoderdataJupyterScriptsDedupFiltered