# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernel_info:
#     name: python3-azureml
#   kernelspec:
#     display_name: Python 3.6 - AzureML
#     language: python
#     name: python3-azureml
# ---

# + [markdown] nteract={"transient": {"deleting": false}} id="uQQ5ihC-ycGn"
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
#
# # Train your own Model and Deploy to Device
#
# **NOTE**
# * Warning: copying *.pb, *.bin, or, *.blob using the web interface can corrupt the files. If needed download and use Azure storage explorer or the CL.
# * Make sure to run each cell individually as some cells will require input parameters.
#
# This notebook shows how to create a Tensorflow object detection model, how to convert the model to the appropriate format for the Perception Development
# Kit, and how to deploy the model to your kit.
#
# This notebook takes a transfer learning approach, using a pre-trained Tensorflow Mobilenet model with custom dataset
# and SSDLite layers that we will train to detect bowls.
# We use the [Tensorflow object detection API](https://github.com/tensorflow/models/tree/master/research/object_detection).
#
# The trained model will be deployed to the Azure Percept Devkit using the Module Twin Update method.

# + [markdown] nteract={"transient": {"deleting": false}} id="Se2G0XW_ycGp"
# # Setting up the GPU for Tensorflow 1.15
#
# We will be using Tensorflow 1.15 for this example, and Tensorflow 2.x examples should be coming soon.
#
# The datascience notebook VMs do not come with the appropriate libraries installed for this version of Tensorflow, and so we will need to
# install them. If your compute node/cluster is a GPU machine, you will want to make sure the appropriate version of CUDA is installed
# into your VM so that Tensorflow can use it to accelerate training substantially.
#
# If you are using a GPU-enabled compute device, you will want to follow these steps. Otherwise, feel free to skip this cell.
#
# 1. Start your compute instance if it is not already started.
# 1. Select Open Terminal, which should be a button next to your compute and kernel selection boxes.
# 1. Run `conda info -e` to list the available conda environments. There should be an azureml_py36 environment.
# 1. Select azureml_py36 by running `conda activate azureml_py36`
# 1. Uninstall Pytorch and Tensorflow (which depend on the wrong version of CUDA) and the wrong version of CUDA with `conda uninstall cudatoolkit pytorch`
# 1. Install the correct version of CUDA with `conda install cudatoolkit=10.0`
#
# You may now close the terminal tab. We will install Tensorflow in a cell of this notebook later.

# + [markdown] nteract={"transient": {"deleting": false}} id="RN0t7wf3ycGr"
# ## Collecting the data
#
# In this notebook, we use a custom dataset that will be used to train a model that detects bowls.
#
# To compile this dataset, we have a couple of options. First, we could use a labeling tool like [VoTT](https://github.com/microsoft/VoTT) or
# [LabelImg](https://github.com/tzutalin/labelImg). Just make sure to export the dataset to Pascal VOC format. There should be one XML file for
# each image, and the XML files should be under annotations/xmls.
#
# The other option for compiling a dataset is to use an already existing dataset like COCO and then filter out all the images and annotations
# that we don't care about. This is the approach we take here.
#
# **NOTE:**
# The first cell in this notebook contains the code necessary to get the dataset and convert it to the format we need. This will involve downloading
# the entire COCO dataset (train and validation splits for 2017), which is tens of GBs in size. This may not fit in your Azure ML compute's storage.
# Our recommendation is to run this cell locally (i.e., copy and paste the commands into your own shell and run them), then upload
# the resulting dataset to Azure via [Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/).

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082546298} id="nq5aYIbTycGt"
# First, get the Advanced Code Repository
# #!wget https://github.com/microsoft/azure-percept-advanced-development/archive/main.zip
# #!unzip azure-percept-advanced-development-main.zip
# #!rm azure-percept-advanced-development-main.zip
# #!cd azure-percept-advanced-development-main/machine-learning-notebooks/transfer-learning/scripts

# Get the data
# #!mkdir -p coco/images

# Get the 2017 training split
# #!wget http://images.cocodataset.org/zips/train2017.zip
# #!unzip train2017.zip
# #!rm train2017.zip
# #!mv train2017 coco/images/

# Get the 2017 validation split
# #!wget http://images.cocodataset.org/zips/val2017.zip
# #!unzip val2017.zip
# #!rm val2017.zip
# #!mv val2017 coco/images/

# Get all the annotations
# #!wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
# #!unzip annotations_trainval2017.zip
# #!rm annotations_trainval2017.zip
# #!mv annotations coco/annotations

# Filter the annotations down to a JSON that only contains images that contain at least one bowl
# #!python filter_coco.py -i coco/annotations/instances_train2017.json -o coco_train2017_bowls.json -c bowl
# #!python filter_coco.py -i coco/annotations/instances_val2017.json -o coco_val2017_bowls.json -c bowl

# Now create a filtered version of the coco train and val split based on their annotation JSONs
# #!python create_coco_subset.py -i coco_train2017_bowls.json coco_val2017_bowls.json -g coco/images/train2017 coco/images/val2017 -o coco_filtered

# Create Pascal VOC formatted dataset from the COCO subset
# #!python convert_coco_to_voc.py coco_filtered --target bowls_voc

# You should now have a data folder called bowls_voc, which should look like this:
# bowls_voc/
#   - annotations/
#       - xmls/
#   - images/

# Now upload 'bowls_voc' to your workspace using the Azure Storage Explorer.
# To do so, please follow these instructions:
# 1. Install Azure Storage Explorer and set it up according to the instructions found here: https://azure.microsoft.com/en-us/features/storage-explorer/
# 2. Once your account is linked, you should be able to open Azure Storage Explorer, select the appropriate subscription, storage account, and file share.
# 3. Upload 'bowls_voc' to the /Users/<your user> folder under the appropriate file share.

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082546638} id="Pkj1xCSAycG0"
# Save current directory for later reference
# modelroot = !pwd
modelroot = modelroot[0]
modelroot

# + gather={"logged": 1599082560851} tags=[] id="Ggi7lGAkycG6"
# Setup workspace for Azure ML
# !pip install azureml.core

import azureml.core
from azureml.core import Workspace
print(azureml.core.VERSION)

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082563544} id="47QAXZ4MycG_"
# Install tensorflow v1.15 required for OpenVINO model conversion
# !pip install tensorflow-gpu==1.15

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082566514} tags=[] id="Mc1rwE54ycHD"
# Install Tensor flow models and scripts, v1.13.0 needed for OpenVINO, which is what we use on the eye development kit.
repository = '--depth 1 --branch v1.13.0 https://github.com/tensorflow/models.git'
# !pip install tf-slim
# !git clone $repository

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082572881} tags=[] id="d4paXYM0ycHH"
# Install required TF packages
# !sudo -s apt-get install -qq protobuf-compiler python-tk
# !pip install Cython contextlib2 pillow lxml matplotlib PyDrive pycocotools build utils dataclasses install azure-iot-device azure-iot-hub numpy==1.17

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082573053} tags=[] id="F06m52qoycHL"
# Setup python path for TF object detection API and TF-Slim
import os
import sys

cwd = os.getcwd()
sys.path.append(cwd)

research = cwd + '/models/research'
sys.path.append(research)

slim = cwd + '/models/research/slim'
sys.path.append(slim)

# %set_env PYTHONPATH=''
os.environ['PYTHONPATH'] = research + ":" + slim +  ":" + cwd
os.environ['PYTHONPATH']

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082576758} tags=[] id="UrdLfSLUycHO"
# Update protocol buffers for TF object detection API
# !curl -OL 'https://github.com/google/protobuf/releases/download/v3.2.0/protoc-3.2.0-linux-x86_64.zip'

# Unzip it
unzipcmd = '-o protoc-3.2.0-linux-x86_64.zip -d protoc3'
# !unzip $unzipcmd

# Put it into /usr/local/bin to put it in the system path
movecmd = 'mv protoc3/bin/* /usr/local/bin/'
# !sudo $movecmd

# Add header files to the linker's include path
movecmd = 'mv protoc3/include/* /usr/local/include/'
# !sudo $movecmd

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082578604} tags=[] id="mXRSTPc5ycHR"
# Jump into the tensorflow object detection API research directory
researchfolder =  modelroot + '/models/research'
# %cd $researchfolder

# As per their installation instructions, compile everything in the protos folder using protoc
protoccmd = '/usr/local/bin/protoc ' + 'object_detection/protos/*.proto --python_out=.'
# !$protoccmd

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082596459} tags=[] id="9X5ZxpG9ycHU"
# Check Tensorflow version. OpenVINO currently requires TF 1.x, so that's what we use
import tensorflow.compat.v1 as tf
print(tf.__version__)
print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082596625} id="aK1s2jQcycHW"
# Note: Cycle jupyter kernel if needed to pickup tensorflow change from 2.1 to 1.15.0, then run the previous cell again.

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082728542} tags=["outputPrepend"] id="gR--fg0FycHZ"
# Run setup for TF object detection API
args = researchfolder + '/setup.py build'
# !python $args

# Install TF object detection API
args = researchfolder + '/setup.py install'
# !python $args

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082743612} tags=[] id="a7_gKdE8ycHc"
# Run tensorflow model builder test just to make sure we have TF object detection API set up correctly
args = modelroot + '/models/research/object_detection/builders/model_builder_test.py'
# !python $args

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599082744040} tags=[] id="ErHYS8EzycHf"
# Create data folder for training dataset and model
# %cd $modelroot
# !mv $modelroot/bowls_voc $modelroot/data
# %cd data

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599083076606} tags=[] id="yOXAG2m5ycHi"
# This is a little wonky, but we are later on using a script from the TF object detection API,
# create_pet_tf_record.py to generate a TF record. This script requires certain things in the
# dataset that we don't have, and which we don't use. We create those things here.
from PIL import Image

image_files = os.listdir('images')
im_files = [os.path.splitext(x)[0] for x in image_files]
with open('annotations/trainval.txt', 'w') as text_file:
  for row in im_files:
    text_file.write(row + '\n')

# %cd ./annotations
# !mkdir trimaps

image = Image.new('RGB', (640, 480))
for fname in os.listdir("xmls"):
  fname, _ = os.path.splitext(fname)
  image.save(os.path.join("trimaps", fname + ".png"))

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599083076894} tags=[] id="W5kFF7RqycHk"
# Create category labels file for Tensorflow training and validation files (label map)
label_map_fpath = modelroot + '/models/research/object_detection/data/bowls.pbtxt'
print(label_map_fpath)

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} tags=[] id="5ZhbS43cycHm"
# %%writefile $label_map_fpath
item {
  id: 1
  name: 'bowl'
}

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599083077182} id="5MZ8Ydd8ycHo"
# Fix to remove ^M nonprintable char from end of string lines in file created above
# to see issue run "!cat -v $label_map_fname" before and after fix
with open(label_map_fpath, 'r') as file:
    label_map_file = file.read()
update_file = open(label_map_fpath, "w")
update_file.writelines(label_map_file)
update_file.close() 

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599083819340} tags=[] id="wvWWs-pyycHp"
# Create Tensorflow training data record files by co-opting this create_pet_tf_record.py script for our purposes
# %cd $modelroot/data

script = modelroot + "/models/research/object_detection/dataset_tools/create_pet_tf_record.py"
args = f"{script} --label_map_path={label_map_fpath} --data_dir=./ --output_dir=./ --num_shards=1"
# !python $args

# Now update the names of the output files
# !mv pet_faces_train.record-00000-of-00001 tf_train.record
# !mv pet_faces_val.record-00000-of-00001 tf_val.record

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599083820953} tags=[] id="Thi00zC6ycHr"
# Download pretrained model for transfer learning: SSD Lite MobileNet V2 COCO
# !curl -OL 'http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz'
model_file = os.path.join(modelroot, "data", "ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz")

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599083823171} id="TrMLAseTycHt"
# Uncompress model
import os
import shutil
import glob
import urllib
import tarfile
import urllib.request

tar = tarfile.open(model_file)
tar.extractall()
tar.close()

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599083823456} id="oA91GdwXycHv"
# Prepare model config file paths and tensorflow configuration file for SSDLiteV2 Model
config_fname = modelroot + '/models/research/object_detection/samples/configs/ssdlite_mobilenet_retrained.config'
fine_tune_checkpoint = '"' + modelroot + '/data/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt' + '"'
input_path_train = '"' + modelroot + '/data/tf_train.record' + '"'
label_map_path = '"' + modelroot + '/models/research/object_detection/data/bowls.pbtxt' + '"'
input_path_eval = '"' + modelroot + '/data/tf_val.record' + '"'

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} tags=[] id="goeXNX__ycHx"
# %%writefile $config_fname
model {
  ssd {
    num_classes: 1
    box_coder {
      faster_rcnn_box_coder {
        y_scale: 10.0
        x_scale: 10.0
        height_scale: 5.0
        width_scale: 5.0
      }
    }
    matcher {
      argmax_matcher {
        matched_threshold: 0.5
        unmatched_threshold: 0.5
        ignore_thresholds: false
        negatives_lower_than_unmatched: true
        force_match_for_each_row: true
      }
    }
    similarity_calculator {
      iou_similarity {
      }
    }
    anchor_generator {
      ssd_anchor_generator {
        num_layers: 6
        min_scale: 0.2
        max_scale: 0.95
        aspect_ratios: 1.0
        aspect_ratios: 2.0
        aspect_ratios: 0.5
        aspect_ratios: 3.0
        aspect_ratios: 0.3333
      }
    }
    image_resizer {
      fixed_shape_resizer {
        height: 300
        width: 300
      }
    }
    box_predictor {
      convolutional_box_predictor {
        min_depth: 0
        max_depth: 0
        num_layers_before_predictor: 0
        use_dropout: false
        dropout_keep_probability: 0.8
        kernel_size: 3
        use_depthwise: true
        box_code_size: 4
        apply_sigmoid_to_scores: false
        conv_hyperparams {
          activation: RELU_6,
          regularizer {
            l2_regularizer {
              weight: 0.00004
            }
          }
          initializer {
            truncated_normal_initializer {
              stddev: 0.03
              mean: 0.0
            }
          }
          batch_norm {
            train: true,
            scale: true,
            center: true,
            decay: 0.9997,
            epsilon: 0.001,
          }
        }
      }
    }
    feature_extractor {
      type: 'ssd_mobilenet_v2'
      min_depth: 16
      depth_multiplier: 1.0
      use_depthwise: true
      conv_hyperparams {
        activation: RELU_6,
        regularizer {
          l2_regularizer {
            weight: 0.00004
          }
        }
        initializer {
          truncated_normal_initializer {
            stddev: 0.03
            mean: 0.0
          }
        }
        batch_norm {
          train: true,
          scale: true,
          center: true,
          decay: 0.9997,
          epsilon: 0.001,
        }
      }
    }
    loss {
      classification_loss {
        weighted_sigmoid {
        }
      }
      localization_loss {
        weighted_smooth_l1 {
        }
      }
      hard_example_miner {
        num_hard_examples: 3000
        iou_threshold: 0.99
        loss_type: CLASSIFICATION
        max_negatives_per_positive: 3
        min_negatives_per_image: 3
      }
      classification_weight: 1.0
      localization_weight: 1.0
    }
    normalize_loss_by_num_matches: true
    post_processing {
      batch_non_max_suppression {
        score_threshold: 1e-8
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SIGMOID
    }
  }
}

train_config: {
  batch_size: 24
  optimizer {
    rms_prop_optimizer: {
      learning_rate: {
        exponential_decay_learning_rate {
          initial_learning_rate: 0.004
          decay_steps: 800720
          decay_factor: 0.95
        }
      }
      momentum_optimizer_value: 0.9
      decay: 0.9
      epsilon: 1.0
    }
  }
  fine_tune_checkpoint: $fine_tune_checkpoint
  fine_tune_checkpoint_type:  "detection"
  # Note: The below line limits the training process to 200K steps, which we
  # empirically found to be sufficient enough to train the pets dataset. This
  # effectively bypasses the learning rate schedule (the learning rate will
  # never decay). Remove the below line to train indefinitely.
  num_steps: 200000
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  data_augmentation_options {
    ssd_random_crop {
    }
  }
}

train_input_reader: {
  tf_record_input_reader {
    input_path: $input_path_train
  }
  label_map_path: $label_map_path
}

eval_config: {
  num_examples: 8000
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 10
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: $input_path_eval
  }
  label_map_path: $label_map_path
  shuffle: false
  num_readers: 1
}

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599083823786} id="LPGdzTCMycHz"
# Update config file paths
with open(config_fname, 'r') as f:
    config_file = f.read()
config_file = config_file.replace('$fine_tune_checkpoint', fine_tune_checkpoint)
config_file = config_file.replace('$label_map_path', label_map_path)
config_file = config_file.replace('$input_path_train', input_path_train)
config_file = config_file.replace('$input_path_eval', input_path_eval)
update_file = open(config_fname, 'w')
update_file.writelines(config_file)
update_file.close() 


# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599081136949} tags=["outputPrepend"] id="Qzl7wPh7ycH1"
# Run tensorflow fine tuning training.
# To train on CPU instead of GPU uncomment the 'CUDA_VISIBLE_DEVICES' line below
# If training for more than 10,000 iterations the notebook will stop updating output after a while, but the training is still happening in the background
# In that case run !nvidia-smi to check the current load on the GPU to see if training is still happening
# You can watch for new ckeckpoint files being written about every 10 minutes with the command !ls $modelroot/data/retrained
#
# Note that we train for 30k steps here. This is sufficient to get reasonable results, though you may prefer to train for longer.
# %cd $modelroot

model_dir = os.path.join(modelroot, "data", "retrained")
# !mkdir $model_dir

import os
# os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
# !python ./models/research/object_detection/model_main.py --pipeline_config_path=$config_fname --model_dir=$model_dir --alsologtostderr --num_train_steps=30000 --num_eval_steps=30000

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599073934210} tags=[] id="Kg6XeDwiycH4"
# Get the last checkpoint to use for exporting a frozen graph
import re

# %cd $modelroot/data

lst = os.listdir('retrained')
lf = filter(lambda k: 'model.ckpt-' in k, lst)
fileList = str(list(lf))
checkPointNumbers = re.findall(r'[0-9]+', fileList)
checkPointNumbers = [int(i) for i in checkPointNumbers]  
last_model = 'model.ckpt-' + str(max(checkPointNumbers))
print(last_model)

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599073987537} tags=["outputPrepend"] id="EtaYwsDmycH5"
# Export frozen graph
# !python $modelroot/models/research/object_detection/export_inference_graph.py --input_type=image_tensor --pipeline_config_path=$config_fname --output_directory=fine_tuned_model --trained_checkpoint_prefix=retrained/$last_model

# + tags=[] gather={"logged": 1599074014829} id="nm6HYqq2ycH7"
# Test inference on photo using frozen graph
import tensorflow.compat.v1 as tf
import numpy as np
import os
import cv2
from object_detection.utils import label_map_util
from object_detection.utils import visualization_utils as vis_util
from matplotlib import pyplot as plt
from PIL import Image

# Path to the frozen graph:
PATH_TO_FROZEN_GRAPH = modelroot + '/data/fine_tuned_model/frozen_inference_graph.pb'

# Path to the label map
PATH_TO_LABEL_MAP = modelroot + '/models/research/object_detection/data/bowls.pbtxt'

# Bowl image
# Note that because the train/eval split is randomized, it is possible this image was in the train split.
IMAGE_PATH = modelroot + '/data/images/bowl_000000084259.jpg'

# Number of classes 
NUM_CLASSES = 1

# Minimum confidence value needed to display the bounding box on the image. In range [0.0, 1.0].
MIN_THRESHOLD = 0.40

# Read the frozen graph
detection_graph = tf.Graph()
with detection_graph.as_default():
    od_graph_def = tf.GraphDef()
    with tf.gfile.GFile(PATH_TO_FROZEN_GRAPH, 'rb') as fid:
        serialized_graph = fid.read()
        od_graph_def.ParseFromString(serialized_graph)
        tf.import_graph_def(od_graph_def, name='')

label_map = label_map_util.load_labelmap(PATH_TO_LABEL_MAP)
categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
category_index = label_map_util.create_category_index(categories)

# Detection
with detection_graph.as_default():
    with tf.Session(graph=detection_graph) as sess:
        # Read in the image and convert BGR to RGB
        image_np = cv2.imread(IMAGE_PATH)[:,:,::-1]

        # Expand dimensions since the model expects images to have shape: [1, None, None, 3] 
        image_np_expanded = np.expand_dims(image_np, axis=0)

        # Extract image tensor
        image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
        
        # Extract detection boxes
        boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
        
        # Extract detection scores
        scores = detection_graph.get_tensor_by_name('detection_scores:0')
        
        # Extract detection classes
        classes = detection_graph.get_tensor_by_name('detection_classes:0')
        
        # Extract number of detections
        num_detections = detection_graph.get_tensor_by_name('num_detections:0')

        # Actual detection.
        boxes, scores, classes, num_detections = sess.run([boxes, scores, classes, num_detections], feed_dict={image_tensor: image_np_expanded})
        print(f"BOXES (shaped {boxes.shape}):\n{boxes}")
        print(f"SCORES (shaped {scores.shape}):\n{scores}")
        print(f"CLASSES (shaped {classes.shape}):\n{classes}")
        print(f"NDETECTIONS (shaped {num_detections.shape}):\n{num_detections}")

        # Visualization of the results of a detection.
        vis_util.visualize_boxes_and_labels_on_image_array(
            image_np,
            np.squeeze(boxes),
            np.squeeze(classes).astype(np.int32),
            np.squeeze(scores),
            category_index,
            use_normalized_coordinates=True,
            line_thickness=3,
            min_score_thresh=MIN_THRESHOLD
            )

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1599074015582} id="T2OLLBMXycH9"
# Display test inference photo
from matplotlib.pyplot import imshow
import numpy as np
from PIL import Image

# %matplotlib inline
plt.figure(figsize = (12, 8))
imshow(image_np)

# + [markdown] nteract={"transient": {"deleting": false}} id="9OrR0JwTycH-"
# ## OpenVINO
#
# At this point, you should have a model exported from Tensorflow and it should be providing good bounding boxes on bowls.
# The rest of this notebook will show you how to convert the model into the format that the EyeSOM dev kit requires,
# and then how to download it to your device.

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} tags=[] gather={"logged": 1598986131005} id="gYmoxuZkycH_"
from azureml.core.authentication import InteractiveLoginAuthentication
interactive_auth = InteractiveLoginAuthentication(tenant_id="<YOUR TENANT ID>")

# If you have an Azure config.json in your workspace, you can use this line
#ws = Workspace.from_config(auth=interactive_auth)

# Otherwise, you can manually fill in your information and use the next lines
ws = Workspace(subscription_id="<YOUR SUBSCRIPTION ID>",
                resource_group="<YOUR RESOURCE GROUP>",
                workspace_name="<YOUR WORKSPACE NAME>",
                auth=interactive_auth)

ws.get_details()
print(ws)

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598986351563} id="-RIkr1JNycIB"
# Reload workspace to register model
from azureml.core import Workspace
ws.get_details()

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598986362581} id="Lj2Qxn-RycIC"
# Register the trained model. Once registered, you'll find the model in the Models section on the left pane
from azureml.core.model import Model

trained_model_path = modelroot + '/data/fine_tuned_model/'
print("Trained model path:", trained_model_path)

model = Model.register(model_path=trained_model_path,
                      model_name="bowl_ssdv2lite",
                      tags={"data": "ssd_mobilenetv2lite", "model": "object_detection", "type": "ssd_mobilenetv2lite"},
                      description="Retrained bowl detection based on ssd_mobilenetv2lite",
                      workspace=ws)

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598986600563} id="CtzZnLRyycIE"
# Install Intel® Distribution of OpenVINO™ toolkit
# https://docs.openvinotoolkit.org/2019_R2/_docs_install_guides_installing_openvino_apt.html
frozen_inference_graph = modelroot + '/data/fine_tuned_model/frozen_inference_graph.pb'

# !curl -OL -o GPG-PUB-KEY-INTEL-OPENVINO-2020 'https://apt.repos.intel.com/openvino/2020/GPG-PUB-KEY-INTEL-OPENVINO-2020'

aptKey = 'GPG-PUB-KEY-INTEL-OPENVINO-2020'
# !sudo apt-key add $aptKey

# !sudo apt-key list

echo_args = '"deb https://apt.repos.intel.com/openvino/2020 all main" | sudo tee /etc/apt/sources.list.d/intel-openvino-2020.list'
# !echo $echo_args

# !sudo apt update

# Make sure everything worked. You should see a bunch of intel-openvino stuff in the apt packages
# !apt-cache search openvino
# !sudo apt-cache search intel-openvino-dev-ubuntu16

package = 'intel-openvino-dev-ubuntu16-2020.2.130'
# !sudo apt install -y $package

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598978236673} id="RY4D9oKjycIF"
#Use OpenVINO to converto frozen graph PB to IR format
# %cd $modelroot/data

#import OpenVINO env vars
convertCMD = 'source /opt/intel/openvino/bin/setupvars.sh && '

#convert frozen graph to Intermediate Representation
convertCMD += '/opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py '
convertCMD += '--input_model ' + modelroot + '/data/fine_tuned_model/frozen_inference_graph.pb '
convertCMD += '--tensorflow_object_detection_api_pipeline_config ' + modelroot + '/data/fine_tuned_model/pipeline.config '
convertCMD += '--tensorflow_use_custom_operations_config /opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/ssd_v2_support.json '
convertCMD += '--reverse_input_channels'

print(convertCMD)
# !$convertCMD

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598978244502} id="s26sD1r2ycIH"
# Use OpenVINO to compile IR format to blob

# Import OpenVINO env vars
convertCMD = 'source /opt/intel/openvino/bin/setupvars.sh && '

# Compile IR to blob 
convertCMD += '/opt/intel/openvino/deployment_tools/inference_engine/lib/intel64/myriad_compile '
convertCMD += '-m frozen_inference_graph.xml '
convertCMD += '-o ssdlite_mobilenet_v2.blob '
convertCMD += '-VPU_MYRIAD_PLATFORM VPU_MYRIAD_2480 '
convertCMD += '-VPU_NUMBER_OF_SHAVES 8 '
convertCMD += '-VPU_NUMBER_OF_CMX_SLICES 8 '
convertCMD += '-ip U8 '
convertCMD += '-op FP32'

print(convertCMD)
# !$convertCMD

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598978258187} id="KQ6pUtBuycIJ"
# Package up blob for delevery to devkit

# Clean recreate directory if it exists and create it if not 
# !rm -rf blob
# mkdir blob

# !cp ssdlite_mobilenet_v2.blob blob/ssdlite_mobilenet_v2.blob

# %cd blob

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598986928982} id="mYXTlazZycIK"
# Create labels file for devkit
# Do not remove zeroindex, it is needed to align graph classification index with lables index
labelFileName = 'labels.txt'

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} id="Z81X4pZLycIN"
# %%writefile $labelFileName
zeroindex
bowl

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598978311133} id="qBbWOwd7ycIP"
# Fix to remove ^M nonprintable char from end of string lines in file created above
# to see issue run "!cat -v $labelFileName" before and after fix
with open(labelFileName,'r') as file:
    labelFile = file.read()
updateFile = open(labelFileName,"w")
updateFile.writelines(labelFile)
updateFile.close() 

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598986939934} id="PXjm-vUCycIR"
# Create json file for devkit
jsonConfigFileName = 'config.json'

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} id="g6WcJ0dnycIT"
# %%writefile $jsonConfigFileName
{
    "DomainType": "ssd100",
    "LabelFileName": "labels.txt",
    "ModelFileName": "ssdlite_mobilenet_v2.blob"
}

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598978363734} id="StZoUOyJycIU"
# Fix to remove ^M nonprintable char from end of string lines in file created above
# to see issue run "!cat -v $jsonConfigFileName" before and after fix
with open(jsonConfigFileName,'r') as file:
    jsonConfigFile = file.read()
updateFile = open(jsonConfigFileName,"w")
updateFile.writelines(jsonConfigFile)
updateFile.close() 

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598978378986} id="-d63Jp9mycIW"
# Package up model and support files for dev kit
# !zip -r model.zip ./*

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598986950202} id="hj1KzqDwycIX"
#Reload workspace details for module twin update
from azureml.core import Workspace
ws.get_details()

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598986960142} id="jku--5pYycIZ"
# Get the default datastore
ds = ws.get_default_datastore()
print(ds.name, ds.datastore_type, ds.account_name, ds.container_name)

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598978411086} id="JbZfvEz5ycIb"
# Set data path for model.zip and upload
data_path = 'model'
ds.upload(src_dir='.', target_path=data_path, overwrite=True)

# + id="_yu2sDVSycIc"
# !pip install azure-storage-blob==2.1.0
# !pip install msrest

# + id="C2__CKYDycId"
#Use one of the options to generate Saas Url

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} gather={"logged": 1598986973636} id="Z5g5B8wsycIf"
# Option 1- Generate download SAS URL for model.zip
from datetime import datetime, timedelta
from azure.storage.blob import (
    BlockBlobService,
    ContainerPermissions,
    BlobPermissions,
    PublicAccess,
)

AZURE_ACC_NAME = ds.account_name
AZURE_PRIMARY_KEY = ds.account_key
AZURE_CONTAINER = ds.container_name
AZURE_BLOB=ds.name
AZURE_File=data_path+'/model.zip'

block_blob_service = BlockBlobService(account_name=AZURE_ACC_NAME, account_key=AZURE_PRIMARY_KEY)
sas_url = block_blob_service.generate_blob_shared_access_signature(AZURE_CONTAINER,AZURE_File,permission=BlobPermissions.READ,expiry= datetime.utcnow() + timedelta(hours=360))
downloadurl ='https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+AZURE_File+'?'+sas_url
print('https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+AZURE_File+'?'+sas_url)
print(sas_url)

# + id="ViK0W96yycIh"
# Option 2- Generate download SAS URL for model.zip
from azure.storage.blob.baseblobservice import BaseBlobService,ContainerPermissions,BlobPermissions
from datetime import datetime, timedelta
AZURE_ACC_NAME = ds.account_name
AZURE_PRIMARY_KEY = ds.account_key
AZURE_CONTAINER = ds.container_name
AZURE_BLOB=ds.name
AZURE_File=data_path+'/model.zip'
service = BaseBlobService(account_name=AZURE_ACC_NAME, account_key=AZURE_PRIMARY_KEY)
sas_url  = service.generate_blob_shared_access_signature(AZURE_CONTAINER,AZURE_File,permission=BlobPermissions.READ,expiry= datetime.utcnow() + timedelta(hours=48))
downloadurl ='https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+AZURE_File+'?'+sas_url
print('https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+AZURE_File+'?'+sas_url)
print(sas_url)

# + id="BYwUm8bPycIk"
# Perform Module twin update
# Incorporate the connection string, device_id and the module_id values from your IoTHub

# !pip install azure-iot-hub
print(downloadurl)
import sys
from azure.iot.hub import IoTHubRegistryManager
from azure.iot.hub.models import Twin, TwinProperties

# Incorporate Iothub connection string and the default module name
# Go to Https://portal.azure.com
# Select your IoTHub
# Click on Shared access policies
# Click service on right
# Copy the iothub connection string primary key

CONNECTION_STRING = '<ENTER YOUR CONNECTION STRING>'
DEVICE_ID = '<ENTER YOUR DEVICE NAME>'
MODULE_ID = "azureeyemodule"

try:
    # RegistryManager
    iothub_registry_manager = IoTHubRegistryManager(CONNECTION_STRING)

    module_twin = iothub_registry_manager.get_module_twin(DEVICE_ID, MODULE_ID)
    print ( "" )
    print ( "Module twin properties before update    :" )
    print ( "{0}".format(module_twin.properties) )

    # Update twin
    twin_patch = Twin()
    twin_patch.properties = TwinProperties(desired={"ModelZipUrl": downloadurl})
    updated_module_twin = iothub_registry_manager.update_module_twin(
        DEVICE_ID, MODULE_ID, twin_patch, module_twin.etag
    )
    print ( "" )
    print ( "Module twin properties after update     :" )
    print ( "{0}".format(updated_module_twin.properties) )

except Exception as ex:
    print ( "Unexpected error {0}".format(ex) )
except KeyboardInterrupt:
    print ( "IoTHubRegistryManager sample stopped" )

# + jupyter={"source_hidden": false, "outputs_hidden": false} nteract={"transient": {"deleting": false}} id="bRR2l3MDycIm"
# The trained model will get pushed to the IoT Edge device via module twin update method
# Check model inferencing by connecting monitor to the devkit or by installing VLC media player : 
#Install VLC from https://www.videolan.org/vlc/ and install on “Windows” to check the camera function of “Azure Eye”.

#Check video stream:
#1.	Select Media -> Open Network Stream…
#2.	Input the network stream: “rtsp://[ip of PE-101]:8554/result” then click “Play” button.
#3. or use webstream http://<ipaddressofcamera>:3000

StarcoderdataJupyterScriptsDedupFiltered