# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Learning how to run benchmarks on DevCloud
# OpenVINO includes a lot of sample and demo programs in the package. The most of sample programs and demo programs are intended to demonstrate the capability and functionality of the OpenVINO, or a simple sample program to demonstrate how to use the OpenVINO APIs to the users. **They are not designed to be used for performance benchmarking.**  
# Among those sample applications, `benchmark_app` sample application is developed and specialized for benchmaking.   
# You can specify many option parameters to the `benchmark_app` but it will choose the appropriate configuration for the specified hardware when you ommit those option parameters.  
# - Automatically choose and apply appropriate option parameters (note: may not be the best)
# - Use asynchronous and simultaneous inferencing for higher throughput
# - Almost all DL models can be used
#  - benchmark_app won't care the input and output of the model. It purely run the model and measure the performance of the processing time
#  - Output data won't be cared nor parsed. benchmark_app won't care the number of outputs and format of the outputs
#  - benchmark_app doesn't require any input data. It won't care the shape, format and number of inputs
# - Both C++ version and Python version are available
#
# #### Here, we'll use the Python version of Benchmark tool and learn how to run benchmarking on the DevCloud

# In this section, you'll learn following things:
# - How to run a benchmark_app on the **host server system**
# - How to submit a job and run benchmark_app on **an edge computing node**
# - How to submit a job and run benchmark_app on **a specific edge computing node**

# ----
# ## 1. Learn how to run benchmark_app on the development server
# Here, we'll use the Python version of benchmark_app. It is convenient because it doesn't require compilation to use. (C++ version is available too)  
# It requires a deep learning model to run. We'll use the `Squeezenet1.1` model for this exercise.

# ### 1.1. Copy benchmark_app
# `benchmark_app.py` is in the OpenVINOの install directory. For convenience, copy the files to the current directory.

# !pwd
# !cp -r $INTEL_OPENVINO_DIR/deployment_tools/tools/benchmark_tool .
# !ls -l benchmark_tool

# ### 1.2. Install required Python modules for the benchmark_app
# Installs Python modules described in `reqirements.txt` file using `pip3` command. This is requied only the first time you do this.

# !pip3 install -r benchmark_tool/requirements.txt

# ### 1.3. Download a neural network model
# Downloads `squeezenet 1.1` DL model using an OpenVINO utility tool `Model downloader`. The `Model downloader` downloads the specified DL models. You can list the available models with `--print_all` option to the `Model downloader`.  
# The downloaded model is in the Caffe format. **You need to convert the model into IR model** which is the DL model format OpenVINO can read.

# Download a Squeezenet1.1 model (Caffe format)
# !python3 $INTEL_OPENVINO_DIR/deployment_tools/tools/model_downloader/downloader.py --name squeezenet1.1

# (optional) List all available models with Model downloader (OMZ=Open Model Zoo, OpenVINO model zoo)
# !python3 $INTEL_OPENVINO_DIR/deployment_tools/tools/model_downloader/downloader.py --print_all

# ### 1.4. Convert the DL model from Caffe formato to IR format
# In general, `Model Optimizer` is the tool to convert the trained DL models into OpenVINO IR model format.  
# However, the DL model downloaded with `Model downloader` can be converted into IR model with `Model converter`. The `Model converter` is a front-end tool for the `Model optimizer` and it will call `Model optimizer` with appropriate option parameters for the model.

# !python3 $INTEL_OPENVINO_DIR/deployment_tools/tools/model_downloader/converter.py --name squeezenet1.1 --precisions FP16

# ### 1.5. Check the converted IR model files
# IR model consists of 2 files, `.xml` and `.bin`. `.xml` represents the topology of the model (graph) and `.bin` contains weight and bias data of the model.

# !ls -l public/squeezenet1.1/FP16

# ### 1.6. Running benchmark_app on the development server
# First, let's try to run the benchmark_app on the **development server**. The development server is the machine which you are currently operating. It's a Xeon server and doesn't have neither integrated GPU nor DL accelerators such as `VPU` or `FPGA`. The `CPU` is the only available processor on the machine.    
# You can directly run the job on the development server without submitting a job. The job won't be kept waiting in the queue. It's simple as you run an OpenVINO application on the local machine.  
#
# Run the benchmark_app and check the performance result on the bottom of the log.

# benchmark_app options:  
# - `-m` specify the IR model file (.xml)
# - `-niter` specify the number of inference in the benchmarking

# !python3 benchmark_tool/benchmark_app.py \
#         -m public/squeezenet1.1/FP16/squeezenet1.1.xml \
#         -niter 100

# ----
# ## 2. Running benchmark_app on an edge inference node
# Send a benchmark job and run it on an edge compute node

# ### 2.1 Create a job script file to submit  
# Use `%%writefile` magic command to create a job script file which contains the tasks you want to run. `%%writefile` magic command will write the rest of contents in the cell to a file.

# %%writefile job.sh
# cd ~/devcloud-workshop-en
pip3 install -r benchmark_tool/requirements.txt
python3 benchmark_tool/benchmark_app.py \
        -m public/squeezenet1.1/FP16/squeezenet1.1.xml \
        -niter 100 \
        $*
# echo completed


# ### 2.2 Send a job script to an edge computing node
# Here, we re-use the same Python code which we used in previous section to detect the completion of the job.  
# The job submitted with `qsub` command will be run on one of available edge computing node adhocery.

# +
# submit a job
# job_id=!qsub job.sh

# generate log file name from job_id
job_num = job_id[0].split('.')[0]
log_file='job.sh.o'+job_num
err_file='job.sh.e'+job_num
print('job_id={}, log_file={}'.format(job_id, log_file))

import time
def waitForJobCompletion(jobNumber):
    print('Waiting for job completion...', end='')
    running=True
    while running:
        time.sleep(1)
        running=False
        # status_list=!qstat         # Check job status
        for status in status_list:
            if jobNumber in status:    # if job_num is found in the status list, the job is still running
                running = True
        print(status.split()[4], end='')
    print('...Job {} completed'.format(job_num))    

# wait for the job to complete
waitForJobCompletion(job_num)
# -

# ### 2.3 Check the log file
# The name of the node which run your job is displayed in `Resources:` line in the log file. Check which node is used to run the job.

# +
# The log_file is set in the previous cell
import os
os.environ['log_file']=log_file

# !cat $log_file
# -

# ----
# ## 3. Running a job on a specific edge compute node
# In previous section, you run a job without specifying the edge compute node. In this case, the job will be run on one of available edge computing node. You need to control the edge computing node to run your job when you run a benchmark.  
# In the DevCloud, wide variety of edge computing node with different hardware configuration are available. From Atom to Xeon, and various accelerators such as VPU and FPGA are available.  
# To display the list of available edge computing node, you can use `pbsnodes` command. The node name you need to specify the node can be found in the line of `properties =` in the output of the `pbsnodes` command.  
# The node name is displayed as a comma-separated list. For instance, your job will be one of edge computing node which contains `intel-hd-530` in the property field when you specify `intel-hd-530` as the node name to run your job. You can specify more specific hardware with more specific and dedicated node name. For example, if you specify `idc001skl` as the node name, your job will be run exactly on that node.

# ### 3.1 Display available edge computing nodes
# First, let's list the available edge computing nodes. The `pbsnodes` command will list the nodes.

# !pbsnodes | grep "properties =" | sort | uniq -c

# ### 3.2 Submit a benchmarking job to a specific edge computing node
# Using `qsub` command and specifying the compute node name with `-l` to submit a benchmarking job. In addition, using `-F` option to pass command option parameters to the job script file.
# ~~~
# # job_id=!qsub -l nodes=1:gold6138 job.sh -F "-d CPU"
# ~~~
# - `qsub` command
#  - You can use `-l` (limit) option to specify the edge computing node to run your job in `qsub` command. For example, if you want to run the job on a `skylake` node, you can specify it like this way `qsub -l nodes=1:skylake job.sh`
#  - You can pass parameters to the job script file with `-F` option. Please be aware that you have to make sure that the job script file can receive the parameters  
# - We'll re-use the `job.sh` job script file which we have created in the previous section.
# - Log file name
#  - Here, we'll auto generate the job file name using a Python snippet from the `job ID` (a list of string like `['27214.v-qsvr-1.devcloud-edge']`) returned from the `qsub` command
# - Job completion detection
#  - You run `qstat` command multiple times manually to know the completion of a job in previous section. Here, we'll use a Python snippet to detect it automatically

# #### Note: benchmark_app option parameters
# benchmark_app has lot of option parameters. You can check available options with `--help` option. Following are the typical options for benchmark_app.  
# Here, we specifies `CPU` as the processor to run the benchmark with `"-d CPU"` but you can specify other accelerators such as integrated GPU (`"GPU"`), VPU (`"MYRIAD"`) and so on. You have to make sure that the specified edge computing node supports the processor you specified.

# |Option|Description|
# |:--|:--|
# |`-m` PATH_TO_MODEL|Specifies IR model to run (`.xml`)|
# |`-d` TARGET_DEVICE|Specified processor to run inference. `CPU`, `GPU`, `MYRIAD`, `HDDL`, `HETERO:FPGA,CPU` and some others are available|
# |`-niter` NUMBER_ITERATIONS|Number of inference to run. If you ommit this, benchmark will run for 1 min|
# |`-nireq` NUMBER_INFER_REQUESTS|Number of inference simultaneously. If you specifies 4, 4 inference request will run at a time. To achieve higher throughput, you need to specify appropriate number of simultaneous inferencing. If you ommit this, benchmark_app will use appropriate number for  the inference processor.|
# |`-b` BATCH_SIZE|number of batch in an inferencing|
# |`-i` PATH_TO_INPUT|Specify the input image. You can ommit this.|
# |`-pc`|Enable performance counter. You can get the detailed layer by layer performance report|

# +
# submit a job
# job_id=!qsub -l nodes=1:gold6138 job.sh -F "-d CPU"

# generate log file name from job_id
print('job_id=', job_id)
job_num = job_id[0].split('.')[0]
log_file='job.sh.o'+job_num
err_file='job.sh.e'+job_num
print('log_file=', log_file)

# wait for the job to complete
waitForJobCompletion(job_num)
# -

# ### 3.3 Check the job result
# We just displayed entire output log in the previous section. We'll use grep and display the minimum required information to make it concise.

# +
os.environ['log_file']=log_file

# !grep '# Resources:'                                  $log_file
# !grep '\[ INFO \] Device info'                   -A 3 $log_file
# !grep '\[Step 11/11\] Dumping statistics report' -A 4 $log_file
# -

# ----
# Now you have learnt how to run benchmark_app in several ways in the DevCloud for Edge.
# The benchmark_app can run any model and take benchmark. You can upload your own model and try it easily.
# The storage of DevClous is isolated and protected. You can upload confidential data and it's secure.
#
# ## Next => [How to develop a C++ project using OpenVINO and run it on DevCloud](./cpp-project.ipynb>Cloud)

StarcoderdataJupyterScriptsDedupFiltered