
**task**: `image-classification`  
**Backend:** `sagemaker-training`  
**Backend args:** `{'instance_type': 'ml.m5.2xlarge', 'supported_instructions': 'avx512'}`  
**Number of evaluation samples:** `All dataset`  

Fixed parameters:
* **dataset**: [{'path': 'beans', 'eval_split': 'validation', 'data_keys': {'primary': 'image'}, 'ref_keys': ['labels'], 'name': None, 'calibration_split': 'train'}]
* **name_or_path**: `nateraw/vit-base-beans`
* **from_transformers**: `True`
* **node_exclusion**: `[]`
* **calibration**:
    * **method**: `percentile`
    * **num_calibration_samples**: `128`
    * **calibration_histogram_percentile**: `99.999`

Benchmarked parameters:
* **framework**: `onnxruntime`,  `pytorch`
* **quantization_approach**: `dynamic`,  `static`
* **operators_to_quantize**: `['Add', 'MatMul']`,  `['Add']`
* **per_channel**: `False`,  `True`
* **framework_args**: `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}`,  `{}`
* **reduce_range**: `True`,  `False`
* **apply_quantization**: `True`,  `False`

# Evaluation
## Non-time metrics
|   framework   | quantization_approach | operators_to_quantize | per_channel |                           framework_args                            | reduce_range | apply_quantization |     | accuracy |
| :-----------: | :-------------------: | :-------------------: | :---------: | :-----------------------------------------------------------------: | :----------: | :----------------: | :-: | :------: |
| `onnxruntime` |        `None`         |        `None`         |   `None`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `None`    |      `False`       |  \|  |  0.977   |
| `onnxruntime` |       `dynamic`       |  `['Add', 'MatMul']`  |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |  0.977   |
| `onnxruntime` |       `dynamic`       |  `['Add', 'MatMul']`  |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |  0.977   |
| `onnxruntime` |       `dynamic`       |  `['Add', 'MatMul']`  |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |  0.977   |
| `onnxruntime` |       `dynamic`       |  `['Add', 'MatMul']`  |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |  0.977   |
| `onnxruntime` |       `dynamic`       |       `['Add']`       |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |  0.977   |
| `onnxruntime` |       `dynamic`       |       `['Add']`       |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |  0.977   |
| `onnxruntime` |       `dynamic`       |       `['Add']`       |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |  0.977   |
| `onnxruntime` |       `dynamic`       |       `['Add']`       |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |  0.977   |
| `onnxruntime` |       `static`        |  `['Add', 'MatMul']`  |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |  0.421   |
| `onnxruntime` |       `static`        |  `['Add', 'MatMul']`  |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |  0.421   |
| `onnxruntime` |       `static`        |  `['Add', 'MatMul']`  |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |  0.316   |
| `onnxruntime` |       `static`        |  `['Add', 'MatMul']`  |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |  0.451   |
| `onnxruntime` |       `static`        |       `['Add']`       |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |  0.361   |
| `onnxruntime` |       `static`        |       `['Add']`       |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |  0.361   |
| `onnxruntime` |       `static`        |       `['Add']`       |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |  0.361   |
| `onnxruntime` |       `static`        |       `['Add']`       |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |  0.361   |
|   `pytorch`   |        `None`         |        `None`         |   `None`    |                                `{}`                                 |    `None`    |       `None`       |  \|  |  0.977   |

## Time metrics
Time benchmarks were run for 15 seconds per config.


Below, time metrics for batch size = 1, input length = 224.

|   framework   | quantization_approach | operators_to_quantize | per_channel |                           framework_args                            | reduce_range | apply_quantization |     | latency_mean (ms) |     | throughput (/s) |
| :-----------: | :-------------------: | :-------------------: | :---------: | :-----------------------------------------------------------------: | :----------: | :----------------: | :-: | :---------------: | :-: | :-------------: |
| `onnxruntime` |        `None`         |        `None`         |   `None`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `None`    |      `False`       |  \|  |      130.41       |  \|  |      7.73       |
| `onnxruntime` |       `dynamic`       |  `['Add', 'MatMul']`  |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |      102.44       |  \|  |      9.80       |
| `onnxruntime` |       `dynamic`       |  `['Add', 'MatMul']`  |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |      101.57       |  \|  |      9.87       |
| `onnxruntime` |       `dynamic`       |  `['Add', 'MatMul']`  |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |      102.37       |  \|  |      9.80       |
| `onnxruntime` |       `dynamic`       |  `['Add', 'MatMul']`  |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |      102.36       |  \|  |      9.80       |
| `onnxruntime` |       `dynamic`       |       `['Add']`       |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |      130.67       |  \|  |      7.67       |
| `onnxruntime` |       `dynamic`       |       `['Add']`       |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |      131.29       |  \|  |      7.67       |
| `onnxruntime` |       `dynamic`       |       `['Add']`       |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |      132.65       |  \|  |      7.60       |
| `onnxruntime` |       `dynamic`       |       `['Add']`       |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |      131.03       |  \|  |      7.67       |
| `onnxruntime` |       `static`        |  `['Add', 'MatMul']`  |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |      127.99       |  \|  |      7.87       |
| `onnxruntime` |       `static`        |  `['Add', 'MatMul']`  |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |      128.27       |  \|  |      7.80       |
| `onnxruntime` |       `static`        |  `['Add', 'MatMul']`  |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |      131.10       |  \|  |      7.67       |
| `onnxruntime` |       `static`        |  `['Add', 'MatMul']`  |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |      130.29       |  \|  |      7.73       |
| `onnxruntime` |       `static`        |       `['Add']`       |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |      164.55       |  \|  |      6.13       |
| `onnxruntime` |       `static`        |       `['Add']`       |   `False`   | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |      168.61       |  \|  |      5.93       |
| `onnxruntime` |       `static`        |       `['Add']`       |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |   `False`    |       `True`       |  \|  |      164.52       |  \|  |      6.13       |
| `onnxruntime` |       `static`        |       `['Add']`       |   `True`    | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` |    `True`    |       `True`       |  \|  |      165.31       |  \|  |      6.07       |
|   `pytorch`   |        `None`         |        `None`         |   `None`    |                                `{}`                                 |    `None`    |       `None`       |  \|  |      149.23       |  \|  |      6.73       |


{"tags": ["vit"], "datasets": ["beans"], "metrics": ["accuracy", "total_time_in_seconds", "samples_per_second", "latency_in_seconds"], "pipeline_tag": "image-classification"}
tensorboard onnx vit image-classification dataset:beans region:us
TAGS
#tensorboard #onnx #vit #image-classification #dataset-beans #region-us 

task: 'image-classification'  

Backend: 'sagemaker-training'  

Backend args: '{'instance\_type': 'ml.m5.2xlarge', 'supported\_instructions': 'avx512'}'  

Number of evaluation samples: 'All dataset'


Fixed parameters:


* dataset: [{'path': 'beans', 'eval\_split': 'validation', 'data\_keys': {'primary': 'image'}, 'ref\_keys': ['labels'], 'name': None, 'calibration\_split': 'train'}]
* name\_or\_path: 'nateraw/vit-base-beans'
* from\_transformers: 'True'
* node\_exclusion: '[]'
* calibration:
	+ method: 'percentile'
	+ num\_calibration\_samples: '128'
	+ calibration\_histogram\_percentile: '99.999'


Benchmarked parameters:


* framework: 'onnxruntime', 'pytorch'
* quantization\_approach: 'dynamic', 'static'
* operators\_to\_quantize: '['Add', 'MatMul']', '['Add']'
* per\_channel: 'False', 'True'
* framework\_args: '{'opset': 13, 'optimization\_level': 1, 'intra\_op\_num\_threads': 4}', '{}'
* reduce\_range: 'True', 'False'
* apply\_quantization: 'True', 'False'


Evaluation
==========


Non-time metrics
----------------



Time metrics
------------


Time benchmarks were run for 15 seconds per config.


Below, time metrics for batch size = 1, input length = 224.




TAGS
#tensorboard #onnx #vit #image-classification #dataset-beans #region-us 

passage: TAGS
#tensorboard #onnx #vit #image-classification #dataset-beans #region-us 
