
**task**: `question-answering`

Fixed parameters:
* **model_name_or_path**: `distilbert-base-uncased-distilled-squad`
* **dataset**:
    * **path**: `squad`
    * **name**: `None`
    * **calibration_split**: `train`
    * **eval_split**: `validation`
    * **data_keys**: `{'question': 'question', 'context': 'context'}`
    * **ref_keys**: `['answers']`
    * **max_seq_length**: `128`
* **node_exclusion**: `[]`
* **per_channel**: `False`
* **calibration**:
    * **method**: `minmax`
    * **num_calibration_samples**: `20`
    * **calibration_histogram_percentile**: `None`
    * **calibration_moving_average**: `None`
    * **calibration_moving_average_constant**: `None`
* **framework**: `onnxruntime`
* **framework_args**:
    * **opset**: `15`
    * **optimization_level**: `1`
* **aware_training**: `False`

Benchmarked parameters:
* **quantization_approach**: `dynamic`,  `static`
* **operators_to_quantize**: `['Add', 'MatMul']`,  `['Add']`

## Evaluation
Below, time metrics for
* Batch size: 8
* Input length: 128
| quantization_approach | operators_to_quantize |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |     | exact_match (original) | exact_match (optimized) |     | f1 (original) | f1 (optimized) |
| :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: | :-: | :--------------------: | :---------------------: | :-: | :-----------: | :------------: |
|       `dynamic`       |  `['Add', 'MatMul']`  |  \|  |           247.65            |            75.80             |  \|  |           4.50            |           13.50            |  \|  |         90.000         |         85.000          |  \|  |    90.000     |     86.667     |
|       `dynamic`       |       `['Add']`       |  \|  |           234.61            |            191.78            |  \|  |           4.50            |            5.50            |  \|  |         90.000         |         90.000          |  \|  |    90.000     |     90.000     |
|       `static`        |  `['Add', 'MatMul']`  |  \|  |           238.04            |            131.81            |  \|  |           4.50            |            8.00            |  \|  |         90.000         |         50.000          |  \|  |    90.000     |     59.362     |
|       `static`        |       `['Add']`       |  \|  |           241.84            |            257.85            |  \|  |           4.50            |            4.00            |  \|  |         90.000         |         70.000          |  \|  |    90.000     |     76.333     |

{"tags": ["distilbert"], "datasets": ["squad"], "metrics": ["exact_match", "f1"], "pipeline_tag": "question-answering"}
tensorboard distilbert question-answering dataset:squad region:us
TAGS
#tensorboard #distilbert #question-answering #dataset-squad #region-us 


task: 'question-answering'

Fixed parameters:
* model_name_or_path: 'distilbert-base-uncased-distilled-squad'
* dataset:
    * path: 'squad'
    * name: 'None'
    * calibration_split: 'train'
    * eval_split: 'validation'
    * data_keys: '{'question': 'question', 'context': 'context'}'
    * ref_keys: '['answers']'
    * max_seq_length: '128'
* node_exclusion: '[]'
* per_channel: 'False'
* calibration:
    * method: 'minmax'
    * num_calibration_samples: '20'
    * calibration_histogram_percentile: 'None'
    * calibration_moving_average: 'None'
    * calibration_moving_average_constant: 'None'
* framework: 'onnxruntime'
* framework_args:
    * opset: '15'
    * optimization_level: '1'
* aware_training: 'False'

Benchmarked parameters:
* quantization_approach: 'dynamic',  'static'
* operators_to_quantize: '['Add', 'MatMul']',  '['Add']'

## Evaluation
Below, time metrics for
* Batch size: 8
* Input length: 128
| quantization_approach | operators_to_quantize |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |     | exact_match (original) | exact_match (optimized) |     | f1 (original) | f1 (optimized) |
| :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: | :-: | :--------------------: | :---------------------: | :-: | :-----------: | :------------: |
|       'dynamic'       |  '['Add', 'MatMul']'  |  \|  |           247.65            |            75.80             |  \|  |           4.50            |           13.50            |  \|  |         90.000         |         85.000          |  \|  |    90.000     |     86.667     |
|       'dynamic'       |       '['Add']'       |  \|  |           234.61            |            191.78            |  \|  |           4.50            |            5.50            |  \|  |         90.000         |         90.000          |  \|  |    90.000     |     90.000     |
|       'static'        |  '['Add', 'MatMul']'  |  \|  |           238.04            |            131.81            |  \|  |           4.50            |            8.00            |  \|  |         90.000         |         50.000          |  \|  |    90.000     |     59.362     |
|       'static'        |       '['Add']'       |  \|  |           241.84            |            257.85            |  \|  |           4.50            |            4.00            |  \|  |         90.000         |         70.000          |  \|  |    90.000     |     76.333     |

## Evaluation
Below, time metrics for
* Batch size: 8
* Input length: 128
| quantization_approach | operators_to_quantize |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |     | exact_match (original) | exact_match (optimized) |     | f1 (original) | f1 (optimized) |
| :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: | :-: | :--------------------: | :---------------------: | :-: | :-----------: | :------------: |
|       'dynamic'       |  '['Add', 'MatMul']'  |  \|  |           247.65            |            75.80             |  \|  |           4.50            |           13.50            |  \|  |         90.000         |         85.000          |  \|  |    90.000     |     86.667     |
|       'dynamic'       |       '['Add']'       |  \|  |           234.61            |            191.78            |  \|  |           4.50            |            5.50            |  \|  |         90.000         |         90.000          |  \|  |    90.000     |     90.000     |
|       'static'        |  '['Add', 'MatMul']'  |  \|  |           238.04            |            131.81            |  \|  |           4.50            |            8.00            |  \|  |         90.000         |         50.000          |  \|  |    90.000     |     59.362     |
|       'static'        |       '['Add']'       |  \|  |           241.84            |            257.85            |  \|  |           4.50            |            4.00            |  \|  |         90.000         |         70.000          |  \|  |    90.000     |     76.333     |
TAGS
#tensorboard #distilbert #question-answering #dataset-squad #region-us 
 ## Evaluation
Below, time metrics for
* Batch size: 8
* Input length: 128
| quantization_approach | operators_to_quantize |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |     | exact_match (original) | exact_match (optimized) |     | f1 (original) | f1 (optimized) |
| :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: | :-: | :--------------------: | :---------------------: | :-: | :-----------: | :------------: |
|       'dynamic'       |  '['Add', 'MatMul']'  |  \|  |           247.65            |            75.80             |  \|  |           4.50            |           13.50            |  \|  |         90.000         |         85.000          |  \|  |    90.000     |     86.667     |
|       'dynamic'       |       '['Add']'       |  \|  |           234.61            |            191.78            |  \|  |           4.50            |            5.50            |  \|  |         90.000         |         90.000          |  \|  |    90.000     |     90.000     |
|       'static'        |  '['Add', 'MatMul']'  |  \|  |           238.04            |            131.81            |  \|  |           4.50            |            8.00            |  \|  |         90.000         |         50.000          |  \|  |    90.000     |     59.362     |
|       'static'        |       '['Add']'       |  \|  |           241.84            |            257.85            |  \|  |           4.50            |            4.00            |  \|  |         90.000         |         70.000          |  \|  |    90.000     |     76.333     |
passage: TAGS
#tensorboard #distilbert #question-answering #dataset-squad #region-us 
