class_docstr: Forecaster base model for lstm, seq2seq, tcn and nbeats forecasters.
class_docstr_tok: ["Forecaster","base","model","for","lstm",",","seq2seq",",","tcn","and","nbeats","forecasters","."]
docstr: Evaluate using a trained forecaster with onnxruntime. The method can only be
  used when forecaster is a non-distributed version.
  
  Directly call this method without calling build_onnx is valid and Forecaster will
  automatically build an onnxruntime session with default settings (thread num is 1).
  
  Please note that evaluate result is calculated by scaled y and yhat. If you scaled
  your data (e.g. use .scale() on the TSDataset) please follow the following code
  snap to evaluate your result if you need to evaluate on unscaled data.
  
  >>> from bigdl.chronos.metric.forecast_metrics import Evaluator
  >>> y_hat = forecaster.predict_with_onnx(x)
  >>> y_hat_unscaled = tsdata.unscale_numpy(y_hat) # or other customized unscale methods
  >>> y_unscaled = tsdata.unscale_numpy(y) # or other customized unscale methods
  >>> Evaluator.evaluate(metric=..., y_unscaled, y_hat_unscaled, multioutput=...)
  
  :param data: The data support following formats:
  
         | 1. a numpy ndarray tuple (x, y):
         | x's shape is (num_samples, lookback, feature_dim) where lookback and feature_dim
         | should be the same as past_seq_len and input_feature_num.
         | y's shape is (num_samples, horizon, target_dim), where horizon and target_dim
         | should be the same as future_seq_len and output_feature_num.
         |
         | 2. pytorch dataloader:
         | should be the same as future_seq_len and output_feature_num.
         | the dataloader should return x, y in each iteration with the shape as following:
         | x's shape is (num_samples, lookback, feature_dim) where lookback and feature_dim
         | should be the same as past_seq_len and input_feature_num.
         | y's shape is (num_samples, horizon, target_dim), where horizon and target_dim
         |
         | 3. A bigdl.chronos.data.tsdataset.TSDataset instance:
         | Forecaster will automatically process the TSDataset.
         | By default, TSDataset will be transformed to a pytorch dataloader,
         | which is memory-friendly while a little bit slower.
         | Users may call `roll` on the TSDataset before calling `fit`
         | Then the training speed will be faster but will consume more memory.
  
  :param batch_size: evaluate batch size. The value will not affect evaluate
         result but will affect resources cost(e.g. memory and time).
  :param multioutput: Defines aggregating of multiple output values.
         String in ['raw_values', 'uniform_average']. The value defaults to
         'raw_values'.
  :param quantize: if use the quantized onnx model to evaluate.
  
  :return: A list of evaluation results. Each item represents a metric.
code: def evaluate_with_onnx(
      self,
      data,
      batch_size=32,
      multioutput="raw_values",
      quantize=False,
  ):
      """
      Evaluate using a trained forecaster with onnxruntime. The method can only be
      used when forecaster is a non-distributed version.
  
      Directly call this method without calling build_onnx is valid and Forecaster will
      automatically build an onnxruntime session with default settings (thread num is 1).
  
      Please note that evaluate result is calculated by scaled y and yhat. If you scaled
      your data (e.g. use .scale() on the TSDataset) please follow the following code
      snap to evaluate your result if you need to evaluate on unscaled data.
  
      >>> from bigdl.chronos.metric.forecast_metrics import Evaluator
      >>> y_hat = forecaster.predict_with_onnx(x)
      >>> y_hat_unscaled = tsdata.unscale_numpy(y_hat) # or other customized unscale methods
      >>> y_unscaled = tsdata.unscale_numpy(y) # or other customized unscale methods
      >>> Evaluator.evaluate(metric=..., y_unscaled, y_hat_unscaled, multioutput=...)
  
      :param data: The data support following formats:
  
             | 1. a numpy ndarray tuple (x, y):
             | x's shape is (num_samples, lookback, feature_dim) where lookback and feature_dim
             | should be the same as past_seq_len and input_feature_num.
             | y's shape is (num_samples, horizon, target_dim), where horizon and target_dim
             | should be the same as future_seq_len and output_feature_num.
             |
             | 2. pytorch dataloader:
             | should be the same as future_seq_len and output_feature_num.
             | the dataloader should return x, y in each iteration with the shape as following:
             | x's shape is (num_samples, lookback, feature_dim) where lookback and feature_dim
             | should be the same as past_seq_len and input_feature_num.
             | y's shape is (num_samples, horizon, target_dim), where horizon and target_dim
             |
             | 3. A bigdl.chronos.data.tsdataset.TSDataset instance:
             | Forecaster will automatically process the TSDataset.
             | By default, TSDataset will be transformed to a pytorch dataloader,
             | which is memory-friendly while a little bit slower.
             | Users may call `roll` on the TSDataset before calling `fit`
             | Then the training speed will be faster but will consume more memory.
  
      :param batch_size: evaluate batch size. The value will not affect evaluate
             result but will affect resources cost(e.g. memory and time).
      :param multioutput: Defines aggregating of multiple output values.
             String in ['raw_values', 'uniform_average']. The value defaults to
             'raw_values'.
      :param quantize: if use the quantized onnx model to evaluate.
  
      :return: A list of evaluation results. Each item represents a metric.
      """
      if self.distributed:
          invalidInputError(
              False,
              "ONNX inference has not been supported for distributed "
              "forecaster. You can call .to_local() to transform the "
              "forecaster to a non-distributed version.",
          )
      if not self.fitted:
          invalidInputError(
              False,
              "You must call fit or restore first before calling evaluate!",
          )
      if isinstance(data, TSDataset):
          _rolled = data.numpy_x is None
          data = data.to_torch_data_loader(
              batch_size=batch_size,
              roll=_rolled,
              lookback=self.data_config["past_seq_len"],
              horizon=self.data_config["future_seq_len"],
              feature_col=data.roll_feature,
              target_col=data.roll_target,
              shuffle=False,
          )
      if isinstance(data, DataLoader):
          from torch.utils.data.sampler import RandomSampler
  
          if isinstance(data.sampler, RandomSampler):
              # If dataloader is shuffled, convert input_data to numpy()
              # Avoid to iterate shuffled dataloader two times
              input_x = []
              input_y = []
              for val in data:
                  input_x.append(val[0].numpy())
                  input_y.append(val[1].numpy())
              input_data = np.concatenate(input_x, axis=0)
              target = np.concatenate(input_y, axis=0)
          else:
              input_data = data
              target = np.concatenate(
                  tuple(val[1] for val in data), axis=0
              )
      else:
          input_data, target = data
      if not self.context_enabled:
          self.cxt_manager = ForecasterContextManager(
              self, self.thread_num, True
          )
      else:
          self.cxt_manager = DummyForecasterContextManager()
      with self.cxt_manager:
          if quantize:
              if self.accelerate_method != "onnxruntime_int8":
                  invalidInputError(
                      False,
                      "Can't find the quantized model, "
                      "please call .quantize() method first",
                  )
              yhat = _pytorch_fashion_inference(
                  model=self.accelerated_model,
                  input_data=input_data,
                  batch_size=batch_size,
                  output_tensor=self.optimized_model_output_tensor,
              )
          else:
              if self.accelerate_method != "onnxruntime_fp32":
                  self.build_onnx(self.thread_num)
              yhat = _pytorch_fashion_inference(
                  model=self.accelerated_model,
                  input_data=input_data,
                  batch_size=batch_size,
                  output_tensor=self.optimized_model_output_tensor,
              )
  
      aggregate = "mean" if multioutput == "uniform_average" else None
      return Evaluator.evaluate(
          self.metrics, target, yhat, aggregate=aggregate
      )
  
imports: from bigdl.chronos.forecaster.abstract import Forecaster
  from bigdl.chronos.forecaster.utils import *
  from bigdl.chronos.metric.forecast_metrics import Evaluator
  from typing import Optional
  import numpy
  import warnings
  import os
  from tempfile import TemporaryDirectory
  import torch
  from torch.utils.data import TensorDataset, DataLoader
  from .utils_hpo import GenericLightningModule, _format_metric_str, _config_has_search_space
  from bigdl.nano.utils.common import invalidOperationError, invalidInputError
  from bigdl.chronos.data.tsdataset import TSDataset
  from bigdl.chronos.pytorch.context_manager import DummyForecasterContextManager, ForecasterContextManager
  from bigdl.chronos.pytorch.utils import _pytorch_fashion_inference
label_desc: Use image node_id 13 for calling the BasePytorchForecaster obj's underlying member method code with example usage: obj.evaluate_with_onnx(data, batch_size, multioutput, quantize) and returns: Evaluator