class_docstr: The abstract class for Turi Create recommender system models. This class
  defines methods common to all recommender system models, but leaves unique
  model details to separate model classes.
class_docstr_tok: ["The","abstract","class","for","Turi","Create","recommender","system","models",".","This","class","defines","methods","common","to","all","recommender","system","models",",","but","leaves","unique","model","details","to","separate","model","classes","."]
docstr: Evaluate the model's ability to make rating predictions or
  recommendations.
  
  If the model is trained to predict a particular target, the
  default metric used for model comparison is root-mean-squared error
  (RMSE). Suppose :math:`y` and :math:`\widehat{y}` are vectors of length
  :math:`N`, where :math:`y` contains the actual ratings and
  :math:`\widehat{y}` the predicted ratings. Then the RMSE is defined as
  
  .. math::
  
      RMSE = \sqrt{\frac{1}{N} \sum_{i=1}^N (\widehat{y}_i - y_i)^2} .
  
  If the model was not trained on a target column, the default metrics for
  model comparison are precision and recall. Let
  :math:`p_k` be a vector of the :math:`k` highest ranked recommendations
  for a particular user, and let :math:`a` be the set of items for that
  user in the groundtruth `dataset`. The "precision at cutoff k" is
  defined as
  
  .. math:: P(k) = \frac{ | a \cap p_k | }{k}
  
  while "recall at cutoff k" is defined as
  
  .. math:: R(k) = \frac{ | a \cap p_k | }{|a|}
  
  Parameters
  ----------
  dataset : SFrame
      An SFrame that is in the same format as provided for training.
  
  metric : str, {'auto', 'rmse', 'precision_recall'}, optional
      Metric to use for evaluation. The default automatically chooses
      'rmse' for models trained with a `target`, and 'precision_recall'
      otherwise.
  
  exclude_known_for_precision_recall : bool, optional
      A useful option for evaluating precision-recall. Recommender models
      have the option to exclude items seen in the training data from the
      final recommendation list. Set this option to True when evaluating
      on test data, and False when evaluating precision-recall on training
      data.
  
  target : str, optional
      The name of the target column for evaluating rmse. If the model is
      trained with a target column, the default is to using the same
      column. If the model is trained without a target column and `metric`
      is set to 'rmse', this option must provided by user.
  
  verbose : bool, optional
      Enables verbose output. Default is verbose.
  
  **kwargs
      When `metric` is set to 'precision_recall', these parameters
      are passed on to :meth:`evaluate_precision_recall`.
  
  Returns
  -------
  out : SFrame or dict
      Results from the model evaluation procedure. If the model is trained
      on a target (i.e. RMSE is the evaluation criterion), a dictionary
      with three items is returned: items *rmse_by_user* and
      *rmse_by_item* are SFrames with per-user and per-item RMSE, while
      *rmse_overall* is the overall RMSE (a float). If the model is
      trained without a target (i.e. precision and recall are the
      evaluation criteria) an :py:class:`~turicreate.SFrame` is returned
      with both of these metrics for each user at several cutoff values.
  
  Examples
  --------
  >>> import turicreate as tc
  >>> sf = tc.SFrame('https://static.turi.com/datasets/audioscrobbler')
  >>> train, test = tc.recommender.util.random_split_by_user(sf)
  >>> m = tc.recommender.create(train, target='target')
  >>> eval = m.evaluate(test)
  
  See Also
  --------
  evaluate_precision_recall, evaluate_rmse, precision_recall_by_user
docstr_tok: ["Evaluate","the","model","'s","ability","to","make","rating","predictions","or","recommendations",".","If","the","model","is","trained","to","predict","a","particular","target",",","the","default","metric","used","for","model","comparison","is","root-mean-squared","error","(","RMSE",")",".","Suppose",":","math",":","`","y","`","and",":","math",":","`","\\widehat","{","y","}","`","are","vectors","of","length",":","math",":","`","N","`",",","where",":","math",":","`","y","`","contains","the","actual","ratings","and",":","math",":","`","\\widehat","{","y","}","`","the","predicted","ratings",".","Then","the","RMSE","is","defined","as","..","math",":",":","RMSE","=","\\sqrt","{","\\frac","{","1","}","{","N","}","\\sum_","{","i=1","}","^N","(","\\widehat","{","y","}","_i","-","y_i",")","^2","}",".","If","the","model","was","not","trained","on","a","target","column",",","the","default","metrics","for","model","comparison","are","precision","and","recall",".","Let",":","math",":","`","p_k","`","be","a","vector","of","the",":","math",":","`","k","`","highest","ranked","recommendations","for","a","particular","user",",","and","let",":","math",":","`","a","`","be","the","set","of","items","for","that","user","in","the","groundtruth","`","dataset","`",".","The","``","precision","at","cutoff","k","''","is","defined","as","..","math",":",":","P","(","k",")","=","\\frac","{","|","a","\\cap","p_k","|","}","{","k","}","while","``","recall","at","cutoff","k","''","is","defined","as","..","math",":",":","R","(","k",")","=","\\frac","{","|","a","\\cap","p_k","|","}","{","|a|","}","Parameters","--","--","--","--","--","dataset",":","SFrame","An","SFrame","that","is","in","the","same","format","as","provided","for","training",".","metric",":","str",",","{","'auto","'",",","'rmse","'",",","'precision_recall","'","}",",","optional","Metric","to","use","for","evaluation",".","The","default","automatically","chooses","'rmse","'","for","models","trained","with","a","`","target","`",",","and","'precision_recall'","otherwise",".","exclude_known_for_precision_recall",":","bool",",","optional","A","useful","option","for","evaluating","precision-recall",".","Recommender","models","have","the","option","to","exclude","items","seen","in","the","training","data","from","the","final","recommendation","list",".","Set","this","option","to","True","when","evaluating","on","test","data",",","and","False","when","evaluating","precision-recall","on","training","data",".","target",":","str",",","optional","The","name","of","the","target","column","for","evaluating","rmse",".","If","the","model","is","trained","with","a","target","column",",","the","default","is","to","using","the","same","column",".","If","the","model","is","trained","without","a","target","column","and","`","metric","`","is","set","to","'rmse","'",",","this","option","must","provided","by","user",".","verbose",":","bool",",","optional","Enables","verbose","output",".","Default","is","verbose",".","*","*","kwargs","When","`","metric","`","is","set","to","'precision_recall","'",",","these","parameters","are","passed","on","to",":","meth",":","`","evaluate_precision_recall","`",".","Returns","--","--","--","-","out",":","SFrame","or","dict","Results","from","the","model","evaluation","procedure",".","If","the","model","is","trained","on","a","target","(","i.e",".","RMSE","is","the","evaluation","criterion",")",",","a","dictionary","with","three","items","is","returned",":","items","*","rmse_by_user","*","and","*","rmse_by_item","*","are","SFrames","with","per-user","and","per-item","RMSE",",","while","*","rmse_overall","*","is","the","overall","RMSE","(","a","float",")",".","If","the","model","is","trained","without","a","target","(","i.e",".","precision","and","recall","are","the","evaluation","criteria",")","an",":","py",":","class",":","`","~turicreate.SFrame","`","is","returned","with","both","of","these","metrics","for","each","user","at","several","cutoff","values",".","Examples","--","--","--","--",">",">",">","import","turicreate","as","tc",">",">",">","sf","=","tc.SFrame","(","'https",":","\/\/static.turi.com\/datasets\/audioscrobbler","'",")",">",">",">","train",",","test","=","tc.recommender.util.random_split_by_user","(","sf",")",">",">",">","m","=","tc.recommender.create","(","train",",","target='target","'",")",">",">",">","eval","=","m.evaluate","(","test",")","See","Also","--","--","--","--","evaluate_precision_recall",",","evaluate_rmse",",","precision_recall_by_user"]
code: def evaluate(
      self,
      dataset,
      metric="auto",
      exclude_known_for_precision_recall=True,
      target=None,
      verbose=True,
      **kwargs,
  ):
      r"""
      Evaluate the model's ability to make rating predictions or
      recommendations.
  
      If the model is trained to predict a particular target, the
      default metric used for model comparison is root-mean-squared error
      (RMSE). Suppose :math:`y` and :math:`\widehat{y}` are vectors of length
      :math:`N`, where :math:`y` contains the actual ratings and
      :math:`\widehat{y}` the predicted ratings. Then the RMSE is defined as
  
      .. math::
  
          RMSE = \sqrt{\frac{1}{N} \sum_{i=1}^N (\widehat{y}_i - y_i)^2} .
  
      If the model was not trained on a target column, the default metrics for
      model comparison are precision and recall. Let
      :math:`p_k` be a vector of the :math:`k` highest ranked recommendations
      for a particular user, and let :math:`a` be the set of items for that
      user in the groundtruth `dataset`. The "precision at cutoff k" is
      defined as
  
      .. math:: P(k) = \frac{ | a \cap p_k | }{k}
  
      while "recall at cutoff k" is defined as
  
      .. math:: R(k) = \frac{ | a \cap p_k | }{|a|}
  
      Parameters
      ----------
      dataset : SFrame
          An SFrame that is in the same format as provided for training.
  
      metric : str, {'auto', 'rmse', 'precision_recall'}, optional
          Metric to use for evaluation. The default automatically chooses
          'rmse' for models trained with a `target`, and 'precision_recall'
          otherwise.
  
      exclude_known_for_precision_recall : bool, optional
          A useful option for evaluating precision-recall. Recommender models
          have the option to exclude items seen in the training data from the
          final recommendation list. Set this option to True when evaluating
          on test data, and False when evaluating precision-recall on training
          data.
  
      target : str, optional
          The name of the target column for evaluating rmse. If the model is
          trained with a target column, the default is to using the same
          column. If the model is trained without a target column and `metric`
          is set to 'rmse', this option must provided by user.
  
      verbose : bool, optional
          Enables verbose output. Default is verbose.
  
      **kwargs
          When `metric` is set to 'precision_recall', these parameters
          are passed on to :meth:`evaluate_precision_recall`.
  
      Returns
      -------
      out : SFrame or dict
          Results from the model evaluation procedure. If the model is trained
          on a target (i.e. RMSE is the evaluation criterion), a dictionary
          with three items is returned: items *rmse_by_user* and
          *rmse_by_item* are SFrames with per-user and per-item RMSE, while
          *rmse_overall* is the overall RMSE (a float). If the model is
          trained without a target (i.e. precision and recall are the
          evaluation criteria) an :py:class:`~turicreate.SFrame` is returned
          with both of these metrics for each user at several cutoff values.
  
      Examples
      --------
      >>> import turicreate as tc
      >>> sf = tc.SFrame('https://static.turi.com/datasets/audioscrobbler')
      >>> train, test = tc.recommender.util.random_split_by_user(sf)
      >>> m = tc.recommender.create(train, target='target')
      >>> eval = m.evaluate(test)
  
      See Also
      --------
      evaluate_precision_recall, evaluate_rmse, precision_recall_by_user
      """
  
      ret = {}
  
      dataset = self.__prepare_dataset_parameter(dataset)
  
      # If the model does not have a target column, compute prec-recall.
      if metric in ["precision_recall", "auto"]:
          results = self.evaluate_precision_recall(
              dataset,
              exclude_known=exclude_known_for_precision_recall,
              verbose=verbose,
              **kwargs,
          )
          ret.update(results)
          if verbose:
              print(
                  "\nPrecision and recall summary statistics by cutoff"
              )
              print(
                  results["precision_recall_by_user"]
                  .groupby(
                      "cutoff",
                      {
                          "mean_precision": _turicreate.aggregate.AVG(
                              "precision"
                          ),
                          "mean_recall": _turicreate.aggregate.AVG(
                              "recall"
                          ),
                      },
                  )
                  .topk("cutoff", reverse=True)
              )
      if metric in ["rmse", "auto"]:
          if target is None:
              target = self.target
          if target is None or target == "":
              _logging.warning(
                  "Model trained without a target. Skipping RMSE computation."
              )
          else:
              results = self.evaluate_rmse(dataset, target)
              ret.update(results)
  
              if verbose:
                  print("\nOverall RMSE:", results["rmse_overall"])
                  print("\nPer User RMSE (best)")
                  print(
                      results["rmse_by_user"].topk(
                          "rmse", 1, reverse=True
                      )
                  )
                  print("\nPer User RMSE (worst)")
                  print(results["rmse_by_user"].topk("rmse", 1))
                  print("\nPer Item RMSE (best)")
                  print(
                      results["rmse_by_item"].topk(
                          "rmse", 1, reverse=True
                      )
                  )
                  print("\nPer Item RMSE (worst)")
                  print(results["rmse_by_item"].topk("rmse", 1))
      if metric not in ["rmse", "precision_recall", "auto"]:
          raise ValueError(
              'Unknown evaluation metric %s, supported metrics are ["rmse", "precision_recall"]'
              % metric
          )
  
      return ret
  
imports: from __future__ import print_function
  from __future__ import division
  from __future__ import absolute_import
  import turicreate
  import random
  import array
  import logging
  import time
  import six
  from turicreate.toolkits import _coreml_utils
  from turicreate.toolkits._main import ToolkitError
  from turicreate.toolkits._model import Model
  from turicreate.toolkits._internal_utils import _toolkit_repr_print, _precomputed_field
  import turicreate.aggregate
  from turicreate.data_structures.sarray import SArray
  from turicreate.data_structures.sframe import SFrame
  from turicreate._deps import numpy, HAS_NUMPY
label_desc: Use image node_id 20 for calling the _Recommender obj's underlying member method code with example usage: obj.evaluate(dataset, metric, exclude_known_for_precision_recall, target, verbose) and returns: ret