**Abstract**

# Open Access publishing: journals vs. repositories

The objective of Open Access (OA) in scientific publishing is to make the results of research freely available to all readers, thereby increasing both accessibility and dissemination. However, from a publishers' viewpoint, making the contents of a journal available to all free of charge, even if only online, implies loss of subscription revenue, requiring that income be generated by other means, such as fees charged to authors. For some non-profit publishers alternative sources may be available, such as professional association dues, or direct subsidies from host institutions, benevolent foundations or other funding organizations. But for commercial publishers, besides advertising revenue, the only real alternative to subscription fees is transferring the cost burden from journal subscribers to authors, leading to an "Author pays" approach to OA.

In most domains of physics, however, and increasingly in mathematics and computer science, OA is already effectively implemented, at no cost to authors, through use of preprint repositories such as arXiv. It seems difficult to argue the need for OA journals financed by publication charges to authors, just to assure online availability of papers in a version formatted by the journal, when they are already freely available to all, at least in preprint form, via such repositories.

There remain good reasons, of course, for publishing in refereed journals; in particular, the "value-added" of quality control assured by the peer-review process. However, the cost of organizing this service, the core of which is currently provided by the scientific community at no charge to the publisher, is far too small to justify the rates that some journals currently charge as subscription or publication fees. Moreover, in the "Author pays" approach to OA, rejection of submitted papers diminishes the number of articles published, and hence decreases revenue to the publisher. Commercial logic tends therefore to militate against the application of overly high acceptance standards in this model, since it may be an impediment to publishing the quantity of articles needed to provide revenue adequate to sustaining the journal.

Other factors, such as the journal's reputation, enter into consideration as well; authors prefer to publish in journals known for their high standards, and this provides a good motive for maintaining quality. But it is not clear, especially for a commercially based publisher, how best to balance these factors in order to ensure viability. A possible implication may be that authors whose papers do not meet the peer review standards of the better subscription based journals, but are able and willing to pay publication fees, could expect higher chances of acceptance in commercially run "Author pays" OA journals. On the other hand, authors without the necessary funds, or unwilling to use scarce research funding for the purpose of paying publication fees, might prefer to publish in journals whose revenue does not depend on viewing authors as paying clients.

# Open access journals, old and new

The New Journal of Physics (NJP), published by the Deutsche Physikalische Gesellschaft (DPG) and the Institute of Physics (IOP), was a pioneer online OA journal that succeeded, since 1998, in publishing a steady volume of articles that are widely read and cited. In fact, of the fifty-one journals listed under "physics" in Lund University's Directory of Open Access Journals (DOAJ) (only thirteen of which merit mention in the Scientific Citation Index), NJP is the only one that is currently self-supporting without the need for institutional or government subsidies. But only recently has it been able to generate enough revenue from publication fees to cover its production costs. Sustaining a journal that operates at a loss for many years is a choice that can be made by publishers answerable to non-profit professional associations like DPG and IOP, but not to the Board of Directors of a commercially motivated firm.

Nevertheless, with the appeal of low production cost, rapid turnover online publishing, and with rising resistance within the user community to inflated subscription rates, several new online journals have been created by commercial publishers in recent years using the "Author pays" model of OA. Biomed Central (BMC), for example, a UK-based commercial publisher, founded in 2000 and acquired in 2008 by Springer-Verlag, has succeeded in running a large number of online OA journals in the biomedical field with revenues mainly generated by publication fees. Two new online physics journals were launched by BMC in 2007 and 2008: PhysMath Central (PMC) Physics A (high energy physics) and PMC Physics B (condensed matter and atomic physics), with revenue generated by levying "processing charges" of €1100 per article. The number of papers appearing so far, however, suggests little enthusiasm on the part of authors from the physics community for paying such rates; in all, only four research articles have appeared in Physics A in 2008 and thirteen in Physics B.

The announced policies of Biomed Central also provide some reason for concern about the implementation of peer review standards. In their "instructions to editors", the editorial guidelines of several BMC journals have until recently stated: "In the absence of compelling reasons to reject, the Journal ... recommends acceptance, as ultimately the quality of an article will be judged by the scientific community after its publication." The editor of PMC Physics A has affirmed, however, that no such editorial guidelines are used by them and that high peer review standards are the norm. It is nevertheless difficult to see what would persuade researchers to pay a fee of €1100 to have a refereed version of their paper made available for free online by the publisher, when the same work is already available to all via the arXiv, and can be published at no charge to the authors in a conventional subscription based journal.

Another publisher, Cairo-based Hindawi Publishing Corporation, founded in 1997, has a respectable track record in subscription-based scientific publishing. In 2007, it decided to sell four of its standard subscription journals and convert another thirty to OA, launching a new range of about 150 OA journals in engineering, medicine, physics, mathematics, biology and some other fields. A majority of these are still at an early start-up stage, posting as yet no more than a handful of articles. Only about half these journals have an editor-in-chief; the others are declared to be "Community Journals", and have large editorial boards, but no scientific editor-in-chief to coordinate their operation. The combined number of editorial board members, garnered mainly through mass e-mail solicitations, is over five thousand. Although they are apparently trying to maintain adequate refereeing standards and procedures, the absence of scientifically qualified coordinating editors in charge of their "Community Journals" suggests a cutting of corners. They all rely strongly on automated methods of operation and communication, presumably with a view to minimizing production costs and keeping publication charges relatively low (between €200 and €400 per article for most of the new journals, but higher rates of up to €1000 for the others). Selection of referees and implementation of their recommendations is up to the board member chosen, but correspondence with referees is largely handled through an automated process of e-mail messages. These appear to be written, signed and sent by the board member, but the latter may, in fact, have never seen or approved the text. It seems unlikely that many editorial board members will be happy to continue providing their services if such procedures are left unchanged, and suitably qualified editors-in-chief are not appointed for these journals,

# Community initiatives

Another significant movement is meanwhile appearing, one that undoubtedly provides cause for large commercial publishers to consider hedging their bets by acquiring low production cost, online journals based on the "Author pays" model. This is the formation of large common interest groups within the scientific community, such as the SCOAP${}^3$ consortium in high energy physics, spearheaded by big laboratories like the CERN particle-physics laboratory, and now including institutional membership from eighteen countries. SCOAP${}^3$ has the objective of encouraging a transition from subscription based journals to lower cost OA publishing through concerted action. With the addition of a sufficient number of further institutional partners, this consumer alliance would represent a dominant share of research authors and journal readership in this field that could have enormous clout.

The size and strategy of the consortium has been evolving with time. After some rethinking of earlier approaches, such as transferring portions of their library budgets to subsidization of individual journal publication charges for authors, they appear to be converging on the more effective strategy of offering the published work of their members on a competitive basis, as a package deal, and replacing the purchase of subscriptions or payment of individual publication fees by a global fee adequate to covering all costs for their members. However, according to their own calculations, based on accepting publishers' quoted estimates of between €1000 and €2000 per article as the appropriate average cost, this would not decrease the overall fees to participating members, but rather double them from an estimated current total of €5m per year to €10m, assuming a base of 500 participating institutions,

Within an increasingly efficient, web-based, low cost publishing environment, this approach could nevertheless help bring about a more realistic correspondence between subscription and publication charges and actual production costs. In this setting smaller, low production cost OA publishers could possibly pose a competitive threat to publishing giants like Elsevier or Springer if the latter are unable to adapt their pricing strategies. But even if SCOAP${}^3$ succeeds in providing an effective alternative that helps reverse trends towards excessive journal charges within the domain of particle physics, it is not clear whether this could extend to other areas. There are few other domains where common interests and shared facilities are so coherently defined, or where the number of journals in which most published research appears is so small. Other alternatives and models for scientific publishing must therefore also be considered, and efforts made to bring market forces to bear in the interests of the community.

# Conclusions

Amongst current models, the shortcomings of the "Author pays" approach to OA scientific publishing are clear: it further taxes scarce research funds by transferring the cost burden to authors, excludes those who are unwilling or unable to pay such charges, and places the implementation of peer review standards in a competitive relation with the financial viability of the journal. The research community provides not only the published material, but also the refereeing services that, with distribution, form the main "value-added" that journals offer. It is therefore up to its members to make the choices, and exert the necessary pressure to assure that they are beneficiaries of transformations in scientific publishing that are occurring as a result of evolution in technology and consumer reaction against inflated subscription prices.

Meanwhile, at least in most branches of physics, mathematics, computer science and some other domains, the benefits of Open Access will continue to be adequately provided by widely used repositories such as arXiv. Other fields seeking to develop effective vehicles for Open Access could do well to first consider such an approach, where preprint/postprint repositories have long been seen to coexist constructively with refereed journals, providing in a complementary fashion for rapid dissemination, universal access, assurance of quality standards and long-term preservation of the results of scientific research.


abstract:
  .
  We introduce YOLO9000, a state-of-the-art, real-time object detection system that can detect over 9000 object categories. First we propose various improvements to the YOLO detection method, both novel and drawn from prior work. The improved model, YOLOv2, is state-of-the-art on standard detection tasks like Pascal VOC and COCO. Using a novel, multi-scale training method the same YOLOv2 model can run at varying sizes, offering an easy tradeoff between speed and accuracy. At 67 FPS, YOLOv2 gets 76.8 mAP on VOC 2007. At 40 FPS, YOLOv2 gets 78.6 mAP, outperforming state-of-the-art methods like Faster R-CNN with ResNet and SSD while still running significantly faster. Finally we propose a method to jointly train on object detection and classification. Using this method we train YOLO9000 simultaneously on the COCO detection dataset and the ImageNet classification dataset. Our joint training allows YOLO9000 to predict detections for object classes that don't have labelled detection data. We validate our approach on the ImageNet detection task. YOLO9000 gets 19.7 mAP on the ImageNet detection validation set despite only having detection data for 44 of the 200 classes. On the 156 classes not in COCO, YOLO9000 gets 16.0 mAP. But YOLO can detect more than just 200 classes; it predicts detections for more than 9000 different object categories. And it still runs in real-time.
author: Joseph Redmon$^{* \dag}$, Ali Farhadi$^{* \dag}$
University of Washington$^*$, Allen Institute for AI$^\dag$
<http://pjreddie.com/yolo9000/>
bibliography: yolo.bib
title: YOLO9000:
  Better, Faster, Stronger

# Introduction

General purpose object detection should be fast, accurate, and able to recognize a wide variety of objects. Since the introduction of neural networks, detection frameworks have become increasingly fast and accurate. However, most detection methods are still constrained to a small set of objects.

Current object detection datasets are limited compared to datasets for other tasks like classification and tagging. The most common detection datasets contain thousands to hundreds of thousands of images with dozens to hundreds of tags . Classification datasets have millions of images with tens or hundreds of thousands of categories .

We would like detection to scale to level of object classification. However, labelling images for detection is far more expensive than labelling for classification or tagging (tags are often user-supplied for free). Thus we are unlikely to see detection datasets on the same scale as classification datasets in the near future.

We propose a new method to harness the large amount of classification data we already have and use it to expand the scope of current detection systems. Our method uses a hierarchical view of object classification that allows us to combine distinct datasets together.

We also propose a joint training algorithm that allows us to train object detectors on both detection and classification data. Our method leverages labeled detection images to learn to precisely localize objects while it uses classification images to increase its vocabulary and robustness.

Using this method we train YOLO9000, a real-time object detector that can detect over 9000 different object categories. First we improve upon the base YOLO detection system to produce YOLOv2, a state-of-the-art, real-time detector. Then we use our dataset combination method and joint training algorithm to train a model on more than 9000 classes from ImageNet as well as detection data from COCO.

All of our code and pre-trained models are available online at <http://pjreddie.com/yolo9000/>.

# Better

YOLO suffers from a variety of shortcomings relative to state-of-the-art detection systems. Error analysis of YOLO compared to Fast R-CNN shows that YOLO makes a significant number of localization errors. Furthermore, YOLO has relatively low recall compared to region proposal-based methods. Thus we focus mainly on improving recall and localization while maintaining classification accuracy.

Computer vision generally trends towards larger, deeper networks . Better performance often hinges on training larger networks or ensembling multiple models together. However, with YOLOv2 we want a more accurate detector that is still fast. Instead of scaling up our network, we simplify the network and then make the representation easier to learn. We pool a variety of ideas from past work with our own novel concepts to improve YOLO's performance. A summary of results can be found in Table .

**Batch Normalization.** Batch normalization leads to significant improvements in convergence while eliminating the need for other forms of regularization . By adding batch normalization on all of the convolutional layers in YOLO we get more than 2% improvement in mAP. Batch normalization also helps regularize the model. With batch normalization we can remove dropout from the model without overfitting.

**High Resolution Classifier.** All state-of-the-art detection methods use classifier pre-trained on ImageNet . Starting with AlexNet most classifiers operate on input images smaller than $256 \times 256$ . The original YOLO trains the classifier network at $224 \times 224$ and increases the resolution to $448 \time 448$ for detection. This means the network has to simultaneously switch to learning object detection and adjust to the new input resolution.

For YOLOv2 we first fine tune the classification network at the full $448 \times 448$ resolution for 10 epochs on ImageNet. This gives the network time to adjust its filters to work better on higher resolution input. We then fine tune the resulting network on detection. This high resolution classification network gives us an increase of almost 4% mAP.

**Convolutional With Anchor Boxes.** YOLO predicts the coordinates of bounding boxes directly using fully connected layers on top of the convolutional feature extractor. Instead of predicting coordinates directly Faster R-CNN predicts bounding boxes using hand-picked priors . Using only convolutional layers the region proposal network (RPN) in Faster R-CNN predicts offsets and confidences for anchor boxes. Since the prediction layer is convolutional, the RPN predicts these offsets at every location in a feature map. Predicting offsets instead of coordinates simplifies the problem and makes it easier for the network to learn.

We remove the fully connected layers from YOLO and use anchor boxes to predict bounding boxes. First we eliminate one pooling layer to make the output of the network's convolutional layers higher resolution. We also shrink the network to operate on $416 \time 416$ input images instead of $448 \times 448$. We do this because we want an odd number of locations in our feature map so there is a single center cell. Objects, especially large objects, tend to occupy the center of the image so it's good to have a single location right at the center to predict these objects instead of four locations that are all nearby. YOLO's convolutional layers downsample the image by a factor of 32 so by using an input image of $416 \time 416$ we get an output feature map of $13 \times 13$.

When we move to anchor boxes we also decouple the class prediction mechanism from the spatial location and instead predict class and objectness for every anchor box. Following YOLO, the objectness prediction still predicts the IOU of the ground truth and the proposed box and the class predictions predict the conditional probability of that class given that there is an object.

Using anchor boxes we get a small decrease in accuracy. YOLO only predicts 98 boxes per image but with anchor boxes our model predicts more than a thousand. Without anchor boxes our intermediate model gets $69.5$ mAP with a recall of $81\%$. With anchor boxes our model gets $69.2$ mAP with a recall of $88\%$. Even though the mAP decreases, the increase in recall means that our model has more room to improve.

**Dimension Clusters.** We encounter two issues with anchor boxes when using them with YOLO. The first is that the box dimensions are hand picked. The network can learn to adjust the boxes appropriately but if we pick better priors for the network to start with we can make it easier for the network to learn to predict good detections.

Instead of choosing priors by hand, we run k-means clustering on the training set bounding boxes to automatically find good priors. If we use standard k-means with Euclidean distance larger boxes generate more error than smaller boxes. However, what we really want are priors that lead to good IOU scores, which is independent of the size of the box. Thus for our distance metric we use:

$$d(\text{box}, \text{centroid}) = 1 - \text{IOU}(\text{box}, \text{centroid})$$

We run k-means for various values of $k$ and plot the average IOU with closest centroid, see Figure . We choose $k=5$ as a good tradeoff between model complexity and high recall. The cluster centroids are significantly different than hand-picked anchor boxes. There are fewer short, wide boxes and more tall, thin boxes.

We compare the average IOU to closest prior of our clustering strategy and the hand-picked anchor boxes in Table . At only 5 priors the centroids perform similarly to 9 anchor boxes with an average IOU of 61.0 compared to 60.9. If we use 9 centroids we see a much higher average IOU. This indicates that using k-means to generate our bounding box starts the model off with a better representation and makes the task easier to learn.

| Box Generation | $\#$ | Avg IOU |
|:---------------|:----:|:-------:|
| Cluster SSE    |  5   |  58.7   |
| Cluster IOU    |  5   |  61.0   |
| Anchor Boxes   |  9   |  60.9   |
| Cluster IOU    |  9   |  67.2   |

**Average IOU of boxes to closest priors on VOC 2007.** The average IOU of objects on VOC 2007 to their closest, unmodified prior using different generation methods. Clustering gives much better results than using hand-picked priors.

**Direct location prediction.** When using anchor boxes with YOLO we encounter a second issue: model instability, especially during early iterations. Most of the instability comes from predicting the $(x,y)$ locations for the box. In region proposal networks the network predicts values $t_x$ and $t_y$ and the $(x,y)$ center coordinates are calculated as:

$$\begin{aligned}
x &= (t_x * w_a) - x_a \\
y &= (t_y * h_a) - y_a
\end{aligned}$$

For example, a prediction of $t_x = 1$ would shift the box to the right by the width of the anchor box, a prediction of $t_x = -1$ would shift it to the left by the same amount.

This formulation is unconstrained so any anchor box can end up at any point in the image, regardless of what location predicted the box. With random initialization the model takes a long time to stabilize to predicting sensible offsets.

Instead of predicting offsets we follow the approach of YOLO and predict location coordinates relative to the location of the grid cell. This bounds the ground truth to fall between $0$ and $1$. We use a logistic activation to constrain the network's predictions to fall in this range.

The network predicts 5 bounding boxes at each cell in the output feature map. The network predicts 5 coordinates for each bounding box, $t_x$, $t_y$, $t_w$, $t_h$, and $t_o$. If the cell is offset from the top left corner of the image by $(c_x, c_y)$ and the bounding box prior has width and height $p_w$, $p_h$, then the predictions correspond to:

$$\begin{aligned}
b_x &= \sigma(t_x) + c_x \\
b_y &= \sigma(t_y)  + c_y\\
b_w &= p_w e^{t_w}\\
b_h &= p_h e^{t_h}\\
Pr(\text{object}) * IOU(b, \text{object}) &= \sigma(t_o)\\
\end{aligned}$$

Since we constrain the location prediction the parametrization is easier to learn, making the network more stable. Using dimension clusters along with directly predicting the bounding box center location improves YOLO by almost 5% over the version with anchor boxes.

**Fine-Grained Features.**This modified YOLO predicts detections on a $13 \times 13$ feature map. While this is sufficient for large objects, it may benefit from finer grained features for localizing smaller objects. Faster R-CNN and SSD both run their proposal networks at various feature maps in the network to get a range of resolutions. We take a different approach, simply adding a passthrough layer that brings features from an earlier layer at $26 \times 26$ resolution.

The passthrough layer concatenates the higher resolution features with the low resolution features by stacking adjacent features into different channels instead of spatial locations, similar to the identity mappings in ResNet. This turns the $26 \times 26 \times 512$ feature map into a $13 \times 13 \times 2048$ feature map, which can be concatenated with the original features. Our detector runs on top of this expanded feature map so that it has access to fine grained features. This gives a modest 1% performance increase.

**Multi-Scale Training.** The original YOLO uses an input resolution of $448 \times 448$. With the addition of anchor boxes we changed the resolution to $416 \times 416$. However, since our model only uses convolutional and pooling layers it can be resized on the fly. We want YOLOv2 to be robust to running on images of different sizes so we train this into the model.

Instead of fixing the input image size we change the network every few iterations. Every 10 batches our network randomly chooses a new image dimension size. Since our model downsamples by a factor of 32, we pull from the following multiples of 32: $\{320, 352, ... ,608\}$. Thus the smallest option is $320 \times 320$ and the largest is $608 \times 608$. We resize the network to that dimension and continue training.

This regime forces the network to learn to predict well across a variety of input dimensions. This means the same network can predict detections at different resolutions. The network runs faster at smaller sizes so YOLOv2 offers an easy tradeoff between speed and accuracy.

At low resolutions YOLOv2 operates as a cheap, fairly accurate detector. At $288 \times 288$ it runs at more than 90 FPS with mAP almost as good as Fast R-CNN. This makes it ideal for smaller GPUs, high framerate video, or multiple video streams.

At high resolution YOLOv2 is a state-of-the-art detector with 78.6 mAP on VOC 2007 while still operating above real-time speeds. See Table for a comparison of YOLOv2 with other frameworks on VOC 2007. Figure

**Further Experiments.** We train YOLOv2 for detection on VOC 2012. Table shows the comparative performance of YOLOv2 versus other state-of-the-art detection systems. YOLOv2 achieves 73.4 mAP while running far faster than competing methods. We also train on COCO and compare to other methods in Table . On the VOC metric (IOU = .5) YOLOv2 gets 44.0 mAP, comparable to SSD and Faster R-CNN.

```latex
\begin{table*}
    \centering
    \setlength{\tabcolsep}{4pt}
    \begin{tabular}{r|c|ccccccc|c}
        & YOLO & & & & & & & & YOLOv2 \\
        \hline
        batch norm? & & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark\\
        hi-res classifier? & & & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark\\
        convolutional? & &  &  & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark\\
        anchor boxes?  & &  &  & \checkmark & \checkmark &  &  & & \\
        new network?   & &  &  &  & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark\\
        dimension priors?    & &  &  &  &  & \checkmark & \checkmark &  \checkmark &  \checkmark\\
        location prediction?    & &  &  &  &  & \checkmark & \checkmark &  \checkmark &  \checkmark\\

        passthrough?   & &  &  &  &  &  & \checkmark & \checkmark & \checkmark\\
        multi-scale?   & &  &  &  &  &  &  & \checkmark & \checkmark\\
        hi-res detector? & &  &  &  &  &  &  & & \checkmark\\
        \hline
        VOC2007 mAP & 63.4 & 65.8 & 69.5 & 69.2 & 69.6 & 74.4 & 75.4 & 76.8& \textbf{78.6} \\
    \end{tabular}
    \caption{\textbf{The path from YOLO to YOLOv2.} Most of the listed design decisions lead to significant increases in mAP. Two exceptions are switching to a fully convolutional network with anchor boxes and using the new network. Switching to the anchor box style approach increased recall without changing mAP while using the new network cut computation by 33\%.}
    \label{analysis}
\end{table*}
```

| Detection Frameworks    |     Train |      mAP | FPS |
|:------------------------|----------:|---------:|----:|
| Fast R-CNN              | 2007+2012 |     70.0 | 0.5 |
| Faster R-CNN VGG-16     | 2007+2012 |     73.2 |   7 |
| Faster R-CNN ResNet     | 2007+2012 |     76.4 |   5 |
| YOLO                    | 2007+2012 |     63.4 |  45 |
| SSD300                  | 2007+2012 |     74.3 |  46 |
| SSD500                  | 2007+2012 |     76.8 |  19 |
| YOLOv2 $288 \times 288$ | 2007+2012 |     69.0 |  91 |
| YOLOv2 $352 \times 352$ | 2007+2012 |     73.7 |  81 |
| YOLOv2 $416 \times 416$ | 2007+2012 |     76.8 |  67 |
| YOLOv2 $480 \times 480$ | 2007+2012 |     77.8 |  59 |
| YOLOv2 $544 \times 544$ | 2007+2012 | **78.6** |  40 |

**Detection frameworks on Pascal VOC 2007.** YOLOv2 is faster and more accurate than prior detection methods. It can also run at different resolutions for an easy tradeoff between speed and accuracy. Each YOLOv2 entry is actually the same trained model with the same weights, just evaluated at a different size. All timing information is on a Geforce GTX Titan X (original, not Pascal model).

```latex
\begin{table*}[ht]\scriptsize
    \centering
    \setlength{\tabcolsep}{1.45pt}
    \begin{tabular}{l|c|c|cccccccccccccccccccc}
        Method & data &  mAP &  aero &  bike &  bird &  boat &  bottle &  bus &  car &  cat &  chair &  cow &  table &  dog &  horse &  mbike &  person &  plant &  sheep &  sofa &  train &  tv \\
        \hline
        Fast R-CNN \cite{fastrcnn} & 07++12 & 68.4 & 82.3 & 78.4 & 70.8 & 52.3 & 38.7 & 77.8 & 71.6 & 89.3 & 44.2 & 73.0 & 55.0 & 87.5 & 80.5 & 80.8 & 72.0 & 35.1 & 68.3 & 65.7 & 80.4 & 64.2\\
        Faster R-CNN \cite{ren2015faster} & 07++12 & 70.4 & 84.9 & 79.8 & 74.3 & 53.9 & 49.8 & 77.5 & 75.9 & 88.5 & 45.6 & 77.1 & 55.3 & 86.9 & 81.7 & 80.9 & 79.6 & 40.1 & 72.6 & 60.9 & 81.2 & 61.5\\
        YOLO \cite{yolo} & 07++12 & 57.9 & 77.0 & 67.2 & 57.7 & 38.3 & 22.7 & 68.3 & 55.9 & 81.4 & 36.2 & 60.8 & 48.5 & 77.2 & 72.3 & 71.3 & 63.5 & 28.9 & 52.2 & 54.8 & 73.9 & 50.8\\
        SSD300 \cite{ssd} & 07++12 & 72.4 & 85.6 & 80.1 & 70.5 & 57.6 & 46.2 & 79.4 & 76.1 & 89.2 & 53.0 & 77.0 & 60.8 & 87.0 & 83.1 & 82.3 & 79.4 & 45.9 & 75.9 & 69.5 & 81.9 & 67.5\\
        SSD512 \cite{ssd} & 07++12 & 74.9 & 87.4 & 82.3 & 75.8 & 59.0 & 52.6 & 81.7 & 81.5 & 90.0 & 55.4 & 79.0 & 59.8 & 88.4 & 84.3 & 84.7 & 83.3 & 50.2 & 78.0 & 66.3 & 86.3 & 72.0\\
        %SSD512 \cite{ssd} & COCO+07++12 & 80.0& 90.7& 86.8& 80.5& 67.8& 60.8 &86.3 &85.5& 93.5& 63.2& 85.7& 64.4& 90.9& 89.0& 88.9& 86.8& 57.2 &85.1& 72.8& 88.4& 75.9\\
        ResNet \cite{resnet} & 07++12 & 73.8 & 86.5 &81.6 &77.2 &58.0 &51.0 &78.6 &76.6 &93.2 &48.6 &80.4 &59.0 &92.1 &85.3 &84.8 &80.7 &48.1 &77.3 &66.5 &84.7 &65.6\\
        %ResNet \cite{resnet} & COCO+07++12 & 83.8& 92.1 &88.4& 84.8& 75.9& 71.4& 86.3& 87.8& 94.2& 66.8& 89.4& 69.2& 93.9& 91.9& 90.9& 89.6& 67.9& 88.2& 76.8& 90.3& 80.0\\
        \hline
        YOLOv2 $544$& 07++12 & 73.4 & 86.3 & 82.0 & 74.8 & 59.2 & 51.8 & 79.8 & 76.5 & 90.6 & 52.1 & 78.2 & 58.5 & 89.3 & 82.5 & 83.4 & 81.3 & 49.1 & 77.2 & 62.4 & 83.8 & 68.7\\
        %YOLOv2 $544$& COCO+07++12 & 73.4 & 86.3 & 82.0 & 74.8 & 59.2 & 51.8 & 79.8 & 76.5 & 90.6 & 52.1 & 78.2 & 58.5 & 89.3 & 82.5 & 83.4 & 81.3 & 49.1 & 77.2 & 62.4 & 83.8 & 68.7\\

        \noalign{\smallskip}
    \end{tabular}
    \caption{\textbf{PASCAL VOC2012 \texttt{test} detection results.} YOLOv2 performs on par with state-of-the-art detectors like Faster R-CNN with ResNet and SSD512 and is $2-10\times$ faster.}
    \label{voc12}
\end{table*}
```
```latex
\begin{table*}
    \centering
    \setlength{\tabcolsep}{1pt}
    \begin{tabular}{l|c|ccc|ccc|ccc|ccc}
        & & 0.5:0.95 & 0.5 & 0.75 & S & M & L & 1 & 10 & 100 & S & M & L\\
        \hline
        Fast R-CNN \cite{fastrcnn} & train & 19.7 & 35.9 & - & - & - & - & - & - & - & - & - & -\\
        Fast R-CNN\cite{bell2015inside} & train & 20.5 & 39.9 & 19.4 & 4.1 & 20.0 & 35.8 & 21.3 & 29.5 & 30.1 & 7.3 & 32.1 & 52.0\\
        Faster R-CNN\cite{ren2015faster} & trainval & 21.9 & 42.7 & - & - & - & - & - & - & - & - & - & -\\
        ION \cite{bell2015inside} & train & 23.6 & 43.2 & 23.6 & 6.4 & 24.1 & 38.3 & 23.2 & 32.7 & 33.5 & 10.1 & 37.7 & 53.6\\
        Faster R-CNN\cite{coco} & trainval & 24.2 & 45.3 & 23.5 & 7.7 & 26.4 & 37.1 & 23.8 & 34.0 & 34.6 & 12.0 & 38.5 & 54.4\\
        SSD300 \cite{ssd} & trainval35k & 23.2 & 41.2 & 23.4 & 5.3 & 23.2 & 39.6 & 22.5 & 33.2 & 35.3 & 9.6 & 37.6 & 56.5\\
        SSD512 \cite{ssd}& trainval35k & \textbf{26.8} & \textbf{46.5} & \textbf{27.8} & \textbf{9.0} & \textbf{28.9} & \textbf{41.9} & \textbf{24.8} & \textbf{37.5} & \textbf{39.8} & \textbf{14.0} & \textbf{43.5} & \textbf{59.0}\\
        \hline
        YOLOv2 \cite{ssd} & trainval35k & 21.6 & 44.0 & 19.2 & 5.0 & 22.4 & 35.5 & 20.7 & 31.6 & 33.3 & 9.8 & 36.5 & 54.4\\

    \end{tabular}
    \caption{\textbf{Results on COCO \texttt{test-dev2015}. Table adapted from \cite{ssd}}}
    \label{tab:coco}
\end{table*}
```

# Faster

We want detection to be accurate but we also want it to be fast. Most applications for detection, like robotics or self-driving cars, rely on low latency predictions. In order to maximize performance we design YOLOv2 to be fast from the ground up.

Most detection frameworks rely on VGG-16 as the base feature extractor . VGG-16 is a powerful, accurate classification network but it is needlessly complex. The convolutional layers of VGG-16 require 30.69 billion floating point operations for a single pass over a single image at $224 \times 224$ resolution.

The YOLO framework uses a custom network based on the Googlenet architecture . This network is faster than VGG-16, only using 8.52 billion operations for a forward pass. However, it's accuracy is slightly worse than VGG-16. For single-crop, top-5 accuracy at $224 \times 224$, YOLO's custom model gets 88.0% ImageNet compared to 90.0% for VGG-16.

**Darknet-19**. We propose a new classification model to be used as the base of YOLOv2. Our model builds off of prior work on network design as well as common knowledge in the field. Similar to the VGG models we use mostly $3 \times 3$ filters and double the number of channels after every pooling step . Following the work on Network in Network (NIN) we use global average pooling to make predictions as well as $1 \times 1$ filters to compress the feature representation between $3 \times 3$ convolutions . We use batch normalization to stabilize training, speed up convergence, and regularize the model .

Our final model, called Darknet-19, has 19 convolutional layers and 5 maxpooling layers. For a full description see Table . Darknet-19 only requires 5.58 billion operations to process an image yet achieves $72.9\%$ top-1 accuracy and $91.2\%$ top-5 accuracy on ImageNet.

**Training for classification.** We train the network on the standard ImageNet 1000 class classification dataset for 160 epochs using stochastic gradient descent with a starting learning rate of $0.1$, polynomial rate decay with a power of $4$, weight decay of $0.0005$ and momentum of $0.9$ using the Darknet neural network framework . During training we use standard data augmentation tricks including random crops, rotations, and hue, saturation, and exposure shifts.

|     Type      | Filters |   Size/Stride    |      Output      |
|:-------------:|:-------:|:----------------:|:----------------:|
| Convolutional |   32    |   $3 \times 3$   | $224 \times 224$ |
|    Maxpool    |         | $2 \times 2 / 2$ | $112 \times 112$ |
| Convolutional |   64    |   $3 \times 3$   | $112 \times 112$ |
|    Maxpool    |         | $2 \times 2 / 2$ |  $56 \times 56$  |
| Convolutional |   128   |   $3 \times 3$   |  $56 \times 56$  |
| Convolutional |   64    |   $1 \times 1$   |  $56 \times 56$  |
| Convolutional |   128   |   $3 \times 3$   |  $56 \times 56$  |
|    Maxpool    |         | $2 \times 2 / 2$ |  $28 \times 28$  |
| Convolutional |   256   |   $3 \times 3$   |  $28 \times 28$  |
| Convolutional |   128   |   $1 \times 1$   |  $28 \times 28$  |
| Convolutional |   256   |   $3 \times 3$   |  $28 \times 28$  |
|    Maxpool    |         | $2 \times 2 / 2$ |  $14 \times 14$  |
| Convolutional |   512   |   $3 \times 3$   |  $14 \times 14$  |
| Convolutional |   256   |   $1 \times 1$   |  $14 \times 14$  |
| Convolutional |   512   |   $3 \times 3$   |  $14 \times 14$  |
| Convolutional |   256   |   $1 \times 1$   |  $14 \times 14$  |
| Convolutional |   512   |   $3 \times 3$   |  $14 \times 14$  |
|    Maxpool    |         | $2 \times 2 / 2$ |   $7 \times 7$   |
| Convolutional |  1024   |   $3 \times 3$   |   $7 \times 7$   |
| Convolutional |   512   |   $1 \times 1$   |   $7 \times 7$   |
| Convolutional |  1024   |   $3 \times 3$   |   $7 \times 7$   |
| Convolutional |   512   |   $1 \times 1$   |   $7 \times 7$   |
| Convolutional |  1024   |   $3 \times 3$   |   $7 \times 7$   |
| Convolutional |  1000   |   $1 \times 1$   |   $7 \times 7$   |
|    Avgpool    |         |      Global      |      $1000$      |
|    Softmax    |         |                  |                  |

**Darknet-19.**

As discussed above, after our initial training on images at $224 \times 224$ we fine tune our network at a larger size, $448 \time 448$. For this fine tuning we train with the above parameters but for only 10 epochs and starting at a learning rate of $10^{-3}$. At this higher resolution our network achieves a top-1 accuracy of $76.5\%$ and a top-5 accuracy of $93.3\%$.

**Training for detection.** We modify this network for detection by removing the last convolutional layer and instead adding on three $3 \times 3$ convolutional layers with $1024$ filters each followed by a final $1\times 1$ convolutional layer with the number of outputs we need for detection. For VOC we predict 5 boxes with 5 coordinates each and 20 classes per box so 125 filters. We also add a passthrough layer from the final $3 \times 3 \times 512$ layer to the second to last convolutional layer so that our model can use fine grain features.

We train the network for 160 epochs with a starting learning rate of $10^{-3}$, dividing it by 10 at 60 and 90 epochs. We use a weight decay of $0.0005$ and momentum of $0.9$. We use a similar data augmentation to YOLO and SSD with random crops, color shifting, etc. We use the same training strategy on COCO and VOC.

# Stronger

We propose a mechanism for jointly training on classification and detection data. Our method uses images labelled for detection to learn detection-specific information like bounding box coordinate prediction and objectness as well as how to classify common objects. It uses images with only class labels to expand the number of categories it can detect.

During training we mix images from both detection and classification datasets. When our network sees an image labelled for detection we can backpropagate based on the full YOLOv2 loss function. When it sees a classification image we only backpropagate loss from the classification-specific parts of the architecture.

This approach presents a few challenges. Detection datasets have only common objects and general labels, like "dog" or "boat". Classification datasets have a much wider and deeper range of labels. ImageNet has more than a hundred breeds of dog, including "Norfolk terrier", "Yorkshire terrier", and "Bedlington terrier". If we want to train on both datasets we need a coherent way to merge these labels.

Most approaches to classification use a softmax layer across all the possible categories to compute the final probability distribution. Using a softmax assumes the classes are mutually exclusive. This presents problems for combining datasets, for example you would not want to combine ImageNet and COCO using this model because the classes "Norfolk terrier" and "dog" are not mutually exclusive.

We could instead use a multi-label model to combine the datasets which does not assume mutual exclusion. This approach ignores all the structure we do know about the data, for example that all of the COCO classes are mutually exclusive.

**Hierarchical classification.** ImageNet labels are pulled from WordNet, a language database that structures concepts and how they relate . In WordNet, "Norfolk terrier" and "Yorkshire terrier" are both hyponyms of "terrier" which is a type of "hunting dog", which is a type of "dog", which is a "canine", etc. Most approaches to classification assume a flat structure to the labels however for combining datasets, structure is exactly what we need.

WordNet is structured as a directed graph, not a tree, because language is complex. For example a "dog" is both a type of "canine" and a type of "domestic animal" which are both synsets in WordNet. Instead of using the full graph structure, we simplify the problem by building a hierarchical tree from the concepts in ImageNet.

To build this tree we examine the visual nouns in ImageNet and look at their paths through the WordNet graph to the root node, in this case "physical object". Many synsets only have one path through the graph so first we add all of those paths to our tree. Then we iteratively examine the concepts we have left and add the paths that grow the tree by as little as possible. So if a concept has two paths to the root and one path would add three edges to our tree and the other would only add one edge, we choose the shorter path.

The final result is WordTree, a hierarchical model of visual concepts. To perform classification with WordTree we predict conditional probabilities at every node for the probability of each hyponym of that synset given that synset. For example, at the "terrier" node we predict:

$$\begin{aligned}
Pr(\text{Norfolk terrier} &| \text{terrier}) \\
Pr(\text{Yorkshire terrier} &| \text{terrier}) \\
Pr(\text{Bedlington terrier} &| \text{terrier})\\
...&\\
\end{aligned}$$

If we want to compute the absolute probability for a particular node we simply follow the path through the tree to the root node and multiply to conditional probabilities. So if we want to know if a picture is of a Norfolk terrier we compute:

$$\begin{aligned}
Pr(\text{Norfolk terrier}) &= Pr(\text{Norfolk terrier} | \text{terrier})\\
 * Pr(\text{terrier} &| \text{hunting dog}) \\
* \ldots& * \\
*Pr(\text{mammal} &| Pr(\text{animal})\\
 * Pr(\text{animal} &| \text{physical object})
\end{aligned}$$

For classification purposes we assume that the the image contains an object: $Pr(\text{physical object}) = 1$.

To validate this approach we train the Darknet-19 model on WordTree built using the 1000 class ImageNet. To build WordTree1k we add in all of the intermediate nodes which expands the label space from 1000 to 1369. During training we propagate ground truth labels up the tree so that if an image is labelled as a "Norfolk terrier" it also gets labelled as a "dog" and a "mammal", etc. To compute the conditional probabilities our model predicts a vector of 1369 values and we compute the softmax over all sysnsets that are hyponyms of the same concept, see Figure .

Using the same training parameters as before, our hierarchical Darknet-19 achieves $71.9\%$ top-1 accuracy and $90.4\%$ top-5 accuracy. Despite adding 369 additional concepts and having our network predict a tree structure our accuracy only drops marginally. Performing classification in this manner also has some benefits. Performance degrades gracefully on new or unknown object categories. For example, if the network sees a picture of a dog but is uncertain what type of dog it is, it will still predict "dog" with high confidence but have lower confidences spread out among the hyponyms.

This formulation also works for detection. Now, instead of assuming every image has an object, we use YOLOv2's objectness predictor to give us the value of $Pr(\text{physical object})$. The detector predicts a bounding box and the tree of probabilities. We traverse the tree down, taking the highest confidence path at every split until we reach some threshold and we predict that object class.

**Dataset combination with WordTree.** We can use WordTree to combine multiple datasets together in a sensible fashion. We simply map the categories in the datasets to synsets in the tree. Figure shows an example of using WordTree to combine the labels from ImageNet and COCO. WordNet is extremely diverse so we can use this technique with most datasets.

**Joint classification and detection.** Now that we can combine datasets using WordTree we can train our joint model on classification and detection. We want to train an extremely large scale detector so we create our combined dataset using the COCO detection dataset and the top 9000 classes from the full ImageNet release. We also need to evaluate our method so we add in any classes from the ImageNet detection challenge that were not already included. The corresponding WordTree for this dataset has 9418 classes. ImageNet is a much larger dataset so we balance the dataset by oversampling COCO so that ImageNet is only larger by a factor of 4:1.

Using this dataset we train YOLO9000. We use the base YOLOv2 architecture but only 3 priors instead of 5 to limit the output size. When our network sees a detection image we backpropagate loss as normal. For classification loss, we only backpropagate loss at or above the corresponding level of the label. For example, if the label is "dog" we do assign any error to predictions further down in the tree, "German Shepherd" versus "Golden Retriever", because we do not have that information.

When it sees a classification image we only backpropagate classification loss. To do this we simply find the bounding box that predicts the highest probability for that class and we compute the loss on just its predicted tree. We also assume that the predicted box overlaps what would be the ground truth label by at least $.3$ IOU and we backpropagate objectness loss based on this assumption.

Using this joint training, YOLO9000 learns to find objects in images using the detection data in COCO and it learns to classify a wide variety of these objects using data from ImageNet.

We evaluate YOLO9000 on the ImageNet detection task. The detection task for ImageNet shares on 44 object categories with COCO which means that YOLO9000 has only seen classification data for the majority of the test images, not detection data. YOLO9000 gets 19.7 mAP overall with 16.0 mAP on the disjoint 156 object classes that it has never seen any labelled detection data for. This mAP is higher than results achieved by DPM but YOLO9000 is trained on different datasets with only partial supervision . It also is simultaneously detecting 9000 other object categories, all in real-time.

|                 |      |
|:----------------|:----:|
| diaper          | 0.0  |
| horizontal bar  | 0.0  |
| rubber eraser   | 0.0  |
| sunglasses      | 0.0  |
| swimming trunks | 0.0  |
| ...             |      |
| red panda       | 50.7 |
| fox             | 52.1 |
| koala bear      | 54.3 |
| tiger           | 61.0 |
| armadillo       | 61.7 |

**YOLO9000 Best and Worst Classes on ImageNet.** The classes with the highest and lowest AP from the 156 weakly supervised classes. YOLO9000 learns good models for a variety of animals but struggles with new classes like clothing or equipment.

When we analyze YOLO9000's performance on ImageNet we see it learns new species of animals well but struggles with learning categories like clothing and equipment. New animals are easier to learn because the objectness predictions generalize well from the animals in COCO. Conversely, COCO does not have bounding box label for any type of clothing, only for person, so YOLO9000 struggles to model categories like "sunglasses" or "swimming trunks".

# Conclusion

We introduce YOLOv2 and YOLO9000, real-time detection systems. YOLOv2 is state-of-the-art and faster than other detection systems across a variety of detection datasets. Furthermore, it can be run at a variety of image sizes to provide a smooth tradeoff between speed and accuracy.

YOLO9000 is a real-time framework for detection more than 9000 object categories by jointly optimizing detection and classification. We use WordTree to combine data from various sources and our joint optimization technique to train simultaneously on ImageNet and COCO. YOLO9000 is a strong step towards closing the dataset size gap between detection and classification.

Many of our techniques generalize outside of object detection. Our WordTree representation of ImageNet offers a richer, more detailed output space for image classification. Dataset combination using hierarchical classification would be useful in the classification and segmentation domains. Training techniques like multi-scale training could provide benefit across a variety of visual tasks.

For future work we hope to use similar techniques for weakly supervised image segmentation. We also plan to improve our detection results using more powerful matching strategies for assigning weak labels to classification data during training. Computer vision is blessed with an enormous amount of labelled data. We will continue looking for ways to bring different sources and structures of data together to make stronger models of the visual world.


abstract: This study investigates the feasibility of developing a Conversational & Universal Recommender Engine (CURE), facilitated by recent advancements in Large Language Models (LLMs). A CURE comprises both conversationality and universality to engage in natural dialogues and generate recommendations across various domains. We propose ten fundamental principles that a CURE should adhere to, each with its corresponding evaluation protocols. We proceed to evaluate whether ChatGPT, a sophisticated LLM, can comply with the proposed principles by engaging in recommendation-oriented dialogues with the model while observing its behavior. Our findings demonstrate the potential for ChatGPT to serve as a CURE, though several limitations and areas for improvement are identified.
author: Ben Trovato; G.K.M. Tobin; Lars Thørväld; Valerie Béranger; Aparna Patel; Huifen Chan; Charles Palmer; John Smith; Julius P. Kumquat
bibliography: sample-base.bib
title: CURE: Conversational Universal Recommendation Engine - Principles and Early Experiments with ChatGPT

# Introduction

Recent advancements in large language models (LLMs) have enabled the possible development of a Conversational $\&$ Universal Recommender Engine, denoted as CURE. At its core, CURE encompass two primary aspects: (1) Conversationality, implying the ability to engage in interactive, natural dialogues with users, and (2) Universality, signifying the ability to execute a multitude of tasks and generate personalized recommendations for a variety of item domains. In this study, we aim to establish the fundamental principles to which a CURE should adhere. Subsequently, we assess the capacity of ChatGPT, a sophisticated LLM, to serve as a CURE by employing the evaluation protocols designed for each principle. Drawing from the insights of Microsoft's recent paper on GPT evaluations , this study aims to extend and adapt the evaluation of ChatGPT to the domain of Recommender Systems, following a similar instance-based evaluation approach employed by the referenced paper.

# CURE Principles

In this section, we outline ten fundamental principles that a CURE should be capable of achieving. Each principle is accompanied by its definition and corresponding evaluation protocols. To appraise ChatGPT's adherence to such principles, we imitate end-users and engage in dialogues with the model to observe its behaviors. We also request the model to perform additional tasks to further examine its alignment with particular principle criteria, as needed.

#### **Mixed Initiative:**

to facilitate a natural conversation, both the user and the system should be able to actively participate in the dialogue, rather than limiting the user to initiating interactions while the system merely responds, or using a system that strictly adheres to a pre-determined script for the dialogue . Additionally, users may engage in rapid reading and accidentally overlook or misinterpret some of the system's utterances . Therefore, the system should exhibit the ability to autonomously initiate actions, including: actively asking questions to solicit user information for more precise recommendations, posing follow-up questions (such as seeking user clarifications) and providing supplementary statements for its previous utterances when necessary .

-   Evaluation Protocol: to evaluate the model's ability to proactively initiate actions, we conduct two tests. Firstly, we intentionally provide responses that do not align or are irrelevant with inquiries previously asked by the model, and observe whether the model could determine that the answers provided were not proper for the questions asked and ask for user clarifications. Secondly, we intentionally alter the meaning of words and/or sentences stated in previous model utterances to simulate cases of misreading and compose user input statements with such false assumptions by explicitly referring to them in the statement. We then observe whether the model could identify the misunderstanding and present the misread portion to the user in an easily understandable manner.

#### **Acquisition Strategy:**

the system should aim to strike a balance between the number of questions posed and the relevance of the recommendations provided by efficiently collecting all necessary information for precise recommendations while avoiding excessive user fatigue arising from multi-turn dialogues . One potential approach could involve employing various heuristics to consistently calculate and identify the most informative questions to present to users, thereby maximizing information gain while minimizing the user's cognitive burden .

-   Evaluation Protocol: to comprehend the underlying rationale behind the model's selection of questions, we engage in a recommendation-oriented dialogue with the model and probe its reasoning for choosing particular questions. Simultaneously, we prompt the model to propose alternative questions that are also relevant to the subject and inquire as to why the model did not choose such queries. We then proceed to assess the explanations offered by the model in response to our inquiries.

#### **Contextual Memory:**

during dialogues, users may frequently refer to prior stated items without providing a full description . For example, in phone recommendations, users may pose questions such as: "How does this phone compare to the ones I mentioned earlier?" To enhance the continuity and naturalness of such interactions, the system should be able to store and retrieve such referential entities for potential future use. Additionally, other user-related information, such as metadata and interaction history, should also be stored and recalled if necessary. Lastly, the system should demonstrate the ability to remember its own previous statements within an ongoing dialogue.

-   Evaluation Protocol: to assess the model's capacity for retaining and employing previously presented information, we engage in dialogues with the model. During the initial round, we provide the user's metadata and/or interaction history. We then evaluate the model's ability to recollect this information after a certain number of rounds. We also prompt the model to output its prior utterances and inspect whether any statements are omitted. Moreover, we introduce a referential entity, such as an additional item or a friend of the user, at the beginning of the dialogue. Later in the conversation, we refer back to this entity for comparative purposes and examine the model's capacity for restoring and utilizing all the information from the entity to respond to user inquiries.

#### **Repair Mechanism:**

given the possibility of modifications in user stated information at a later stage of the dialogue, it is imperative to incorporate a repair mechanism. This feature enables users to provide supplementary details, amend inaccuracies, or even remove previous utterances. To accommodate these adjustments, the system must be designed to generate suitable recommendations and explanations based solely on the updated information.

-   Evaluation Protocol: to examine the model's ability to adapt to changes stemming from previous user statements, we refer back to previously provided metadata and/or user interactions and make modifications to them in subsequent rounds of the conversation. The model's capacity to discard outdated data while utilizing only the modified information for recommendation and reasoning will then be observed.

#### **Feedback Mechanism:**

users may struggle in explicitly articulating the rationale behind their dissatisfaction with the recommended items . This predicament poses an obstacle for the system in acquiring user preferences from recommendations it may propose throughout the dialogue. To mitigate this issue, the system should be designed to make adjustments based on both feedback that convey reasons for the undesirability of the recommended items, as well as feedback that merely states that the items are unsatisfactory without any providing any explanations. In the latter scenario, the system should be able to generate educated conjectures about why the recommended outputs are undesired, according to all information collected during the dialogue.

-   Evaluation Protocol: we intentionally provide negative feedback in two forms: explicit natural language-based explanations detailing the dissatisfaction, and implicit expressions of dissatisfaction without accompanying reasons. In cases of explicit feedback, we prompt the model to generate an alternative recommendations, accompanied by a rationale, and observe whether the newly recommended items addresses all concerns raised in the feedback. For implicit feedback, the model is assessed on its ability to infer the underlying causes of dissatisfaction using contextual and user information.

#### **Entity Inference:**

The system should be designed to handle situations where users struggle to remember the exact name or title of an item they want to discuss during a conversation, but can recall specific attributes or details . In such instances, the system should be able to make informed guesses, providing a brief summary for each proposed item based on the available information, and seek user confirmation. This functionality should be maintained even when the user-supplied details contain errors or inaccuracies.

-   Evaluation Protocol: to assess the model's ability to infer the intended items mentioned by the user based on limited details, We manually selected a number of movies on Amazon in order to simulate items that users may have forgotten. We then request the model to provide an informed guess for the forgotten item by providing a certain amount of metadata and/or item description. Note that the guessed item can still be considered qualified if the model proposes a similar item that fulfills the intended functionality of the reserved item and align with the provided item description, even if it fails to identify the exact item.

#### **Behavioral Analysis:**

The system should exhibit the capability to systematically examine user interaction history to identify potential alternations in user behavior , thus recognizing the evolution of user preferences, such as transitions in preferred movie genres as one ages. This functionality allows the system to capture "at-the-moment" user preferences and offer contextually pertinent recommendations in real-time. Additionally, the system should possess the capacity to propose supplementary facets beyond the ones from item metadata, to facilitate a more in-depth analysis of user preferences.

-   Evaluation Protocol: we propose interaction histories that include facet-based behavior changes, such as transitioning from purchasing items of one brand or franchise to another or switching from a preferred size to an alternative. We then examine the system's capacity to detect potential shifts and/or expansions in user preferences when prompted. Additionally, we prompt the model to propose supplementary facets and conduct preference elicitation based on these additional aspects then evaluate the rationale behind their utterances.

#### **Inconsistency Detection:**

this principle aims at addressing the three major types of inconsistencies: logical, expectational, and factual, as defined by . Specifically, the system should be able to accurately detect user made inconsistent statements that fall under each of the three aforementioned types within a dialogue to establish system accountability. Additionally, it should provide explanations for the detected inconsistencies, while incorporating an appropriate amount of contextual information derived or cited from user inputs.

-   Evaluation Protocol: to assess the model's capacity to detect inconsistencies in user statements for each type, we design three distinct scenarios: $(1)$. logical inconsistency: for this scenario, we first provide the model with the interaction history and/or metadata of a sampled user, followed by statements that clearly contradict the previously provided information. $(2)$. expectational inconsistency: our second scenario investigates the model's capability to recognize both $(a)$. scenarios where the desired facet(s) are physically and/or technologically difficult to achieve simultaneously in a single item, and $(b)$. scenarios where the desired facet(s), although unexpected, can be incorporated into the same item. For the first case type, we observe the model's capacity to request facet prioritization from the user or to propose items that offer a compromise. In contrast, for the second case type, we evaluate the model's capability to recommend items that successfully integrate all desired facets. $(3)$. factual inconsistency: the final scenario tests the model's proficiency in identifying factual inconsistencies presented in two different contexts: $(a)$. In a concise statement containing only the inconsistency without any additional descriptions or details. and $(b)$. In a complex statement where the inconsistency is embedded within a substantial amount of descriptive information, making the statement appear more credible. For each scenario, we evaluate the model's ability to identify the inconsistencies within user utterance(s), explain the underlying rationale of such inconsistencies with great readability, and accurately classify the type of inconsistency.

#### **Personalized Recommendation:**

given the limited number of items that can be recommended simultaneously , providing a detailed explanation for each item becomes crucial. Thus, the system should be designed to integrate personalized information, such as user interaction history and metadata, by generating explanations that clearly illustrate how such personalized information has been employed to derive the recommended items. Furthermore, the system should be capable of relating each recommended item to the user's previous interactions to foster familiarity.

-   Evaluation Protocol: we focus on the movie domain, providing user metadata such as age, gender, occupation, and zip code, as well as a viewing history containing chronologically ordered films watched with accompanying ratings. We then prompt the model to generate movie recommendations with explanations and engage in dialogue to identify any potential shortcomings in its ability to incorporate user information in its reasoning process. Moreover, we assess the validity of the connections established between the recommended items and the items the user has interacted with.

#### **Extrinsic Factors:**

the system should possess the ability to recognize and utilize a comprehensive set of external factors such as the user's upcoming tasks or plans, time of the day, and recent events in order to generate recommendations that align with the user's immediate needs and preferences . By incorporating these elements, the system can offer personalized and context-aware recommendations that are more relevant to the user's current situation.

-   Evaluation Protocol: we prompt the model to observe whether it possesses the knowledge, and is able to explain, of which extrinsic factors can impact the decision-making for an item domain. We also provide extrinsic factors to the model during recommendation process to evaluate how well it proposes recommendations based on such data.

# Evaluation Details & Insights

As a CURE operates by generating recommendations through interactive conversations with users, we conduct instance-based reflective analysis by engaging in recommendation-oriented dialogues with ChatGPT. We prompt the model to carry out specific tasks, enabling us to observe its behavior and evaluate its potential to serve as a CURE. Our approach is greatly inspired by Microsoft's survey paper on GPTs , which presented meticulously designed instances to assess ChatGPT's capabilities for various natural language tasks, with an emphasis on identifying potential limitations.

**Mixed Initiative**: As demonstrated in Figure MI-1, we initiate the process by providing instruction prompts that explicitly convey our desired actions for the model, ensuring that it has a clear understanding of the principle in question. We then request book recommendations without providing any user information. In response, the model generates a list of five questions in order to perform more personalized recommendations.

Adhering to the protocol, we deliberately provide answers that are clearly inconsistent with the proposed questions, such as mentioning a lack of preference for video games (rather than book genres) and acknowledging a list of favorite writers without specifying any names. Despite these discrepancies, the model accepts the incorrect answers and offers a list of book recommendations without posing follow-up questions to seek clarification on the ambiguous responses.

To evaluate the second case in the protocol, we intentionally present the model with a statement that clearly contradicts a previous model utterance by reversing its meaning, and then observe whether the model identifies the inconsistency (as shown in the penultimate statement in Figure MI-1). Once again, the model fails to detect the user's misunderstanding and merely responds with a thank you message.

**Acquisition Strategy**: In Figure AS-1, the user provides metadata and viewing history, prompting the model to generate movie recommendations (omitted due to space constraints). We then ask the model to suggest additional questions for more personalized recommendations, and the model proposes five. To assess the relevance of these questions, we first inquire about the rationale behind each; the model offers reasonable explanations, demonstrating its thought process. Subsequently, we question why another query is deemed less relevant than the five suggested questions. However, the model's reasoning appears to be imprecise, as the reasoning could also apply to the question "Do you prefer movies that are more serious or light-hearted in tone?" initially proposed by the model. When we point out this oversight, the model apologizes and concedes that both questions are equally relevant.

**Contextual Memory**: While the model demonstrates the ability to retain previously stated user information for recommendations, it appears to primarily align its outputs with the most recent user input, without accounting for the interrelationships between various user utterances. As Figure CM-1 showcases, we first indicated having severe allergies with any nuts, then request restaurants with dishes featuring walnuts and peanut butter. The model simply proceeds to generate recommendations based on the latter request, without ascertaining whether walnuts and peanut butter belong to the nut category. However, in subsequent interactions, the model acknowledges that walnuts and peanut butter are classified as nuts, and simultaneously recalls the user's severe nut allergy, advising against their consumption. This observation underscores the model's capacity to remember prior user utterances across multiple rounds of interaction. However, it may also reveal a propensity for the model to generate immediate responses that address the current user prompt, without thoroughly evaluating the consequences in light of all relevant context.

In a separate dialogue (Figure CM-2), we present a referential entity and request for phone recommendations. We then prompt the model to compare its proposed items with the referential entity to evaluate its information retention. Moreover, we assess the model's ability to retain its own utterances by requesting the model to output the total number of phones recommended and the most expensive ones. The results show that the model successfully performs all the given tasks.

**Repair Mechanism**: Following the proposed protocol, we observe that although the model can modify retained user information based upon user requests, the user needs to explicitly state the statements to discard to avoid the model's usage of invalid information after modification. To illustrate, Figure RM-1 depicts in scenario where the user initially states that he had sushi once in his lifetime but enjoyed it. He than modifies his statement to never had sushi before. The model promptly adjusts its response to accommodate this new information and provides a revised recommendation. However, the model fails to recognize the causal relationship between this statement with the user's initial statement. When the user asks the model to guess whether he is a sushi lover, the model responds that "*based on the user's initial statement that he/she enjoyed the taste of sushi, it's possible that the user might enjoy sushi and other Japanese cuisine*". This demonstrates that the model treats both statements as separate pieces of user information without considering relevant causal relationships and discard outdated data, which is what the user would expect the model to perform upon processing the second statement.

When the user provides a correction statement: "*Since I have never had sushi before, the statement 'that I really enjoyed the taste of sushi' is also invalid, as I have not tried it*," the model interprets the relationship between the previous statements and internally "deletes" the initial utterance made by the user. It then replies correctly by stating "*Since you mentioned that you have never had sushi before, it's difficult to say whether you will enjoy it or not*," in response to whether the user favors sushi (omitted due to space constraints).

**Feedback Mechanism**: In Figure FM-1, the model recommends three movies based on the provided user information. We then offer implicit feedback, stating our disinterest in the movie "Inception," and request the model to speculate on potential reasons. The model successfully proposes two valid points concerning personal taste and occupation. It highlights that "Inception" has complex plots, whereas the movies previously watched by the user typically feature straightforward narratives. Furthermore, the model emphasizes the sharp contrast between the user's preference for "The Social Network," a grounded and realistic drama, and "Inception," a film that delves profoundly into philosophical questions (omitted due to space constraints). We continue by offering the model explicit feedback, clearly outlining our preferences. In response, the model suggests three new movies that meet all of our criteria (manually confirmed via external sources) and provides a concise summary for each recommended item.

**Entity Inference**: We propose four scenarios for a holistic assessment. Figure EI-1 demonstrates a case where the model infers an item based on accurate information provided by the user. The model successfully identifies the movie's name, even with limited details, and presents a concise summary of the film's main storyline, along with relevant metadata such as the director and the main cast.

Building upon the first case, we simulate a scenario in which the user provides partially correct information, as depicted in Figure EI-2. The erroneous information that we provide pertains to the movie's World War II background and appears plausible with specific actions and plot-line. We then observe whether the model can still accurately infer the item under these conditions. While the model manages to deduce the movie correctly, it begins to hallucinate in its movie summary by incorporating false information that aligns and further elaborates on the user's incorrect input. This behavior could potentially be problematic, as users may possess vague memories and inadvertently mix up details from different items. Ideally, the model should be able to discern the incorrect information and provide an accurate description of the item. Erroneous information presented by the model may adversely affect users' decision-making and could be utilized in subsequent recommendation reasoning by the model, further compromising its effectiveness.

In the third case, we present a movie description identical to the one displayed on Amazon for a film released in 2023. Given the model's knowledge cutoff in 2021, we aim to observe whether the model can identify that the movie is not within its knowledge base and also suggest movies with similar storylines, as illustrated in Figure EI-3. We observe that the model is unable to recognize that the movie falls outside its knowledge base and subsequently proposes two guesses. Although the model correctly identifies "Sweet Home Alabama" as a movie with a similar opening to the user description, it exhibits hallucination by altering the name of a character in the film to match a name from the user description, possibly in an attempt to align with the user's input (omitted due to space constraints).

In the final case, we manually select an unpopular movie released on Amazon in 2020 (prior to the model's knowledge cutoff) that has obtained only 68 global reviews. As depicted in Figure EI-4, we present the complete movie description as it appears on Amazon, along with the name of a main actor, and observe that the model is unable to accurately identify the movie. Instead, the model exhibits hallucination again by suggesting a nonexistent movie with a fabricated plot, while incorporating portions of the provided description to enhance the credibility of its response.

**Behavioral Analysis**: As depicted by Figure BA-1, the model demonstrates the ability to conduct facet-based comparisons among different products based on facets provided in the metadata to identify changes in user preferences. Furthermore, the model is also capable of conducting facet-based comparisons for preference elicitation on more abstract facets, in addition to basic item metadata (which could be considered as metadata augmentation). For instance, in the context of movie recommendations, the model suggests seven aspects pertinent to decision-making upon user request, extending beyond elementary metadata such as genre and movie duration, including: theme and motif, historical context, plot and storyline, style and visual aesthetics, diction difficulty, emotional resonance, and cultural relevance. The model then analyzes the user's watch history based on these augmented facets. Figure BA-2 illustrates how the model combines user ratings with each movie's plot and storyline to deduce that the user may have a preference for heartwarming stories while showing less interest in whimsical or musical-themed movies.

Nevertheless, the conclusion formulated by the model may present potential logical inconsistencies. As depicted in Figure BA-3, upon further inquiry, the model identifies both "My Neighbor Totoro" and "Singin' in the Rain" as heartwarming stories featuring relatable characters and relationships. This observation contradicts the earlier conclusion that emphasizes the user's preference for heartwarming stories, considering that all four movies possess heartwarming narratives, yet two of them received low ratings from the user. Subsequently, the model concedes that its initial conclusion is an oversimplification.

**Inconsistency Detection**: As depicted in Figure ID-1, ChatGPT successfully identifies the logical inconsistency within the user's statement, highlighting that the referenced movies did not align with the user's description. For expectational inconsistency, we first propose a scenario where we prompt the model for a cutting-edge smartphone with month-long battery. The model successfully recognizes the in-feasibility in the combination of facets by stating that such phones don't exist in the market. We proceed to propose another inquiry for a novel with two seemingly contradicting traits, and the model exhibits ability to propose items that integrate the two traits, as illustrated by Figure ID-2. For factual inconsistency, we follow the protocol by presenting an extended statement (shown in Figure ID-3), wherein the user requests movie recommendations while providing a list of movies accompanied by their viewing dates. The inconsistency is embedded within one of the dates, while all other dates are valid, in order to make the statement appear more credible. As a result, the model does not detect the factual inconsistency and proceeds to recommend a set of movies. We further prompt the model on whether it identifies any inconsistencies; the model summarizes the user's statement and maintains that "*everything appears to be logically consistent and factually accurate.*" We then proceed by explicitly inquiring if "02/29/2021" is a valid date, corresponding to the evaluation of a concise factual inconsistency statement without additional descriptions. The model ultimately recognizes the inconsistency, provides an explanation, and also identifies the inconsistency within the initial user statement.

**Personalized Recommendation**: Although the model exhibits ability to utilize user metadata and interaction history in movie recommendations, it may produce over-generalizations based on the user metadata, as evidenced by its recommendation of the movie "Amélie (2001)" based on the assumption that "*a 30-year-old female may be more likely to enjoy romantic comedies*" (omitted due to space constraints). Furthermore, the model exhibits a tendency to generate statements that lack logical coherence in its recommendation explanations. As evidenced in Figure PR-1, the model-generated explanation fails to clarify why being a 30-year-old female residing in Columbus, Ohio would be pertinent or adequate information for inferring a preference for watching movies focused on historical events and social justice issues. Upon prompting the model to explain the logical coherence behind its conclusion, it acknowledges the errors and extends an apology.

We continue our evaluation by prompting the model to identify connections between the recommended movies and the movies previously watched by the user, as demonstrated in Figure PR-2 (note in this dialogue, we set Singin' in the Rain to be rated at 4-star instead). The model displays competence in establishing valid associations between a recommended movie and a film the user has watched before. We further request the model to rank recommended films by taking into account the provided user ratings while offering a rationale for each ranking. Throughout the process, the model demonstrates a number of logical inconsistencies in its explanations. As an example, it places "Amélie" second, based on the fact that the user has previously viewed a film with similar style and aesthetic, while rating the film with two-star. In contrast, the model ranks "Arrival" at last, citing the user's three-star rating for "Forest Gump" as evidence that the user may not prefer movies with intricate conclusions, as depicted in Figure PR-3. Consequently, the model displays a contradiction in its approach to ranking movies based on user ratings; it assigns a higher ranking to "Amélie" despite the user having given a similar film a lower rating (two stars), while allocating a lower ranking to "Arrival" based on the user's higher rating (three stars) for "Forest Gump".

**Extrinsic Factors**: As Figure EF-1 illustrates, we initially prompt the model to propose various external factors, in addition to metadata and interaction history, to consider when generating movie recommendations. The model responds by presenting five relevant factors: time of day, weather, occasion, social media activity, and popular trends, each accompanied by a rationale (Figure omitted due to space constraints). We follow up on the first factor, inquiring about quantitative support. The model is unable to provide this but offers speculative reasoning grounded in common sense. A similar trend is observed for the second factor upon further inquiry. Subsequently, we assess the model's capacity to leverage common sense by requesting movie recommendations on the eve of a critical exam. The model demonstrates effective utilization of common sense by suggesting movies that are relaxing without being excessively emotionally or mentally taxing. Lastly, we complicate the scenario by prompting the model to generate recommendations based on two known external factors: on the eve of a critical exam and a subsequent travel plan. The model initially recommends engaging movies that align with the travel plan. However, upon further inquiry regarding potential concerns associated with suggesting travel-related movies on the night before a significant exam, the model exhibits awareness of the potential impact on the student's focus by recommending such films. Instead, it proposes suggesting relaxing and calming movies rather than ones that may heighten the student's excitement or anxiety levels.

# Conclusion and Future work

In this work, we propose ten core principles a CURE should comply with, and evaluate the potential for ChatGPT

 ———————————————————————————————— dropped work

# Conclusion

-   Summarizing study contributions.

-   Limitations in both testing strategies and some potential relevant points (ex. confidence, language alignment, etc.)

-   Some future work (more human-centric testings, evaluating the effective of prompting on GPT, etc.)