We propose a novel method for predicting image labels by fusing image content descriptors with the social media context of each image. An image uploaded to a social media site such as Flickr often has meaningful, associated information, such as comments and other images the user has uploaded, that is complementary to pixel content and helpful in predicting labels. Prediction challenges such as ImageNet imagenet_cvpr09 and MSCOCO LinMBHPRDZ:ECCV14 use only pixels, while other methods make predictions purely from social media context McAuleyECCV12 . Our method is based on a novel fully connected Conditional Random Field (CRF) framework, where each node is an image, and consists of two deep Convolutional Neural Networks (CNN) and one Recurrent Neural Network (RNN) that model both textual and visual node image information. The edge weights of the CRF graph represent textual similarity and link-based metadata such as user sets and image groups. We model the CRF as an RNN for both learning and inference, and incorporate the weighted ranking loss and cross entropy loss into the CRF parameter optimization to handle the training data imbalance issue. Our proposed approach is evaluated on the MIR-9K dataset and experimentally outperforms current state-of-the-art approaches.
@cite_13: 
@cite_14: 
@cite_7: Typical tag recommendation systems for photos shared on social networks such as Flickr, use visual content analysis, collaborative filtering or personalization strategies to produce annotations. However, the dependence on manual intervention and the knowledge of sufficient personal preferences coupled with the folksonomic issues limit the scope of these strategies. In this paper, we present a fully automatic and folksonomically scalable tag recommendation model that can recommend tags for a user's photos without an explicit knowledge of the user's personal tagging preferences. The model is learned using the collective tagging behavior of other users in the user's local interaction network, which we believe approximates the user's preferences, at least partially. The tag recommendation model generates content-based annotations and then uses a Naive Bayes formulation to translate these annotations to a set of folksonomic tags selected from the tags used by the users in the local interaction network. Quantitative and qualitative comparisons with 890 Flickr networks show that this approach is highly useful for tag recommendation in the presence of insufficient information of a user's own preferences.
@cite_28: 
@cite_6: 
@cite_27: Imagine you are member of an online social system and want to upload a picture into the community pool. In current social software systems, you can probably tag your photo, share it or send it to a photo printing service and multiple other stuff. The system creates around you a space full of pictures, other interesting content (descriptions, comments) and full of users as well. The one thing current systems do not do, is understand what your pictures are about. We present here a collection of functionalities that make a step in that direction when put together to be consumed by a tag recommendation system for pictures. We use the data richness inherent in social online environments for recommending tags by analysing different aspects of the same data (text, visual content and user context). We also give an assessment of the quality of thus recommended tags.
@cite_23: 
@cite_10: Most personal photos that are shared online are embedded in some form of social network, and these social networks are a potent source of contextual information that can be leveraged for automatic image understanding. In this paper, we investigate the utility of social network context for the task of automatic face recognition in personal photographs. We combine face recognition scores with social context in a conditional random field (CRF) model and apply this model to label faces in photos from the popular online social network Facebook, which is now the top photo-sharing site on the Web with billions of photos in total. We demonstrate that our simple method of enhancing face recognition with social network context substantially increases recognition performance beyond that of a baseline face recognition system.
@cite_20: Online photo services such as Flickr and Zooomr allow users to share their photos with family, friends, and the online community at large. An important facet of these services is that users manually annotate their photos using so called tags, which describe the contents of the photo or provide additional contextual and semantical information. In this paper we investigate how we can assist users in the tagging phase. The contribution of our research is twofold. We analyse a representative snapshot of Flickr and present the results by means of a tag characterisation focussing on how users tags photos and what information is contained in the tagging. Based on this analysis, we present and evaluate tag recommendation strategies to support the user in the photo annotation task by recommending a set of tags that can be added to the photo. The results of the empirical evaluation show that we can effectively recommend relevant tags for a variety of photos with different levels of exhaustiveness of original tagging.
@cite_11: Large-scale image retrieval benchmarks invariably consist of images from the Web. Many of these benchmarks are derived from online photo sharing networks, like Flickr, which in addition to hosting images also provide a highly interactive social community. Such communities generate rich metadata that can naturally be harnessed for image classification and retrieval. Here we study four popular benchmark datasets, extending them with social-network metadata, such as the groups to which each image belongs, the comment thread associated with the image, who uploaded it, their location, and their network of friends. Since these types of data are inherently relational, we propose a model that explicitly accounts for the interdependencies between images sharing common properties. We model the task as a binary labeling problem on a network, and use structured learning techniques to learn model parameters. We find that social-network metadata are useful in a variety of classification tasks, in many cases outperforming methods based on image content.
Social media context for labeling . A set of tags associated with each image is commonly used in multimodal classification settings. Guillamumin @cite_23 explored the relationship between tags and manual annotations to recover annotations using a combination of tags and image content. Lindstaedt @cite_27 and Sigurbjornsson @cite_20 studied the problem of recommending tags that were obtained from similar images and similar users. Sawant @cite_7 and Stone @cite_10 investigates friendship information between users for tag recommendation in social networks. EXIF and GPS are two commonly used sources of metadata that come directly from the camera @cite_6 @cite_14 @cite_28 @cite_13 . Such metadata can be used to help determine who captured the photo and where, and also provide informative signals for image labeling tasks. Our method differs from all these and also @cite_11 in that we use a much larger range of social media information, including free-form text as well as links, with deep learning based pixel descriptors incorporated into our novel deep learning fully connected CRF framework.