Dynamic Deep Multi-modal Fusion for Image Privacy Prediction
Ashwini Tonge Cornelia Caragea















Department of Computer Science 

  Kansas State University



atonge@ksu.edu


Department of Computer Science 

  University of Illinois at Chicago



cornelia@uic.edu




op-tical net-works semi-conduc-tor




















With millions of images that are shared online on social networking sites, effective methods for image privacy prediction are highly needed. In this paper, we propose an approach for fusing object, scene context, and image tags modalities derived from convolutional neural networks for accurately predicting the privacy of images shared online. Specifically, our approach identifies the set of most competent modalities on the fly, according to each new target image whose privacy has to be predicted. 


The approach considers three stages to predict the privacy of a target image, wherein we first identify the neighborhood images that are visually similar and/or have similar sensitive content as the target image.  Then, we estimate the competence of the modalities based on the neighborhood images. Finally, we fuse the decisions of the most competent modalities and predict the privacy label for the target image. 
Experimental results show that our approach predicts the sensitive (or private) content more accurately than the models trained on individual modalities (object, scene, and tags) and prior privacy prediction works. Also, our approach outperforms strong baselines, that train meta-classifiers to obtain an optimal combination of modalities.



<ccs2012>
<concept>
<concept_id>10002978.10003022.10003027</concept_id>
<concept_desc>Security and privacy Social network security and privacy</concept_desc>
<concept_significance>500</concept_significance>
</concept>
</ccs2012>


[500]Security and privacy Social network security and privacy



Dynamic Deep Multi-modal Fusion for Image Privacy Prediction
    Cornelia Caragea
    
============================================================





§ INTRODUCTION

Technology today offers innovative ways to share photos with people all around the world,  making online photo sharing an incredibly popular activity for Internet users. These users document daily details about their whereabouts through images and also post pictures of their significant milestones and private events, e.g., family photos and cocktail parties <cit.>. Furthermore, smartphones and other mobile devices facilitate the exchange of information in content sharing sites virtually at any time, in any place. 

Although current social networking sites allow users to change their privacy preferences, this is often a cumbersome task for the vast majority of users on the Web, who face difficulties in assigning and managing privacy settings <cit.>. Even though users change their privacy settings to comply with their personal privacy preference, they often misjudge the private information in images, which fails to enforce their own privacy preferences <cit.>.

Thus, new privacy concerns 
<cit.>

are on the rise and mostly emerge due to users' lack of understanding that semantically rich images may reveal sensitive information <cit.>. 
For example, a seemingly harmless photo of a birthday party may unintentionally reveal sensitive information about a person's location, personal habits, and friends. 

Along these lines, Gross and Acquisti <cit.> analyzed more than 4,000 Carnegie Mellon University students' Facebook profiles and outlined potential threats to privacy. The authors found that users often provide personal information generously on social networking sites, but they rarely change default privacy settings, which could jeopardize their privacy. Employers often perform background checks for their future employees using social networking sites and about 8% of companies have already fired employees due to their inappropriate social media content <cit.>. 
A study carried out by the Pew Research center reported that 11% of the users of social networking sites regret the content they posted <cit.>. 

Motivated by the fact that increasingly online users' privacy is routinely compromised by using social and content sharing applications <cit.>, recently, researchers started to explore machine learning and deep learning models to automatically identify private or sensitive content in images 

<cit.>. Starting from the premise that the objects and scene contexts present in images impact images' privacy, many of these studies used objects, scenes, and user tags, or their combination (i.e., feature-level or decision-level fusion) 
 to infer adequate privacy classification for online images. 



























However, we conjecture that simply combining objects, scenes and user tags modalities using feature-level fusion (i.e., concatenation of all object, scene and user tag features) or decision-level fusion (i.e., aggregation of decisions from classifiers trained on objects, scenes and tags) may not always help to identify the sensitive content of images. Figure <ref> illustrates this phenomenon through several images. For example, let us consider image (a) in the figure. Both feature-level and decision-level fusion models yield very low private class probabilities (Feature-level fusion: 0.21 and decision-level fusion: 0.33). Interestingly, a model based on the scene context (bedroom) of the image outputs a high probability of 0.62, showing that the scene based model is competent to capture the sensitive content of the image on its own. Similarly, for the image (b) (self-portrait) in Figure <ref>, where scene context is seldom in the visual content, the objects in the image (the “persons,” “cocktail dress”)  are more useful (0.78) to predict appropriate image's privacy. Moreover, for images such as “personal documents” (image (c)), user-annotated tags provide broader context (such as type and purpose of the document), capturing the sensitive content (0.86), that objects and scene obtained through images' content failed to capture. On the other hand, in some cases, we can find more than one competent model for an image (e.g., for image (d)). 
To this end, we propose a novel approach that dynamically fuses multi-modal information of online images, derived through  Convolutional Neural Networks (CNNs), to adequately identify the sensitive image content. In summary, we make the following contributions:




























    
  * Our significant contribution is to estimate the competence of object, scene and tag modalities for privacy prediction and dynamically identify the most competent modalities for a target image whose privacy has to be predicted.
    
    
    
  * We derive “competence” features from the neighborhood regions of a target image and learn classifiers on them to identify whether a modality is competent to accurately predict the privacy of the target image. To derive these features, we consider privacy and visual neighborhoods of the target image to bring both sensitive and visually similar image content closer.    
    
  * We provide an in-depth analysis of our algorithm in an ablation setting, where we record the performance of the proposed approach by removing its various components. The analysis outline the crucial components of our approach.
    
    
    
    
  * Our results show that we identify images' sensitive content more accurately than single modality models (object, scene, and tag), multi-modality baselines and prior approaches of privacy prediction, depicting that the approach optimally combines the multi-modality for privacy prediction. 






§ RELATED WORK

We briefly review the related work as follows.


Ensemble models and Multi-Modality.
Several works used ensemble classifiers (or bagging) to improve image classifications <cit.>. Bagging is an ensemble technique that builds a set of diverse classifiers, each trained on a random sample of the training data to improve the final (aggregated) classifier confidence <cit.>. 




Ensemble classifiers are also used in the multi-modal settings <cit.>, where 

different modalities have been coupled, such as images and text being used for image retrieval <cit.>, image classification <cit.>, whereas audio and visual signals are combined for speech classification <cit.>. Zahavy et al. DBLP:conf/aaai/ZahavyKMM18 discuss challenges with respect to combining features or classifiers corresponding to different modalities. Essentially, classifiers can vary in their discriminative ability, and  therefore an optimal unification method to combine different classifiers needs to be developed. Besides, merging the Convolutional Neural Network (CNN) architectures corresponding to various data modalities, that can vary in depth, width, and the optimization algorithm can become very complex. Additionally, a multi-modal architecture can have more local minima and thus, may not give desirable results. However, there is a potential to improve the performance through multi-modal information fusion, which intrigued various researchers <cit.>.  For example, Frome et al. <cit.> merged an image network <cit.> with a Skip-gram Language Model to improve classification on ImageNet. 

Zahavy et al. DBLP:conf/aaai/ZahavyKMM18 proposed a policy network for multi-modal product classification using text and visual content, which learns to choose between the input signals. Feichtenhofer et al. Feichtenhofer16 fused CNNs both spatially and temporally for activity recognition in videos to take advantage of the spatio–temporal information present in videos.  
Wang et al. Object-Scene designed an architecture to combine object networks and scene networks, which extract useful information as objects and scene context for event understanding. 



Co-training approaches <cit.> used multiple views (can be considered as modalities) to combine different classifiers. However, co-training methods are semi-supervised and assume that all views are competent to classify all kinds of test instances.
All these works did not focus on dynamically identifying the most competent CNN (or model) based on the target image's subject (e.g. documents). Since online images are comprised of various image subjects, this is particularly critical for predicting correct images' privacy. 
Our approach adopts dynamic ensemble techniques (DES) <cit.>, wherein a pool of classifiers are trained on a single feature set (single modality) using the bagging technique <cit.>, and the competence of these weak classifiers are determined dynamically. 
However, unlike DES techniques, we capture different outlooks of the same information (in our case, online images), obtained from distinct sources (object, scene, and tag), and apply them to perform multi-modal fusion in the  privacy prediction setting.
We derive objects, scene and tag features through CNNs and generate robust classifiers instead of weak classifiers. The ability to perform an optimal fusion of these features is essential to identify the sensitive content from images successfully. In contrast to single modality-based fusion approaches, one of the primary obstacles to multimodal fusion is the development and specification of a methodology to integrate cognitive and effective information from disparate sources <cit.>.  




Online Image Privacy.


 Several works are carried out to study users' privacy concerns in social networks, privacy decisions about sharing resources, and the risk associated with them <cit.>. 
<cit.> examined privacy decisions and considerations in mobile and online photo sharing. They explored critical aspects of privacy such as users' consideration for privacy decisions, content and context based patterns of privacy decisions, how different users adjust their privacy decisions and user behavior towards personal information disclosure. The authors concluded that applications, which could support and influence user's privacy decision-making process should be developed. 
<cit.>  reinforced the role of privacy-relevant image concepts. For instance, they determined that people are more reluctant to share photos capturing social relationships than photos taken for functional purposes; certain settings such as work, bars, concerts cause users to share less.  <cit.> mentioned that users want to regain control over their shared content, but meanwhile, they feel that configuring proper privacy settings for each image is a burden.




Buschek et al. Buschek:2015 presented an approach to assign privacy to shared images using metadata (location, time, shot details) and visual features (faces, colors, edges). Zerr et al. Zerr:2012  developed the PicAlert dataset, containing Flickr photos, to help detect private images and also proposed a privacy-aware image classification approach to learn classifiers on these Flickr photos. Authors considered image tags and visual features such as color histograms, faces, edge-direction coherence, and SIFT for the privacy classification task. 


Squicciarini et al. Squicciarini2014,Squicciarini:2017:TAO:3062397.2983644 
 found that SIFT and image tags work best for predicting sensitivity of user's images. 

Given the recent success of CNNs, Tran et al. Tran:2016:PFD:3015812.3016006, and Tonge and Caragea DBLP:conf/aaai/TongeC16,tongemsm18 showed promising privacy predictions compared with visual features such as SIFT and GIST. 
Yu et al. DBLP:journals/tifs/YuZKLF17 adopted CNNs to achieve semantic image segmentation and also learned object-privacy relatedness to identify privacy-sensitive objects. Spyromitros-Xioufis et al. Spyromitros-Xioufis:2016:PPI:2911996.2912018 used features extracted from CNNs to provide personalized image privacy classification. Zhong et  al. zhong2017ijcai proposed a Group-Based Personalized Model for image privacy classification in online social media sites. 
Despite that an individual's sharing behavior is unique, personalized models generally require large amounts of user data to learn reliable models, and are time and space consuming to train and to store models for each user, while taking into account possible deviations due to sudden changes of users' sharing activities and privacy preferences.  
 <cit.> defined a set of privacy attributes and correlated to image content.  The authors first predicted these privacy attributes from the image content and then used them in combination with users preferences to estimate personalized privacy risk.  The authors used official online social network rules to define the set of attributes, instead of collecting real user's opinions about sensitive content and hence, the definition of sensitive content may not meet the user's actual need <cit.>.  Generic models can be used as a primary level to identify the sensitive content based on which attributes depicting actual user's privacy needs can be derived.  Additionally, for privacy attribute predictor, the authors fine-tuned a CNN pre-trained on object dataset. Using multi-modal information associated with online images more accurate privacy attributes can be predicted.

To this end, we propose an approach to fuse features corresponding to multi-modal information of online images that have the potential to improve the performance of generic privacy prediction.




§ RELATED WORK

We briefly review the related work as follows.


Ensemble models and Multi-Modality.
Several works used ensemble classifiers (or bagging) to improve image classifications <cit.>. 

Bagging is an ensemble technique that builds a set of diverse classifiers, each trained on a random sample of the training data to improve the final (aggregated) classifier confidence <cit.>. Dynamic ensembles that extend bagging have also been proposed <cit.> wherein a pool of classifiers are trained on a single feature set (single modality) using the bagging technique <cit.>, and the competence of the base classifiers is determined dynamically. 






Ensemble classifiers are also used in the multi-modal setting <cit.>, where 

different modalities have been coupled, e.g., images and text for image retrieval <cit.> and image classification <cit.>, and audio and visual signals for speech classification <cit.>. Zahavy et al. DBLP:conf/aaai/ZahavyKMM18 
highlighted that classifiers trained on different modalities can vary in their discriminative ability and urged the development of optimal unification methods to combine different classifiers. 
Besides, merging the Convolutional Neural Network (CNN) architectures corresponding to various modalities, that can vary in depth, width, and the optimization algorithm can become very complex. 
However, there is a potential to improve the performance through multi-modal information fusion, which intrigued various researchers <cit.>.  For example, Frome et al. <cit.> merged an image network <cit.> with a Skip-gram Language Model to improve classification on ImageNet. 

Zahavy et al. DBLP:conf/aaai/ZahavyKMM18 proposed a policy network for multi-modal product classification in e-commerce using text and visual content, which learns to choose between the input signals. Feichtenhofer et al. Feichtenhofer16 fused CNNs both spatially and temporally for activity recognition in videos to take advantage of the spatio-temporal information present in videos.  
Wang et al. Object-Scene designed an architecture to combine object networks and scene networks, which extract useful information such as objects and scene contexts for event understanding. 



Co-training approaches <cit.> use multiple views (or modalities) to “guide” different classifiers in the learning process.  However, co-training methods are semi-supervised and assume that all views are “sufficient” for learning. 




In contrast with the above approaches, we aim to capture different aspects of images, obtained from multiple modalities (object, scene, and tags), with each modality having a different competence power, 
and perform dynamic multi-modal fusion for image privacy prediction.






Online Image Privacy.


 Several works are carried out to study users' privacy concerns in social networks, privacy decisions about sharing resources, and the risk associated with them <cit.>. 
<cit.> examined privacy decisions and considerations in mobile and online photo sharing. They explored critical aspects of privacy such as users' consideration for privacy decisions, content and context based patterns of privacy decisions, how different users adjust their privacy decisions and user behavior towards personal information disclosure. The authors concluded that applications, which could support and influence user's privacy decision-making process should be developed. 
<cit.>  reinforced the role of privacy-relevant image concepts. For instance, they determined that people are more reluctant to share photos capturing social relationships than photos taken for functional purposes; certain settings such as work, bars, concerts cause users to share less.  <cit.> mentioned that users want to regain control over their shared content, but meanwhile, they feel that configuring proper privacy settings for each image is a burden.




Buschek et al. Buschek:2015 presented an approach to assign privacy to shared images using metadata (location, time, shot details) and visual features (faces, colors, edges). Zerr et al. Zerr:2012  developed the PicAlert dataset, containing Flickr photos, to help detect private images and also proposed a privacy-aware image classification approach to learn classifiers on these Flickr photos. Authors considered image tags and visual features such as color histograms, faces, edge-direction coherence, and SIFT for the privacy classification task. 


Squicciarini et al. Squicciarini2014,Squicciarini:2017:TAO:3062397.2983644 
 found that SIFT and image tags work best for predicting sensitivity of user's images. 

Given the recent success of CNNs, Tran et al. Tran:2016:PFD:3015812.3016006, and Tonge and Caragea DBLP:conf/aaai/TongeC16,tongemsm18 showed promising privacy predictions compared with visual features such as SIFT and GIST. 
Yu et al. DBLP:journals/tifs/YuZKLF17 adopted CNNs to achieve semantic image segmentation and also learned object-privacy relatedness to identify privacy-sensitive objects. 

Spyromitros-Xioufis et al. Spyromitros-Xioufis:2016:PPI:2911996.2912018 used features extracted from CNNs to provide personalized image privacy classification, whereas Zhong et  al. zhong2017ijcai proposed a Group-Based Personalized Model for image privacy classification in online social media sites. 
Despite that an individual's sharing behavior is unique, Zhong et  al. zhong2017ijcai argued that personalized models generally require large amounts of user data to learn reliable models, and are time and space consuming to train and to store models for each user, while taking into account possible deviations due to sudden changes of users' sharing activities and privacy preferences.  
<cit.> defined a set of privacy attributes, which were 
first predicted 
from the image content and then used these attributes in combination with users preferences to estimate personalized privacy risk.  The authors used official online social network rules to define the set of attributes, instead of collecting real user's opinions about sensitive content and hence, the definition of sensitive content may not meet a user's actual needs <cit.>.  
Additionally, for privacy attribute prediction, the authors fine-tuned a CNN pre-trained on object dataset. In contrast, we proposed a dynamic multi-modal fusion approach to determine which aspects of images (objects, scenes or tags) are more competent to predict images' privacy. 










§ MULTI-MODALITY

The sensitive content of an image can be perceived by the presence of one or more objects, the scenes described by the visual content and the description associated with it in the form of tags <cit.>. We derive features (object, scene, tags) corresponding to the multi-modal information of online images as follows. 










Object (F^o):
Detecting objects from images 
is clearly fundamental to assessing whether an image  is of private nature. For example, a single element such as a firearm, political signs, may be a strong indicator of a private image. Hence, we explore the image descriptions extracted from VGG-16 <cit.>, a CNN pre-trained on the ImageNet dataset <cit.> that has 1.2M+ images labeled with 1,000 object categories. 
The VGG-16 network implements a 16 layer deep network; a stack of convolutional layers with a very small receptive field: 3 × 3 followed by fully-connected layers. The architecture contains 13 convolutional layers and 3 fully-connected layers. 
The input to the network is a fixed-size 224 × 224 RGB image. The activation of the fully connected layers capture the complete object contained in the region of interest. Hence,  we use the activation of the last fully-connected layer of VGG-16, i.e., fc_8 as a feature vector. The dimension of object features F^o is 1000.

Scene (F^s): 
As consistently shown in various user-centered studies <cit.>, the context of an image is a potentially strong indicator of what type of message or event users are trying to share online. These scenes, e.g., some nudity, home, fashion events, concerts are also often linked with certain privacy preferences. Similar to object features, we obtain the scene descriptors derived from the last fully-connected layer of the pre-trained VGG-16 <cit.> on the Places2 dataset which contains 365 scene classes within 2.5 million images <cit.>. The dimension of scene features F^s is 365.






Image Tags (F^t):



For image tags, we employ 


the CNN architecture of <cit.>. 
The network contains one convolution layer on top of word vectors obtained from an unsupervised neural language model.  The first layer embeds words into the word vectors pre-trained by <cit.> on 100 billion words of Google News, and are publicly available. The next layer performs convolutions  on the embedded word vectors using multiple filter sizes of 3, 4 and 5, where we use 128 filters from each size and produce a tag feature representation. A max-pooling operation over the feature map is applied to capture the most important feature of length 256 for each feature map.  

To derive these features, we consider two types of tags: (1) user tags, and (2) deep tags. 

Because not all images on social networking sites have user tags 
or the set of user tags is very sparse  <cit.>, 
we predict the top d object categories (or deep tags) from the probability distribution extracted from CNN. 

Object + Scene + Tag (F^ost):
We use the combination of the object, scene, and tag features to identify the neighborhood of a target image. 
We explore various ways given in <cit.> to combine the features. For example, we use fc_7 layer of VGG to extract features of equal length of 4096 from both object-net and scene-net and consider the max-pooling of these vectors to combine these features. Note that, in this work, we only describe the combination of features that worked best for the approach. We obtain high-level object “F^o” and scene “F^s” features from fc_8 layer of object-net and scene-net respectively and concatenate them with the tag features as follows:
F^ost = f^cat(F^o, F^s, F^t). F^ost = F^o(i), 1 ≤ i ≤ 1000,  F^ost(i + 1000) = F^s(i), 1≤ i ≤ 365, F^ost(i + 1365) = F^t(i), 1≤ i ≤ 256.



§ PROPOSED APPROACH










We seek to classify a given image into one of the two classes: private or public, based on users' general privacy preferences. 
To achieve this, we depart from previous works 
that use the same model 
on all image types (e.g., portraits, bedrooms, and legal documents), 

and propose an approach called “Dynamic Multi-Modal Fusion for Privacy Prediction” (or DMFP), that effectively fuses 
multi-modalities (object, scene, and tags) 
and dynamically captures different aspects or particularities from image. Specifically, the proposed approach aims to estimate the competence of models 
trained on these individual modalities for each target image (whose privacy has to be predicted) 
and dynamically identifies the subset of the most “competent” models for that image. 
The rationale for the proposed method is that for a particular type of sensitive content, some modalities may be important, whereas others may be irrelevant and may simply introduce noise. 
Instead, a smaller subset of modalities may be significant 
in capturing a particular type of sensitive content (e.g., objects for portraits, scenes for interior home or bedroom, and tags for legal documents, as shown in Figure <ref>).  


The proposed approach considers three stages to predict the privacy of a target image, wherein we first identify the neighborhood images that are visually similar and/or have similar sensitive content as the target image (Section <ref>).  Then, using a set of base classifiers, each trained on an individual modality, we estimate the competence of the modalities by determining which modalities classify the neighborhood images correctly (Section <ref>). The goal here is to select the most competent modalities for a particular type of images (e.g., scene for home images). Finally, we fuse the decisions of the most competent base classifiers (corresponding to the most competent modalities) and predict the privacy label for the target image (Section <ref>).

Our approach considers two datasets, denoted as 𝒟^T and 𝒟^E, that contain images labeled as private or public. We use the dataset 𝒟^T to train a base classifier for each modality to predict whether an image is public or private. Particularly, we train 3 base classifiers ℬ = {B^o, B^s, B^t} on the corresponding modality feature sets from ℱ. 
Note that we use the combination of feature sets F^ost only for visual content similarity and do not train a base classifier on it.
The competences of these base classifiers are estimated on the 𝒟^E dataset. We explain the stages of the proposed approach as follows. The notation used is shown in Table <ref>.





























  §.§.§ Mathematical Notations






  * 𝒟^T = {(I_1, Y_1),⋯,(I_m, Y_m)} of labeled images.

  * 𝒟^E = {(X_1, Y_1),⋯,(X_n, Y_n)} of labeled images.


  * T a target image with an unknown privacy label.

  * ℱ = {F^o, F^s, F^t, F^ost} a collection of modality feature sets corresponding to object, scene, tag, and object+scene+tag.


  * ℬ = {B^o, B^s, B^t} a set of base classifiers trained on corresponding modality feature sets from ℱ (e.g., B^o is trained on F^o). 



  * T = ⋃_B_i ∈ℬ{P(Y_T = private | T,B_i), P(Y_T = public | T,B_i)} is the privacy profile of image T is given as a vector of posterior privacy probabilities obtained by base classifiers ℬ. For image (a) in Figure <ref>, T = {0.62, 0.38, 0.5, 0.5, 0.29, 0.71, 0.21, 0.79}.


  * N^T_V the visual similarity based neighborhood of image T estimated using visual content features F^ost.

  * k_v the number of nearest neighbors obtained for visual similarity based neighborhood N^T_V, 1 ≤ k_v < n. 

  * N^T_P the privacy profile based neighborhood of target T.


  * k_p the number of most similar images obtained for privacy profiles based neighborhood N^T_P, 1 ≤ k_p < n. 




  * C = {C^o, C^s, C^t} a collection of “competence” classifiers corresponding to the base classifiers from ℬ (e.g., C^o for B^o).


  * Φ = {ϕ^o, ϕ^s, ϕ^t} a set of “competence” feature vectors.










 §.§ Identification of Neighborhoods

The competence of a base classifier is estimated based on a local region where the target image is located. Thus, given a target image T, we first estimate two neighborhoods for T: (1) visual similarity based (N^T_V) and (2) privacy profile based (N^T_P) neighborhoods.








The neighborhood N^T_V of target image T consists of the k_v most similar images from 𝒟^E using visual content similarity. Specifically, using the F^ost features obtained by concatenating object, scene, and tag features (as explained in Section <ref>), we determine the k_v most visually similar images to T by applying the K-Nearest Neighbors algorithm on the 𝒟^E dataset.









The neighborhood N^T_P of target image T consists of k_p most similar images to T by calculating the cosine similarity between the privacy profile of T and images from the dataset 𝒟^E. We define privacy profile (denoted by T) of image T as a vector of posterior privacy probabilities obtained by the base classifiers ℬ i.e., T = ⋃_B_i ∈ℬ{P(Y_T = private | T,B_i), P(Y_T = public | T,B_i)}. For image (a) in Figure <ref>, T = [0.62, 0.38, 0.5, 0.5, 0.29, 0.71]. We consider the privacy profile of images because images of particular image content (bedroom images or legal documents) tend to possess similar privacy probabilities with respect to the set of base classifiers ℬ. For instance, irrespective of various kinds of bedroom images, the probabilities for a private class obtained by base classifiers ℬ, would be similar. This enables us to bring sensitive content closer irrespective of their disparate visual content. Moreover, we consider two different numbers of nearest neighbors k_v and k_p to find the neighborhoods since the competence of a base classifier is dependent on the neighborhood and estimating an appropriate number of neighbors for the respective neighborhoods reduces the noise.








 §.§ “Competence” Estimation





We now describe how we estimate the “competence” of a base classifier. For instance, for the image (a) in Figure <ref>, scene model has a higher competence than the others, and here, we capture this competence through “competence” features and “competence” classifiers. 
Specifically, we train a competence classifier for each base classifier that predicts if the base classifier is competent or not for a target image T. The features for learning the competence classifiers and the competence learning are described below. 




  §.§.§ Derivation of “Competence” Features



 We define three different sets of “competence” features wherein each set of these features captures a different criterion to estimate the level of competence of base classifiers dynamically. 
 The first competence feature ϕ_1, for image T, is derived from the neighborhood N^T_V (based on visual similarity) whereas the second competence feature ϕ_2 is obtained from the neighborhood N^T_P (based on privacy profile).
 The third competence feature ϕ_3 captures the level of confidence of base classifiers for predicting the privacy of the image (T) itself. We create a “competence” feature vector by concatenating all these competence features ϕ = {ϕ_1 ∪ϕ_2 ∪ϕ_3 }  into a vector of length |ϕ| = k_v + k_p + 1. We extract such competence vectors corresponding to each base classifier in ℬ (e.g., ϕ^o for B^o, refer to Figure <ref>). We extract these “competence” features as follows.


ϕ_1: A vector of k_v entries that captures the correctness of a base classifier in the visual neighborhood region N_V^T. 

An entry j in ϕ_1 is 1 if a base classifier B_i ∈ℬ accurately predicts privacy of image X_j ∈ N_V^T, 
and is 0 otherwise, where j = 1,⋯,k_v. For the target image in Figure <ref>, ϕ_1 = {1, 1, 0, 1, 0, 1, 1}, obtained by B^o.


 
 

  
  
ϕ_2: A vector of k_p entries that captures the correctness of a base classifier in the privacy profile neighborhood region N_P^T. 

An entry j in ϕ_2 is 1 if a base classifier B_i ∈ℬ accurately predicts privacy of image X_j ∈ N_P^T, 
and is 0 otherwise, where j = 1,⋯,k_p. For the target image in Figure <ref>, ϕ_2 = {1, 1, 1, 1, 1}, obtained using B^o.



ϕ_3: We capture a degree of confidence of base classifiers for target image T. Particularly, we consider the maximum posterior probability obtained for target image T using base classifier B_i i.e. Max(P(Y_T = Private | T,B_i), P(Y_T = Public | T,B_i)), where B_i ∈ℬ. For the target image in Figure <ref>, ϕ_3 = 0.67, obtained using B^o.





  
  f_1 : A vector of k_v entries captures the the correctness of a base classifier in the neighborhood region N^T_V (obtained using visual content similarity). If a base classifier B_i ∈ℬ accurately predicts privacy of instance X_k_v from the neighborhood N^T_V, then the vector contains 1 at (k_v - 1)^th position, 0 otherwise. For the target image in Figure <ref>, f_1 = {1, 1, 0, 1, 0, 1, 1}, obtained by B^o.
 

 
 

  
  
  
  f_2 : We create a vector of k_p  to capture the accuracy in the neighborhood N^T_P, obtained using privacy profiles. If base classifier B_i predicts correct privacy label for X_k_p, then we set 1 in the (k_p - 1)^th position, 0 otherwise. For the target image in Figure <ref>, f_2 = {1, 1, 1, 1, 1}, obtained using B^o.


  
  f_3 : We capture a degree of confidence of models for target image T. Particularly, we consider the maximum posterior probability obtained for target image T using base classifier B_i i.e. Max(P(Y_T = Private | T,B_i), P(Y_T = Public | T,B_i)), where B_i ∈ℬ. For the target image in Figure <ref>, f_3 = 0.67, obtained using B^o.








  
    












  §.§.§ “Competence” Learning



We learn the “competence” of a base classifier by training a binary “competence” classifier on the dataset 𝒟^E in a Training Phase. A competence classifier predicts whether a base classifier is competent or not for a target image. Algorithm <ref> describes the “competence” learning process in details. Mainly, we consider images from 𝒟^E  as target images (for the training purpose only) and identify both the neighborhoods (N_V,  N_P) from the dataset 𝒟^E itself (Alg. <ref>, lines 6–8). Then, we extract “competence” features for each base classifier in ℬ based on the images from these neighborhoods (Alg. <ref>, line 10). To reduce noise, we extract “competence” features by considering only the images belonging to both the neighborhoods. On these “competence” features, we train a collection of “competence” classifiers 𝒞 corresponding to each base classifier in ℬ (Alg. <ref>, lines 19–22). Precisely, we train 3 competence classifiers 𝒞 = {C^o, C^s, C^t}. To train “competence” classifier C_i ∈𝒞, we consider label L_i = 1 if base classifier B_i ∈ℬ predicts the correct privacy of a target image (here, X_j ∈𝒟^E), otherwise 0  (Alg. <ref>, lines 11–16). 








  §.§.§ Meta-features:

We consider five different sets of meta-features proposed in dynamic ensemble framework <cit.>. Each set of these features captures a different criterion to dynamically estimate the level of competence of the base classifier (or features). The first three meta-features, f_1, f_2, and f_3, for a sample x_j, are derived from the region of competence θ_j. The meta-feature f_4 is obtained from the set of similar output privacy profiles θ̅_j whereas the meta-feature f_5 correlates to the level of confidence of classifier c_i for the privacy prediction of input image x_j, where c_i ∈𝒞.


  
  f_1 – A vector of k entries captures the local accuracy of the region of competence. The vector contains 1 at (k - 1)^th position, if a classifier c_i accurately predict the privacy of an instance x_k, belonging to the region of competence θ_j, otherwise, it is 0. 
  
  
  f_2 – Similar to f_1, a vector of k elements is created to capture the extent of agreement between the classifiers in the region of competence. Instead of considering the boolean outcome of the classifier c_i, the posterior probability of an image x_k P(ω_p | x_k,c_i) is computed using classifier c_i  and is inserted at (k - 1)^th  position in the vector.
  
  
  f_3 – The overall accuracy of classifier c_i ∈𝒞 obtained on the region of competence θ_j, i.e., mean(f_1).
  
  
  f_4 – A vector of k_op features is created to capture the accuracy in the privacy space. For each member x̅_k_op in the set of output profiles θ̅_j, if classifier c_i predict the correct privacy label for x_k_op, then 1 is set in the (k_op - 1) position, 0 otherwise.
  
  
  f_5 – The feature captures degree of confidence for the input sample image x_j. The feature value is given as the maximum posterior probability of x_j using classifier c_i is obtained i.e. max(P(ω_p = private | x_j,c_i), P(ω_p = public | x_j,c_i)).
  









 §.§ Dynamic Fusion of Multi-Modality

In this stage, for given target image T, we dynamically determine the subset of most competent base classifiers. We formalize the process of base classifier selection in Algorithm <ref>. The algorithm first checks the agreement on the privacy label between all the base classifiers in ℬ (Alg. <ref>, line 5). If not all the base classifiers agree, then we estimate the competence of all the base classifiers and identify the subset of most competent base classifiers for the target image as follows. Given target image T, Algorithm <ref> first identifies both the neighborhoods (N^T_V,  N^T_P) using the visual features F^ost and privacy profile from 𝒟^E dataset (Alg. <ref>, lines 7–9). Using these neighborhoods, we extract “competence” feature vectors (explained in Section <ref>) and provide them to the respective “competence” classifiers in 𝒞 (learned in the Training Phase) to predict competence score of base classifier B_i. If the competence score is greater than 0.5, then base classifier B_i is identified as competent to predict the privacy of target image T (Alg. <ref>, lines 10–17). Finally, we weight votes of the privacy labels predicted by the subset of most competent base classifiers by their respective “competence” score and take a majority vote to obtain the final privacy label for target image T (Alg. <ref>, line 18). A “competence” score CS_i is given as a probability of base classifier B_i being competent. We consider the majority vote of the most competent base classifiers because certain images (e.g., vacation) might require more than one base classifiers (object and scene) to predict the appropriate privacy. If both the privacy classes (private and public) get the same number of votes, then the class of a highest posterior probability is selected.














  §.§.§ Illustration of the Proposed Approach 


Figure <ref> shows the illustration of the proposed approach through an anecdotal example. We consider a target image T whose privacy has to be predicted. For T, we first identify two neighborhoods: (1) visual content (N_V^T), 2. privacy profile (N_P^T). For N_V^T, we use visual content features F^ost to compute the similarity between target image T and the images from the dataset 𝒟^E. The top k_v = 7 similar images for N_V^T are shown in the figure (left blue rectangle).  Likewise, for N_P^T, we compute the similarity between privacy profile of the target image T and privacy profiles of images in 𝒟^E. We show the top k_p = 5 similar images for N_P^T in the right blue rectangle of Figure <ref>. 
From these neighborhoods, we derive a “competence” feature vector ϕ for each base classifier in ℬ (e.g., ϕ^o for B^o). We show these “competence” features in the figure as a matrix of feature values. We input these features to the respective “competence” classifiers from 𝒞 (e.g., ϕ^o to C^o), that predict whether a base classifier B_i ∈ B is competent to predict correct privacy label of the target image (T). The “competence” classifiers (C^o, C^s, C^t) are shown as blue rectangles on the right side of Figure <ref>. The base classifiers B^o and B^t are predicted as competent and hence are selected to obtain the final privacy label for the target image. The competent base classifiers are shown in green rectangles on the right side of Figure <ref>. Once we select the competent base classifiers, we take a weighted majority vote on the privacy labels, predicted by these base classifiers. For example, in this case, the competent base classifiers B^o and B^t predict the privacy of T as “private,” and hence, the final privacy label of T is selected as “private.” It is interesting to note that the target image (T) contains “outdoor” scene context that is not useful to predict the correct privacy label and hence, the scene model  B^s is not selected by the proposed approach for the target image.




§ DATASET

We evaluate our approach on a subset of 32,000 Flickr images sampled from the PicAlert dataset, made available by Zerr et al. Zerr:2012. PicAlert consists of Flickr images on various subjects, which are manually labeled as public or private by external viewers. 


The guideline to select the label is given as: private images belong to the private sphere (like self-portraits, family, friends, someone's home) or contain information that one would not share with everyone else (such as private documents). The remaining images are labeled as public. 

The dataset of 32,000 images is split in 𝒟^T, 𝒟^E and Test sets of 15,000, 10,000 and 7,000 images, respectively. Each experiment is repeated 5 times with a different split of the three subsets (obtained using 5 different random seeds) and the results are averaged across the five runs. 

The public and private images are in the ratio of 3:1 in all subsets. 



§ EXPERIMENTS AND RESULTS



 


We evaluate the privacy prediction performance obtained using the proposed approach DMFP, where we train a set of base classifiers ℬ on images in the dataset 𝒟^T, and dynamically estimate the “competence” of these base classifiers for target images in Test by identifying neighborhoods (N_V, N_P) using images in 𝒟^E. We first consider various values of neighborhood parameters k_v and k_p and show their impact on the performance of the proposed approach. Then, we compare the performance of the proposed approach with the performance obtained using three types of mechanisms: 
(1) components of the proposed approach, that are used to fuse the multi–modal characteristics of online images, (2) the state-of-the-art approaches for privacy prediction, and (3) strong baselines that select models based on their competence (e.g., <cit.>) and that attempt to yield the optimal combination of base classifiers (for instance, using stacked ensemble classifiers). 




Evaluation Setting. 



We train base classifiers (ℬ) using the Calibrated linear Support Vector Machine (SVM) implemented in Scikit-learn library 
to predict more accurate probabilities. We use 3-fold cross-validation on the dataset 𝒟^T to fit the linear SVM on the 2-folds, and the remaining fold is used for calibration. The probabilities for each of the folds are then averaged for prediction. We train “competence” classifiers (𝒞) on the dataset 𝒟^E using logistic regression to predict “competence” scores between 0-1 for base classifiers. If base classifier B_i gets a “competence” score greater than 0.5 then the base classifier is considered competent. 
To derive features from CNN, we use pre-trained models presented by the VGG-16 team in the ILSVRC-2014 competition <cit.> and the  CAFFE framework <cit.>. For deep tags, we consider top d=10 object labels as d=10 worked best.





  §.§.§ Exploratory Analysis.
 We provide exploratory analysis in Table <ref> to highlight the potential of merging object, scene and tag 

r0.29




Test     Pr(%)     Pu(%)     O(%) 

Object is correct     49     95.7     84.8

Scene is correct     51     94.7     84.4

Tag is correct     57     91.1     83 






All are correct     30     87.3     73.9 
 

All are wrong     27     1.5     7.4 


Atleast one       73      98.5      92.6 

modality is correct          





Exploratory analysis. 





modality for privacy prediction.  We predict privacy for images in the Test set using base classifiers in ℬ and obtain  “private” (Pr), “public” (Pu) and “overall” (O) accuracy for: (a) a modality is correct (e.g., object), (b) all modalities are correct, (c) all modalities are wrong, and (d) at least one modality is correct. Table <ref> shows that out of the three base classifiers (top 3 rows), the tag model yields the best accuracy for the private class (57%). Interestingly,  the results for “at least one modality is correct” (73%) show that using multi-modality, there is a huge potential (16%) to improve the performance of the private class. This large gap is a promising result for developing multi-modality approaches for privacy prediction. Next, we evaluate DMFP that achieved the best boost in the performance for the private class using these modalities.













 §.§ Impact of parameters k_v and k_p on DMFP








In this experiment, we show the impact of neighborhood parameters, i.e., k_v and k_p on the privacy prediction performance obtained by the proposed approach DMFP. For a target image, k_v and k_p are used to identify the neighborhoods using visual content similarity N_V and privacy profiles N_P, respectively (Alg. <ref> lines 5–7). We compare the performance in several settings: (1) we vary both the parameters k_v and k_p, denoted as “k_v,k_p”, (2) we keep k_v = 100 constant and vary only k_p to see the impact of the number of most similar images obtained by privacy profile on the performance, denoted as “k_v-const”, (3) similarly, we vary k_v parameter and keep k_p = 100 constant to understand the effect of the k_v-nearest neighbors obtained using visual content similarity on the performance, denoted as “k_p-const”. We experiment with range of values for both the parameters kv, k_p = {10, 20, ⋯, 100, 200, ⋯,1000}, in steps of 10 upto 100 and then in steps of 100. 










Figure <ref> shows the performance (F1-measure) obtained (using 3-fold cross-validation on the 𝒟^E dataset) for the private class based on several cases discussed above. We notice that for most of the cases k_v, k_p = 700 give the best results and the overall best result is obtained by the setting where we keep k_p constant i.e k_v = 700, k_p = 100. The plot shows that when we consider the neighborhood obtained by visual content similarity (N_V), the performance increases whereas adding more similar samples from the neighborhood obtained using privacy profiles (N_P) does not increase the performance significantly. This suggests that the neighborhood parameter k_v (obtained for N_V) impact the performance more than the parameter k_p (obtained for N_P). In the next experiments, we show the performance with the best values of the parameters, i.e., k_v = 700 and k_p = 100.






 §.§ Impact of Parameters k_v and k_p on DMFP

We show the impact of neighborhood parameters, i.e., k_v and k_p on the privacy prediction performance obtained by the proposed approach DMFP. k_v and k_p are used to identify visual (N_V) and privacy profile (N_P) neighborhoods of a target image, respectively (Alg. <ref> lines 7–8). We experiment with a range of values for both the parameters k_v, k_p = {10, 20, ⋯, 100, 200, ⋯, 1000}, in steps of 
10 upto 100 and then in steps of 100. We also experiment with larger k_v and k_p values, but for better visualization, we only show the values with significant results. Figure <ref> shows the F1-measure r0.26


    < g r a p h i c s >



 F1-measure obtained for various k_v and k_p values.



obtained (using 3-fold cross-validation on the 𝒟^E dataset) for the private class for various k_v and k_p values. We notice that when we increase the k_v parameter the performance increase whereas when we increase k_p parameter, the performance increases upto k_p = 200, then the performance decreases gradually. The results show that the performance is quite sensitive to changes in the privacy neighborhood (N_P) parameter k_p, but relatively insensitive to changes in the visual neighborhood (N_V) parameter k_v.

We get the best performance for k_v = 900 and k_p = 100. We use these parameter values in the next experiments.











 §.§ Evaluation of the Proposed Approach


We evaluate the proposed approach DMFP for privacy prediction in an ablation experiment setting. Specifically, we remove a particular component of the proposed approach DMFP and compare the performance of DMFP before and after the removal of that component. We consider excluding several components from DMFP: (1) the visual neighborhood N_V (DMFP-N_V), (2) the privacy profile neighborhood N_P (DMFP-N_P), (3) “competence” features (e.g., DMFP- ϕ_1), and (4) base classifier selection without “competence” learning (e.g., N_V-CL). For option (4), we consider a simpler version of the proposed algorithm, in which we do not learn a competence classifier for a base classifier; instead, we rely solely on the number of accurate predictions of the samples from a neighborhood. We evaluate it using images from three regions: (a) neighborhood N_V only (N_V-CL), (b) neighborhood N_P only (N_P-CL), and (c) both the neighborhoods N_P and N_V ({N_P + N_V}-CL).  





Table <ref> shows the class-specific (private and public) and overall performance obtained by the proposed approach (DMFP) and after removal of its various components detailed above. Primarily, we wish to identify whether the proposed approach characterizes the private class effectively as sharing private images on the Web with everyone is not desirable. 

We observe that the proposed approach achieves the highest recall of 0.627 and F1-score of 0.684 (private class), which is better than the performance obtained by eliminating the essential components (e.g., neighborhoods) of the proposed approach. We notice that if we remove either of the neighborhood N_V or N_P, the recall and F1-score drop by 5% and 4%. This suggests that both the neighborhoods (N_V, N_P) are required to identify an appropriate local region surrounding a target image. It is also interesting to note that the performance of DMFP-N_P (removal of N_P) is slightly lower than the performance of DMFP-N_V (removal of N_V), depicting that neighborhood N_P is helping more to identify the competent base classifier(s) for a target image. The N_P neighborhood brings images closer based on their privacy probabilities and hence, is useful to identify the competent base classifier(s) (this is evident in Figure <ref>). 


We also show that when we remove competence learning (CL) i.e., N_V-CL, N_P-CL, and {NV + NP}-CL, the precision improves by 4% (private class), but the recall and F1-score (private class) drops by 9% and 5% respectively, showing that competence learning is necessary to achieve the best performance.

We also remove the “competence” features one by one and record the performance of DMFP to understand which competence features are essential. Table <ref> shows that when we remove feature ϕ_1 corresponding to the neighborhood N_V, the performance drops significantly (≈ 4%). 

Likewise, when we remove ϕ_2 (feature corresponding the N_P region), we notice a similar decrease of 4% in the F1-score of private class. Note that, when we remove the “competence” features corresponding to their neighborhoods (such as ϕ_1 for N_V and ϕ_2 for N_P), we get nearly similar performance as we remove the respective neighborhoods from the proposed approach (DMFP-N_V and DMFP-N_P); implying that removing “competence” features (e.g., ϕ_1) is as good as removing the corresponding neighborhood (N_V). However, a close look at the performance suggests that the performance obtained using DMFP-ϕ_1 (recall of 0.553) is slightly worse than the performance of DMFP-N_V (recall of 0.57). Similarly, for DMFP-ϕ_2, the performance (recall) decrease from 0.572 obtained using DMFP-N_P to 0.565. The performance decrease can be explained as when we remove the neighborhood N_V or N_P, the respective “competence” features are empty, and that might be helpful for some cases (as zero-valued ϕ_2 feature of ϕ^s was helpful in Figure <ref>). Additionally, the recall of DMFP-N_V and DMFP-N_P are similar whereas the recall of DMFP-ϕ_1 (0.553) is slightly worse than the recall of DMFP-ϕ_2 (0.565). The results suggests that the neighborhood N_V is more dependent on the “competence” features as compared to the neighborhood N_P. We experimented with the probability based “competence” features (instead of boolean features), but did not yield improvements in the performance.

























 §.§ Proposed Approach vs. Base Classifiers
 
We compare privacy prediction performance obtained by the proposed approach DMFP with the set of base classifiers ℬ: 1. object (B^o), 2. scene (B^s), and 3. image tags (B^t). 








Table <ref> compares the performance obtained by the proposed approach (DMFP) and base classifiers. We achieve the highest performance as compared to the base classifiers and show a maximum improvement of ≈ 10% in the F1-score of private class. We notice that our approach based on multi-modality yields an improvement of 11% over the recall of object and scene models and an improvement of ≈ 6% over the recall of the tag model, that is the best-performing single modality model obtained for the private class from the exploratory analysis (refer to Table <ref>). Still, our approach makes some errors (See Table <ref> and <ref>, 73% vs. 62%). A close look at the errors discovered that a slight subjectivity of annotators could obtain different labels for similar image subjects (e.g., food images are very subjective). 











  §.§.§ Error Analysis












We perform error analysis to further analyze 

r0.26






ℬ     overall     private     public  


object     16.52     14.00     23.75 

scene     27.71     21.78     42.63  




Tags     37.02     26.90     58.79 




Errors corrected (%). 




the results of the proposed approach. We first determine the errors generated by all the base classifiers in ℬ and corrected by the proposed approach DMFP. We calculate the percentage of corrected errors for private class, public class and overall (considering both the classes) and show them in Table <ref>. We compute the percentage of corrected errors as the number of corrected errors of private (or public) class over the total number of private (or public) class errors. We calculate the fraction of overall corrected errors by considering both public and private classes. The table shows that we correct 14% - 27% of private class errors, 18% - 58% of public class errors and overall we eliminate 16% - 37% errors. Note that errors generated for the private class are much larger than the public class (See Table <ref>) and hence, even a comparatively smaller percentage of corrected errors constitute to a significant improvement. We also analyze results by showing predictions of samples in Figure <ref>, for which at least one base classifier fails to predict the correct privacy of an image.  For instance, for example (b), scene model failed to predict the correct privacy of the image; however, DMFP identifies the competent base classifiers, i.e., object, and tag and predict the correct privacy label. We also show an example (image (d)) for which all the base classifiers fail to predict the correct privacy class and hence, the proposed approach also fails to predict the correct label. The image of a food is very subjective and hence, generic base classifiers will not be sufficient to predict the correct labels of such images. In the future, these generic models can be extended to develop hybrid approaches, that consider both generic and subjective privacy notions to predict  personalized privacy labels.





 §.§ Proposed Approach vs. Baselines









We compare the performance of the proposed approach DMFP with multi-modality based baselines described below. 




1. Model Selection by <cit.>: 
The authors proposed a deep multi-modal architecture for product classification in e-commerce, wherein they learn a decision-level fusion policy to choose between image and text CNN for an input product. Specifically, the authors provide class probabilities of a product as input to the policy trained on a validation dataset and use it to predict whether image CNN (or text CNN) should be selected for the input. In other words, policy determines the competence of the CNNs for its input and thus, we consider it as our baseline. For a fair comparison, we learn 3 policies (corresponding to the competence classifiers 𝒞), wherein each policy (say object policy) predicts whether the respective base classifier (object) should be selected for a target image. Note that we learn these policies on the 𝒟^E dataset. Finally, we take a majority vote of the privacy label predicted by the selected base classifiers (identified by the policies) for a target image.

2. Majority Vote: We consider a majority vote as another baseline, as we use it for final selection of privacy label for a target image. Unlike our approach, a vote is taken without any pre-selection of base classifiers. We predict privacy of a target image using base classifiers in ℬ and select a label having highest number of votes. 


3. Decision-level Fusion: Fixed rules, that average the predictions of the different CNNs <cit.> or select the CNN with the highest confidence <cit.>. The first rule is equivalent to the majority vote baseline, and hence, we show the results for the second rule only. The second rule is given as: Y_T = argmax_i([P_i^o + P_i^s + P_i^t] / 3), where i = 0 (public), 1 (private). P^o, P^s, and P^t denotes the posterior probabilities (private or public) obtained using object (B^o), scene (B^s) and tag (B^t) modality respectively.


4. Stacked Ensemble (Stacked–en):   

Stacking learns a meta-classifier to find an optimal combination of the base learners <cit.>. Unlike bagging and boosting, stacking ensembles robust and diverse set of base classifiers together, and hence, we consider it as one of the baselines. We use the same set of base classifiers ℬ to encode images in 𝒟^T using posterior probabilities P(Y_I = private| I, B_i) and P(Y_I = public| I, B_i) where B_i ∈ℬ. We train a meta-classifier on this encoded 𝒟^T dataset using calibrated SVM classifier. We use this meta-classifier to predict privacy class of an encoded target image T (using the posterior probabilities obtained by the base classifiers P(Y_T=private | T, B_i) and P(Y_T=public | T, B_i)). As we use 𝒟^E only to learn “competence” classifiers, we do not consider it for training a meta-classifier for a fair comparison.  

5. Clusters-based Models (Cluster–en):   We create 5 clusters of 𝒟^T dataset using hierarchical clustering mechanism and the combination of object, scene and tag features (F^ost). We train a calibrated SVM model on each cluster using the combination of features F^ost. For target image T, the most relevant cluster is identified using k = 15 nearest neighbors, and the model trained on that cluster is used to predict the privacy of the target image. We consider this as another baseline, because clustering images that are shared online, brings similar image types (e.g., portraits) together and models trained on these clusters can be competent to predict privacy of target images of respective image types. The number of clusters and neighbors are estimated based on the 𝒟^E dataset.











Table <ref> compares the performance obtained by the proposed approach DMFP with the performance obtained using the baseline models. We observe that DMFP learns better privacy characteristics than baselines with respect to private class by providing improvements of 4.5%-14% and 4%-20% in the F1-score and recall of private class. 

When we learn the “competence” of the base classifiers (ℬ) on the 𝒟^E dataset without identifying the neighborhoods (the first baseline, <cit.>), the precision, recall and F1-score drop by 9%, ≈ 6%, ≈ 7%. It is interesting to note that the precision of DMFP- CL (Refer Table <ref>, N_V-CL, N_P-CL, {N_V + N_P}-CL), i.e., 0.79 is better than the first baseline (<cit.>), i.e., 0.662 whereas the recall of the first baseline (0.568) is better than DMFP-CL (0.534). However, when we combine the neighborhoods ({N_V + N_P}) and the first baseline (competence learning), i.e., the proposed approach DMFP, we get better performance than each of these methods. Another detail to note that the performance of the first baseline (<cit.>) is very close to the image tags model (see Table <ref>, <ref>) and even though the baseline uses multi-modality, the performance does not exceed significantly over the individual base classifiers (object, scene, image). <cit.> performed well for product classification, but it failed to yield improved results for privacy prediction because unlike product images or ImageNet images (that contains single object in the image), images that are shared online are much more complex (containing multiple objects, and scene) and diverse (having different image subjects such as self-portraits, personal events). The results suggest that it is hard to generalize the competency of base classifiers on all types of image subjects and hence, the competence of the base classifiers needs to be determined dynamically based on the neighborhoods of a target image. 




Table <ref> also shows that F1-measure of private class improves from 0.637 achieved by majority vote (the second baseline), 0.65 obtained by decision-level fusion (the third baseline), 0.636 obtained by stacked–en (the fourth baseline), and 0.545 obtained by cluster–en (the fifth baseline) to 0.684 obtained by DMFP. Additionally, we notice that the proposed approach is able to achieve a performance higher than 85% in terms of all compared measures. Note that a naive baseline which classifies every image as “public” obtains an accuracy of 75%. With a paired T-test, the improvements over the baseline approaches for F1-measure  of a private class are statistically significant for p-values < 0.05.








 §.§ Proposed Approach vs. Prior Image Privacy Prediction Works

We compare the privacy prediction performance obtained by the proposed approach DMFP with the state-of-the-art works of privacy prediction: 1. object <cit.> (B^o), 2. scene <cit.> (B^s), 3. image tags <cit.> (B^t), 4.  PCNH privacy framework <cit.>, and 5. Concatenation of all features <cit.>. Note that the first three works are the feature sets of DMFP and are evaluated in the Experiment <ref>.  
We describe the remaining prior works (i.e., 4 and 5) in what follows. 
4. PCNH privacy framework <cit.>:
The framework combines features obtained from two architectures: one that extracts convolutional features (size = 24), and another that extracts object features (size = 24). The object CNN is a very deep network of 11 layers obtained by appending three fully-connected layers of size 512, 512, 24 at the end of the fully-connected layer of AlexNet <cit.>. The PCNH framework is first trained on the ImageNet dataset <cit.> and then fine-tuned on a privacy dataset.
5. Combination of Object, Scene and User Tags (Concat) <cit.>: 
<cit.> combined object and scene tags with user tags and achieved an improved performance over the individual sets of tags. Thus, we compare the proposed approach with the SVM models trained on the combination of all feature sets (F^ost) to show that it will not be adequate to predict the privacy of an image accurately. In our case, we consider object and scene visual features instead of tags and combine them with user tags to study multi-modality with the concatenation of visual and tag features.


Table <ref> compares the performance obtained by the proposed approach (DMFP) and prior works. We achieve the highest performance as compared to the prior works and show a maximum improvement of ≈ 10% in the F1-score of private class. We notice that our approach based on multi-modality yields an improvement of 11% over the recall of almost all the prior works (Refer Table <ref> and <ref>). 
Particularly, we show improvements in terms of all measures over the PCNH framework, that uses two kinds of features object and convolutional. We found that adding high-level descriptive features such as scene context and image tags to the object features help improve the performance. In addition to the individual feature sets, we also outperform the concatenation of these feature sets (denoted as “Concat”), showing that “Concat” could not yield an optimal combination of multi-modality.  We notice that the performance of “Concat” is slightly lower than the performance of base classifiers (Refer Tables <ref> and <ref>). We find this is consistent with <cit.> results, that concatenated various layers of image and tag CNN and trained the fused CNN end-to-end but did not yield a better performance than the individual CNN (image or tag). 



§ CONCLUSIONS AND FUTURE WORK

In this work, we estimate the competence of object, scene and image tag modalities, derived through convolutional neural networks and dynamically identify the set of most competent modalities for a target image to adequately predict the class of the image as private or public.  



The proposed approach contains three stages wherein we first identify neighborhoods for a target image based on visual content similarity and privacy profile similarity.  Then, we derive “competence” features from these neighborhoods and provide them to the “competence” classifiers to predict whether a modality is competent for the target image. Lastly, we select the subset of the most competent modalities and take a majority vote to predict privacy class of the target image.
Experimental results show that our approach predicts the sensitive (or private) content more accurately than the models trained on an individual modality (object, scene, and tags), multi-modality baselines and prior privacy prediction approaches.

Also, our approach could aid other applications such as event understanding, image classification, to on the fly decide which CNN (object, scene or tag) to use based on a target image. 


In the future, it will be interesting to study dynamic multi-modal fusion in personalized privacy setting. Also, other types of competence learning approaches 
and competence features can be developed for estimating the competence of base classifiers. 




§ ACKNOWLEDGMENTS

This research is supported by the NSF grant #1421970. The computing for this project was performed on Amazon Web Services.





ACM-Reference-Format




