P $\approx$ NP, at least in Visual Question Answering
Shailza Jolly Sebastian Palacio Joachim Folz Federico Raue Joern Hees Andreas Dengel
cs.CV






P ≈ NP, at least in Visual Question Answering
This work was supported by NVIDIA's NVAIL program and the BMBF project DeFuseNN (Grant 01IW17002)
    

1 Authors contributed equally
Shailza Jolly1,1,2,
Sebastian Palacio1,1,2,
Joachim Folz1,2,
Federico Raue2,
Jörn Hees2,
Andreas Dengel1,2


1DFKI GmbH

2TU Kaiserslautern 

Kaiserslautern, Germany 

<firstname.lastname@dfki.de>


        ^1 Department of Statistical Sciences, University College London, London WC1E 6BT, UK

    ^2 School of Mathematical Sciences, Queen Mary University of London, London E1 4NS, UK

^3 The Alan Turing Institute, The British Library, London NW1 2DB, UK 


===================================================================================================================================================================================================================================================================


In recent years, progress in the Visual Question Answering (VQA) field has largely been driven by public challenges and large datasets.
One of the most widely-used of these is the VQA 2.0 dataset, consisting of polar (“yes/no”) and non-polar questions.
Looking at the question distribution over all answers, we find that the answers “yes” and “no” account for 38 % of the questions, while the remaining 62 % are spread over the more than 3000 remaining answers.
While several sources of biases have already been investigated in the field, the effects of such an over-representation of polar vs. non-polar questions remain unclear.

In this paper, we measure the potential confounding factors when polar and non-polar samples are used jointly to train a baseline VQA classifier, and compare it to an upper bound where the over-representation of polar questions is excluded from the training.
Further, we perform cross-over experiments to analyze how well the feature spaces align.

Contrary to expectations, we find no evidence of counterproductive effects in the joint training of unbalanced classes.
In fact, by exploring the intermediate feature space of visual-text embeddings, we find that the feature space of polar questions already encodes sufficient structure to answer many non-polar questions.
Our results indicate that the polar (P) and the non-polar (NP) feature spaces are strongly aligned, hence the expression P≈NP.






§ INTRODUCTION



The task of Visual Question Answering (VQA) is highly interesting as it requires machine learning (ML) models to concurrently optimize multi-modal objectives related to natural language processing, object detection, and instance segmentation. As initial advancements in the field were quickly made, some research started analyzing the extent to which good results were merely achieved by exploiting low level biases in the datasets themselves. Not long after the release of several VQA datasets, a single study already found consistent significant sources of bias for at least six of them <cit.>.
Said biases are found primarily on the questions, and more precisely, on the way certain questions strongly correlate with the correct answer (e.g., questions that start with “what sport...” can be answered with “tennis” more often than not).
Imbalances like these have been widely studied and have even resulted in new metrics <cit.>,
regularizers <cit.> or re-balanced partitions for existing datasets <cit.>.

In this paper, we explore a different, potential source of bias in datasets that include both polar () and non-polar () questions together.
Polar questions are those with answer yes or no (commonly referred to as “yes/no” questions).
By extension, non-polar questions are the complement of the set of polar questions, i.e., questions with an answer that is something other than yes or no.
Polar questions have been criticized for their simplicity and often excluded in favor of questions that are richer in complexity <cit.>.
In fact, datasets like COCO-QA, VQA 1.0 or VQA 2.0 that make use of polar questions show an imbalance of the answer distribution where polar questions are significantly over-represented.
For the case of VQA 2.0, polar questions comprise 38 % of the available corpus.
This means that the answer yes appears roughly 19 % of the time, and so does the answer no.
In contrast, each of the remaining 3127 classes (answers) occur, on average, 2 % of the time (see Figure <ref>).



So far, the over-representation of polar samples has been approached in isolation, mainly in two different ways:
First, polar questions are either excluded entirely form the corpus or used exclusively <cit.>.
Second, for datasets with a mix of polar and non-polar questions, evaluation protocol dictates that the accuracy has to be reported separately for polar samples.
Despite this strict separation of the second scenario, VQA models are frequently trained jointly, treating each unique answer independently, regardless of polarity, and under i.i.d. conditions.
Moreover, state-of-the-art approaches do not make any specific mention about balancing techniques like class regularization or mini-batch resampling <cit.>.
This simple imbalance can cause severe performance issues, as proposed models could allocate more capacity to answering polar questions just because they appear more often during training.
In fact, we see how performance of polar questions is consistently superior to that of other non-polar sub-categories for the popular VQA challenge[<http://visualqa.org>].
Even after compensating for the number of classes per group, the expected mean accuracy of any random pair of non-polar classes is a lot lower than the one of polar questions.
The question we ask is if there is any measurable impact (positive or negative) stemming from the over-representation of polar questions in VQA datasets.
Are there confounding factors between polar and non-polar questions when projected into a common feature space?
Are polar questions occupying a non-overlapping region of the feature space w.r.t. their non-polar counterparts?
We investigate these questions and their implications by conducting a series of experiments on a high-performance VQA classifier.
By comparing its behaviour when data distribution changes during training and testing, we conclude, contrary to intuition, that there is a considerable overlap between features from polar questions and non-polar questions.
Moreover, this overlap is favorable to the overall optimization objective such that non-polar questions can be successfully answered based purely on polar features and vice versa.

The contributions of this paper are thereby two-fold:

  
  * An evaluation of the potential confounding factors (i.e. bias) that polar questions induce due to over-representation during training.
  
  * Empirical evidence indicating that the feature space from polar features can be used to answer non-polar questions and vice versa (≈).


The rest of the paper is organized as follows.
Section <ref> goes over previous work related to the study of polar and non-polar questions, as well as biases in VQA datasets.
Section <ref> outlines the experimental setup and training regimes.
Section <ref> describes the experimental results and Section <ref> brings results into perspective, discussing the implications in the context of the joint feature space of polar and non-polar questions.



§ RELATED WORK



Current developments in the field of deep learning have demanded large amounts of data for training and evaluation.
Models rely on the usefulness of these carefully curated corpuses, often taking for granted how representative a training set is.
Datasets being permeable to biases (spurious, undesired patterns) can drastically undermine the novelty and usefulness of certain ML models, as well as their claimed performance.

Problems dealing with natural text often struggle with issues of this nature.
For example, a strong gender bias has been reported for imSitu, a visual semantic role labeling dataset where activities like “cooking” were strongly biased towards women <cit.>.
Not only was the bias present in the dataset but trained models were amplifying the bias during testing as well.
The widely used MS-COCO dataset <cit.> presents biases for image captioning because objects and background often co-occur e.g., giraffes appear next to a tree with grass in the background <cit.>.
Being largely based on MS-COCO, the VQA 1.0 dataset <cit.> was therefore affected by this bias too, with the now infamous example of “tennis” being the correct answer to 41% of questions starting with “what sport...”.
Not long after, a study focusing on the limitations of current VQA datasets found that six of the most commonly used corpuses contained some sort of bias related to the textual domain <cit.>.
The ubiquity of said imbalances results in an inability to measure the extent by which VQA models are indeed capable of extracting meaningful, and visually grounded semantics.

From this point on, several advances in VQA were directly addressing biases in the existing datasets.
Alternative training and testing splits for VQA 1.0 and 2.0 were introduced in order to measure the extend by which VQA models can cope with unseen composition of concepts e.g., after learning about “green plate” and “white shirt” the network is then evaluated on “white plates” or “green shirts” <cit.>.
Simultaneously, the emergence of several regularization techniques helped models compensate or at least mitigate the effects of biases picked up by language models <cit.>.

On the behavior of polar and non-polar questions, research has focused, among other things, on the importance of having polar questions balanced, i.e., having the same number of questions with “yes” and “no” as an answer <cit.>.
Moreover, a polar-only dataset was proposed to measure the extent by which logic reasoning can be solved by a VQA system <cit.>.
So far, these experiments only analyze effects of imbalances within the polar space in isolation, disregarding the interactions that polar and non-polar questions may have in a joint feature space.
A recapitulation of good practices for training models using VQA 2.0 recommends balancing each mini-batch w.r.t. opposite questions <cit.>.
This policy ensures that there is always a balance between samples with “yes” and “no” answers, but does not consider the balance between polar and non-polar questions.
An explicit separation of polar and non-polar questions was proposed for the GVQA model, which processes polar questions separately from non-polar ones, using the former as a verification mechanism for the non polar concepts contained within <cit.>.

Since polar questions can be obtained easier than non-polar ones, datasets relying on human annotators end up with a distribution of question types that is heavily skewed towards polar questions.
Some synthetically generated datasets like CLEVR compensate for such over-representation, having a more uniform distribution along different answer types <cit.>.
On the other side of the spectrum, synthetic and natural datasets have outright dismissed the use of polar questions to alleviate such over-representation issues <cit.>, and because of this restriction, more complex questions can be attained <cit.>.

In this paper, we are concerned with the influence that over-represented polar questions exert on the non-polar counterparts when trained jointly, as it is the case for a majority of modern VQA models.
Experiments that eliminate confounding factors between the two categories are conducted in order to assess the upper bound of a VQA model when using only one or the other kind of question.
Furthermore, we find a strong alignment between the feature space of polar questions and that of non-polar questions, indicating that polar questions can be used to answer non-polar answers and vice versa.



§ METHODS



In this section, we present the main model and variations thereof used for all the experiments, motivate the use of the dataset, and describe the metrics to be compared afterwards.



 §.§ Dataset


We use the VQA 2.0 dataset for all experiments.
With 443 757 training samples and 214 354 for validation, it is currently the largest available non-synthetic corpus for VQA containing both polar and non-polar questions.
Additional properties that make this dataset suitable for our analysis include a uniform distribution between questions with “yes” and “no” as ground-truth, as well as an adjusted distribution of non-polar answers w.r.t. VQA 1.0.
We split the VQA 2.0 corpus into two disjoint sets corresponding to the polar and non-polar questions.
Notwithstanding, samples also remain in the training and validation set as originally assigned.



 §.§ VQA Reference Model


For our experiments, we use a high-performance VQA system with region-based attention <cit.>.
Figure <ref> shows an overview of the model which, at its core, consists of three modules: an image embedding, a text embedding and a joint classification module.
To model the joint visual-text space, both image and text embeddings are first projected to a 512-dimensional space, and then fused together through an element-wise product.
Subsequently, the joint embedding passes through a sequence of fully connected layers before reaching the output layer, where the output is normalized by a softmax operation.
In this paper, we refer to the first part of the network (up until the point-wise multiplication of the 512-dimensional projection of the visual-text space) as the base VQA network, denoted as Φ.
The remaining part of the network consisting of two fully connected layers and the output layer is referred to as the classifier and denoted as f.
This way, a prediction by the network can be written as ŷ = f(Φ(x)).
When only using polar questions for training the model, we use Φ_ and f_ to denote the corresponding modules.
Similarly, Φ_ and f_ refer to modules that have been trained using non-polar samples only.
For completeness, Φ_Ω and f_Ω denote modules trained on both polar and non-polar questions. 





 §.§ Experiments

There are three main experiments in this work, and they can be summarized as follows:

Baseline: We train the entire model on VQA 2.0 without any additional considerations regarding polar and non-polar questions.
As in the original work <cit.>, we make use of samples from VQA 2.0 containing answers that appear at least eight times in the entire dataset. 
This yields an answer space of 3129 dimensions, two of which correspond to the classes “yes” and “no” (i.e., the polar space).
No additional pre-training from Visual Genome is used to avoid potential effects of biases coming from another dataset.
The number of regions used for the image embedding is fixed at 36.
Our final implementation uses ReLUs instead of the originally proposed GatedTanh, since it saves computation and produces similar results. 
The model is trained using Adamax <cit.> with an initial learning rate of 2× 10^-3 on the full training set, and the standard VQA accuracy is reported for the validation set <cit.>.
We report the VQA accuracy of the baseline w.r.t. three splits of the validation set: 1) using only the polar questions in the validation set, 2) using only the non-polar questions and 3) passing the entire validation set which contains both polar and non-polar questions (see Figure <ref>).
The baseline is used as a reference to quantify the impact of polar and non-polar questions when used together during training (this experiment) or separately (following experiments).




Unbiased Upper Bound: To get an empirical upper bound of the model, where the issue of over-representation does not play a role, we train two separate versions of the same model from scratch.
First, we train a model only using the polar questions.
Then we train a second model only using the non-polar questions.
We refer to these two models as f_∘Φ_ and f_∘Φ_ respectively.
For these two models, their corresponding VQA accuracy is reported (see Figure <ref>): f_∘Φ_ is evaluated on the polar questions of the validation set, and f_∘Φ_ on the non-polar ones.
The purpose of this setup is to compare the capacity of the same VQA model used for the baseline, when dealing only with one kind of question.
By training on polar or non-polar questions only, the network can use 100 % of its capacity to extract the necessary semantics without the burden of modeling features to distinguish between polar and non-polar questions.
In other words, any bias caused by the imbalance of polar and non-polar questions is excluded for these two models.
The accuracy for both variants is thereby expected to be higher than the corresponding value for the baseline experiment.
If small or no deviations arise w.r.t. the baseline, then we can conclude that the confounding factors between polar and non-polar questions are not affecting the baseline VQA model negatively.



Cross-Polarity Evaluation: Independent of the potential confounding factors between polar and non-polar samples, there is still the question on how the distribution of polar features overlap with that of the non-polar ones.
As the feature projections from Φ_ and Φ_ share the same dimensional space, an experiment using transfer learning can be performed.
The outline of this experiment is shown in Figure <ref>.
First, we use Φ_ and Φ_ from the previous experiment as fixed pre-trained feature extractors.
Then, we train a new polar classifier f_ using features from the fixed pre-trained non-polar module Φ_.
Similarly, we train a new non-polar classifier f_ with features coming from Φ_.
In order to be able to compare the results, we use the same architecture for f_ and f_ as in the previous unbiased upper bound experiments.
Intuitively, this cross-over experiment helps us measuring how descriptive the feature space of polar questions is, so that non-polar questions can also be answered, and vice versa.
At a semantic level, this is theoretically possible since non-polar questions can be asked using a polar structure.
E.g., the question “What color are the bird's tail feathers?” with answer “white” can be transformed into the polar question “Are the bird's tail feathers white?”.
Intuitively, it is expected that the space of non-polar concepts (i.e., the joint visual-text projection trained on the non-polar questions Φ_ covering more than 3000 answer classes) represents a rich enough structure that can be condensed and reused to answer polar questions (just 2 answer classes).
However, it is not expected that the set of polar-questions yields a rich enough embedding space covering the wide spectrum of non-polar concepts found in VQA 2.0.





§ RESULTS





In this section, we report the results from experiments described in Section <ref>.

Table <ref> gives an aggregated overview of the main experimental results from all three proposed experiments.
The columns show, from left to right, the name of the experiment, the subset (: Polar; : Non-polar; Ω: All) of the set of samples used for training the corresponding module of the whole VQA ensemble (Φ or f), the subset of the validation set used for evaluation, and the resulting VQA accuracy.
For the cross-polarity experiments, Φ is assumed to be pre-trained and fixed, and only the corresponding f has been trained from scratch.

We observe that the baseline experiment reaches an accuracy that is within one percentage point of the one reported by the original authors <cit.>. 
The decisions to not involve another dataset for pre-training (e.g., the Visual Genome as in the original work) naturally affects the overall accuracy, but allows the results presented here to reflect more closely the behavior of the polar and non-polar disparity, while ruling out other potential sources of bias.

For the characterization of the upper bound, we see that the reached accuracy falls almost exactly within the range of the baseline.
Upper bound results show an increase of 0.2 pp for a system trained on non-polar questions while training with polar questions decreases accuracy by 0.8 pp.
These negligible fluctuations between the upper bound and the baseline strongly suggest that no confounding factors exist between polar and non-polar samples when trained jointly.
In fact, we see that polar questions rarely get confused with any non-polar alternative (Table <ref>).
Similarly, non-polar questions are not frequently mistaken for any of the two polar answers.
For further in-depth analysis, please refer to the discussion in Section <ref>.




The cross-polarity experiments exhibit a slightly different behaviour.
First, training a polar classifier f_ based on non-polar features from Φ_ yields results almost as high as when Φ_ is used.
Only 3.8 pp of accuracy separate the cross polar model f_∘Φ_ and the polar upper bound f_∘Φ_.
Taking into account that both results are above 75 % accuracy, their difference can be considered low, especially when compared against the probability of randomly guessing which lies at 50 %.
Second, the inverse cross-polar experiment, where non-polar questions are classified using a polar-only feature space f_∘Φ_ shows an accuracy of 28.7 %.
This presents a loss of -22.9 pp w.r.t. the upper bound f_∘Φ_.
Note that this result still lies notably above random chance (which corresponds to 0.03 %), and will be discussed further in the next section.




§ DISCUSSION



In this section, we analyze the results from all conducted experiments and their implications, in the context of the joint space shared by polar and non-polar features.

The first observation comes from comparing the baseline experiments against their corresponding empirical upper bound for both polar and non-polar features.
Having almost indistinguishable results when the model is being trained with non-polar and over-represented polar samples together, indicates that the model is capable of coping with both question types simultaneously, without compromising performance.
In essence, this simple comparison shows no measurable confounding factors (i.e., source of bias) by populating the visual-text feature space with polar and non-polar questions at the same time.
The need for balancing strategies, commonly used for long tail datasets like mini-batch resampling or weighted labels <cit.> are thereby unlikely to improve the performance of typical VQA model architectures like the one used in this work.
Regarding the distribution of polar and non-polar questions in the joint feature space Φ_Ω, we are left with two possible scenarios: either each distribution occupies a different (disjoint) sub-region of the feature space or they overlap and hence, they (at least partially) model the same semantic concepts.
An analysis of the remaining experiments will help identifying which of these two conjectures can be verified.

The second observation comes from the first cross-polarity experiment.
Here, by training a classifier on polar questions that are first projected to a feature manifold of non-polar concepts, we measure how possible it is to answer polar questions by using the feature space of non-polar concepts.
This scenario is intuitively simple, because of the vast complexity of topics covered by non-polar questions (3127 answer classes in our experiments).
Also, by reducing the number of classes from 3127 to 2, the chance level of the classification problem increases from 13127 to 12, which makes the problem much easier.
Hence, the observed result is not really surprising: with only a minor drop in performance, we can say that most polar questions can be answered even by using a non-polar feature space representation.
Looking for the alignment between a non-polar concept (e.g., “green” or “bicycle”) and the occurrence of that concept in the question embedding, makes polar questions straightforward to answer, even by a simple classifier.

The third, and perhaps most interesting observation comes from the other direction of the cross-polarity experiments.
In this experiment we trained a classifier on non-polar questions while relying on a polar feature space.
Unlike before, we are now going up from 2 to 3127 classes, intuitively making the problem much more challenging.
Also it is unclear to what extent we can expect the polar feature space to be able to express the intricacies of non-polar questions.
As mentioned in Section <ref>, several VQA datasets have decided to entirely leave out polar samples to favor the more complex questions arising from only allowing non-polar queries.
According to these premises, we expect the non-polar classifier based on features from a polar embedding space to perform poorly.
Recall that the reported accuracy for this setup is 28.7 % which is indeed lower than the upper bound of 51.6 %.
However, we note that this value is still significantly higher than random chance of 13127.
This behavior suggests that there is a notable subset of non-polar questions that can be answered with high accuracy based only on the feature space of polar questions, immediately leading to the question:
How to find the subset of non-polar questions that can be answered through the polar feature space?
The cross-polarity experiments already show that non-polar questions with numeric answers are well conveyed by the polar feature space (accuracy of f_∘Φ_ for numeric labels such as “0”, “1” and “2” is high).
We theorize that a general alignment between polar questions about non-polar concepts and the corresponding non-polar questions exists.





  §.§.§ Polar Questions About Non-Polar Concepts

Take the example in Figure <ref>: The non-polar question “What is the woman holding?” and the the polar question “Is the woman holding a glass?” relate to the same semantic concept, namely “glass”.
By counting the number of polar questions that talk about each of the non-polar concepts, we can focus on the non-polar concepts which appear in most polar questions.
We call this category of non-polar answers “well covered”.
We use the notation 𝒳_' to refer to non-polar samples (i.e., questions and answers) with answers that are well covered by polar questions.
The complement of 𝒳_' (i.e., non-polar samples whose answers are not well covered by polar questions) is denoted by 𝒳_'.
We can then test the f_∘Φ_ model used in the last experiment of the cross-polarity evaluation w.r.t. 𝒳_' and 𝒳_' separately.



To populate the subset 𝒳_', we start by selecting polar questions in which any of the 3127 non-polar answers occur textually using a simple regular expression. 
We then count the number of polar question occurrences for each non-polar answer (with replacement) and sort them in ascending order.
The resulting histogram is shown in Figure <ref>.
The x-axis represents the 3127 non-polar answers and the y-axis represents the number of polar questions matching the non-polar answer.
We see that 73.87% of non-polar concepts are matched by at least 1 polar question.
To guarantee that each non-polar concept in the subset is covered by a large number of polar questions, we select the top 500 non-polar answers that occur the most often within polar questions (i.e., the 500 best covered non-polar answers) to assign non-polar samples to 𝒳_' (shown in blue in Figure <ref>).
Once 𝒳_' (500 classes), and thereby 𝒳_' (2627 classes) have been defined, we use them to evaluate f_∘Φ_ separately.
Results of this experiment are shown in Table <ref>.



When evaluating the polar feature space w.r.t. well-covered non-polar questions, the VQA model exhibits an ample improvement (from 28.7 % to 40 %) compared to the initial cross-polar experiment from Section <ref>.
These results are in turn, closer to the upper bound of non-polar questions that can be answered with the VQA model used throughout all experiments.
In contrast, the polar feature space of poorly covered non-polar concepts presents a steep decrease in accuracy, from 28.7 % to 14 %.
These results give a strong indication that non-polar questions can be answered by using a feature space based on polar samples.
The caveat is, quite naturally, that the set of polar questions used for training, has to convey enough semantics about the corresponding non-polar questions.

In light of these results, we construct a logical argument, where a sufficient condition for modeling a non-polar questions with a polar feature space depends on having polar questions that deal with the corresponding non-polar concepts.
Therefore, we arrive at the conclusion that the polar feature space can carry an equivalent semantic value as the non-polar feature space, hence ≈.



§ CONCLUSION AND FUTURE WORK



In this paper, we have presented an in-depth evaluation of the influence that polar and non-polar questions exert on each other when used jointly for training a VQA system.
The over-representation of polar samples w.r.t. the non-polar counterparts poses two main questions which we developed throughout this work: (1) Are there any source of bias stemming from the polar and non-polar imbalance and (2) what relationship exists between polar and non-polar samples when projected into the joint feature space that they are usually represented in?

On the first question, we found no confounding factors from the imbalance of polar and non-polar questions, and thereby no detrimental source of bias which may require special attention by doing class weighting or mini-batch re-sampling.

On the second question, we establish a clear correlation between the distribution of polar and non-polar feature embeddings.
We show that polar features can be used to answer non-polar questions, provided that the polar questions used for training refer to the semantic concepts being considered in the non-polar questions.
Based on these findings we conclude that the space of polar features () provides a rich semantic structure, similar to that of the non-polar counterparts ().
We use the expression ≈ to refer to this alignment.



  §.§.§ Future Work
 the results of this work indicate that the problem of visual question answering for non-polar concepts can be solved using polar questions, as long as polar questions cover the relevant non-polar topics.
We are interested in measuring the empirical extent by which this phenomenon holds.
The usefulness of a feature space based on polar samples for answering non-polar questions is not only surprising, but also potentially ground-breaking because it can change the way future VQA datasets are compiled.
Given the reduced cost of collecting polar questions (compared to non-polar questions), crowd sourcing efforts to amass a critical amount of polar questions for VQA 2.0 could benefit from our findings.
This will allow us to bridge the gap between non-polar concepts that are not well covered by polar questions and complement today's training data.
We also want to explore automatic means to turn non-polar questions into polar ones using natural-language-processing tools.

Furthermore, we are interested in measuring the extent to which a growing number of non-polar concepts can be modeled in the joint visual-text space by only using polar input samples.
Therefore, an arbitrary number of concepts could be explicitly imposed in the joint feature space while keeping a fixed 2-dimensional classification objective. 
This training regime, resembles the conditions of generative adversarial networks (GANs), and could open the possibility to learn new classes over time, which has potential applications in the field of continuous learning. 

In this paper, we presented a simple way to check for biases coming from the over-representation of 2 classes.
We used state-of-the-art VQA 2.0 dataset in our experiments due to its property of holding imbalance between polar (yes/no) and non-polar samples.
We trained these two samples separately, and in our experiments discovered that polar samples in spite of their huge number don't add any biases onto non-polar samples.
Investigating this no-bias situation, we observed that feature spaces of  &  samples are modeling the same information. Our experimental results, for transfer learning from one set to another, shows that P and NP are not disjoint.
Their feature spaces are highly correlated due to an overlap between NP classes and P questions. Hence, there are no biases from polar questions in VQA 2.0.

Since polar and non-polar questions convey the same information, presenting everything in polar questions would be an optimal choice for future VQA datasets.
These questions can cover multiple concepts in just two classes.
Non-polar questions, on the other hand, create a new class for such coverage.
For ex. NP class 'Red' can encode limited information, whereas P class 'Yes or No' has an infinite scope to encode the verification of absence or presence of different concepts. 
However, the information gain by network stays the same when trained using 2 and 3127 classes for VQA 2.0.
Therefore, turning non-polar questions into polar will both scale and make VQA task more flexible.
We hope that researchers will be benefited from our findings and will refer it to the creation of new VQA datasets.


00

b1 Anderson, Peter, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. "Bottom-up and top-down attention for image captioning and visual question answering." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077-6086. 2018.

b2 Teney, Damien, Peter Anderson, Xiaodong He, and Anton van den Hengel. "Tips and tricks for visual question answering: Learnings from the 2017 challenge." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4223-4232. 2018.

b3 Antol, Stanislaw, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. "Vqa: Visual question answering." In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433. 2015.

b4 Agrawal, Aishwarya, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. "Don't just assume; look and answer: Overcoming priors for visual question answering." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4971-4980. 2018.

b5 Ramakrishnan, Sainandan, Aishwarya Agrawal, and Stefan Lee. "Overcoming language priors in visual question answering with adversarial regularization." In Advances in Neural Information Processing Systems, pp. 1541-1551. 2018.

b6 Zhao, Jieyu, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. "Men also like shopping: Reducing gender bias amplification using corpus-level constraints." arXiv preprint arXiv:1707.09457 (2017).

b7 Agrawal, Aishwarya, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. "C-vqa: A compositional split of the visual question answering (vqa) v1. 0 dataset." arXiv preprint arXiv:1704.08243 (2017).

b8 Lin, Tsung-Yi, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. "Focal loss for dense object detection." In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988. 2017.

b9 Kafle, Kushal, and Christopher Kanan. "An analysis of visual question answering algorithms." In Proceedings of the IEEE International Conference on Computer Vision, pp. 1965-1973. 2017.

b10 Krishna, Ranjay, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen et al. "Visual genome: Connecting language and vision using crowdsourced dense image annotations." International Journal of Computer Vision 123, no. 1 (2017): 32-73.

b11 Johnson, Justin, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. "Clevr: A diagnostic dataset for compositional language and elementary visual reasoning." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910. 2017.

b12 Suhr, Alane, Mike Lewis, James Yeh, and Yoav Artzi. "A corpus of natural language for visual reasoning." In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 217-223. 2017.

b13 Zhang, Peng, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. "Yin and yang: Balancing and answering binary visual questions." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5014-5022. 2016.

b14 Zhu, Yuke, Oliver Groth, Michael Bernstein, and Li Fei-Fei. "Visual7w: Grounded question answering in images." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995-5004. 2016.

b15 Andreas, Jacob, Marcus Rohrbach, Trevor Darrell, and Dan Klein. "Neural module networks." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 39-48. 2016.

b16 Lin, Tsung-Yi, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. "Microsoft coco: Common objects in context." In European conference on computer vision, pp. 740-755. Springer, Cham, 2014.

b17 Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014).



