Grounded PCFG Induction with Images

Recent work in unsupervised parsing has tried to incorporate visual information into learning, but results suggest that these models need linguistic bias to compete against models that only rely on text. This work proposes grammar induction models which use visual information from images for labeled parsing, and achieve state-of-the-art results on grounded grammar induction on several languages. Results indicate that visual information is especially helpful in languages where high frequency words are more broadly distributed. Comparison between models with and without visual information shows that the grounded models are able to use visual information for proposing noun phrases, gathering useful information from images for unknown words, and achieving better performance at prepositional phrase attachment prediction.


Introduction
Recent grammar induction models are able to produce accurate grammars and labeled parses with raw text only (Jin et al., 2018bKim et al., 2019b,a;Drozdov et al., 2019), providing evidence against the poverty of the stimulus argument (Chomsky, 1965), and showing that many linguistic distinctions like lexical and phrasal categories can be directly induced from raw text statistics. However, as computational-level models of human syntax acquisition, they lack semantic, pragmatic and environmental information which human learners seem to use (Gleitman, 1990;Pinker and MacWhinney, 1987;Tomasello, 2003). This paper proposes novel grounded neuralnetwork-based models of grammar induction which take into account information extracted from images in learning. Performance comparisons show Figure 1: Examples of disambiguating information provided by images for the prepositional phrase attachment of the sentence Mary eats spaghetti with a friend (Gokcen et al., 2018). that the proposed models achieve state-of-the-art results on multilingual induction datasets, even without help from linguistic knowledge or pretrained image encoders. Experiments show several specific benefits attributable to the use of visual information in induction. First, as a proxy to semantics, the co-occurrences between objects in images and referring words and expressions, such as the word spaghetti and the plate of spaghetti in Figure 1, 2 provide clues to the induction model about the syntactic categories of such linguistic units, which may complement distributional cues from word collocation which normal grammar inducers rely on solely for induction. Also, pictures may help disambiguate different syntactic relations: induction models are not able to resolve many prepositional phrase attachment ambiguities with only text -for example in Figure 1, there is little information in the text of Mary eats spaghetti with a friend for the induction models to induce a high attachment structure where a friend is a companion -and images may provide information to resolve these ambiguities. Finally, images may provide grounding information for unknown words when their syntactic properties are not clearly indicated by sentential context.

Related work
Existing unsupervised PCFG inducers exploit naturally-occurring cognitive and developmental constraints, such as punctuation as a proxy to prosody (Seginer, 2007), human memory constraints (Noji and Johnson, 2016;Shain et al., 2016;Jin et al., 2018b), and morphology , to regulate the posterior of grammars which are known to be extremely multimodal (Johnson et al., 2007). Models in Shi et al. (2019) also match embeddings of word spans to encoded images to induce unlabeled hierarchical structures with a concreteness measure . Additionally, visual information is observed to provide grounding for words describing concrete objects, helping to identify and categorize such words. This hypothesis is termed 'noun bias' in language acquisition (Gentner, 1982(Gentner, , 2006Waxman et al., 2013), through which the early acquisition of nouns is attributed to nouns referring to observable objects. However, the models in Shi et al. (2019) also rely on language-specific branching bias to outperform other text-based models, and images are encoded by pretrained object classifiers trained with large datasets, with no ablation to show the benefit of visual information for unsupervised parsing. Visual information has also been used for joint training of prepositional phrase attachment models (Christie et al., 2016) suggesting that visual information may contain semantic information to help disambiguate prepositional phrase attachment.

Grounded Grammar Induction Model
The full grounded grammar induction model used in these experiments, ImagePCFG, consists of two parts: a word-based PCFG induction model and a vision model, as shown in Figure 2. The two parts have their own objective functions. The PCFG induction model, called NoImagePCFG when trained by itself, can be trained by maximizing the marginal probability P(σ) of sentences σ. This part functions similarly to previously proposed PCFG induction models (Jin et al., 2018a;Kim et al., 2019a) where a PCFG is induced through maximization of the data likelihood of the training corpus marginalized over latent syntactic trees.
The image encoder-decoder network in the vision model is trained to reconstruct the original image after passing through an information bottleneck. The latent encoding from the image encoder may be seen as a compressed representation of vi-sual information in the image, some of which is semantic, relating to objects in the image. We hypothesize that semantic information can be helpful in syntax induction, potentially through helping three tasks mentioned above.
In contrast to the full model where the encoded visual representations are trained from scratch, the ImagePrePCFG model uses image embeddings encoded by pretrained image classifiers with parameters fixed during induction training. We hypothesize that pretrained image classifiers may provide useful information about objects in an image, but for grammar induction it is better to allow the inducer to decide which kind of information may help induction.
The two parts are connected through a syntacticvisual loss function connecting a syntactic sentence embedding projected from word embeddings and an image embedding. We hypothesize that visual information in the encoded images may help constrain the search space of syntactic embeddings of words with supporting evidence of lexical attributes such as concreteness for nouns or correlating adjectives with properties of objects. 3

Induction model
The PCFG induction model is factored into three submodels: a nonterminal expansion model, a terminal expansion model and a split model, which distinguishes terminal and nonterminal expansions. The binary-branching non-terminal expansion rule probabilities, 4 and unary-branching terminal expansion rule probabilities in a factored Chomskynormal-form PCFG can be parameterized with these three submodels. Given a tree as a set τ of nodes η undergoing non-terminal expansions c η → c η1 c η2 (where η ∈ {1, 2} * is a Gorn address specifying a path of left or right branches from the root), and a set τ of nodes η undergoing terminal expansions c η → w η (where w η is the word at node η) in a parse of sentence σ, the marginal a giraffe is eating leaves < l a t e x i t s h a 1 _ b a s e 6 4 = " / t + R k o Y i T P P 2 q C O / l 4 7 1 V f u q 1 1 4 = " > A A A C H n i c b Z B L S w M x F I U z 9 V X r q 9 a l m 2 A R K k i Z K Q V 1 V 3 D j s o J 9 Q G c o m T T T h u Y x J B m 1 D P N X x J 3 + E n f i V n + I e 9 P H Q l s P B A 7 n 3 E s u X x g z q o 3 r f j m 5 t f W N z a 3 8 d m F n d 2 / / o H h Y a m u Z K E x a W D K p u i H S h F F B W o Y a R r q x I o i H j H T C 8 f W 0 7 9 w T p a k U d 2 Y S k 4 C j o a A R x c j Y q F 8 s Q S s / V j J M f U 2 H H G X n / W L Z r b o z w V X j L U w Z L N T s F 7 / 9 g c Q J J 8 J g h r T u e W 5 s g h Q p Q z E j W c F P N I k R H q M h 6 V k r E C c 6 S G e 3 Z / D U J g M Y S W W f M H C W / t 5 I E d d 6 w k M 7 y Z E Z 6 e V u G v 7 X 9 R I T X C v a C Z A W s q J M d 3 A T 2 2 U E 2 0 Z 5 V B U w s x A S u i I D K D r q C Q C T J R P o x f 4 w C l 9 n C j t R l o 8 V b 9 f 5 E Q Y M x a x 2 5 w E N b + 9 i f i f 1 8 1 s c h r l T K a Z B U l n j 5 K M Y 6 v w p A f c Z x q o 5 W N H C N X M Z c V 0 S D S h 1 r V V C S X c U y U E k f 0 8 T L W K i 2 4 Q 5 X l o E t w s D m v B U V F x P Q W / W / l L 2 o 1 6 c F w / u z q u n T f m j Z X R H t p H h y h A J + g c X a I m a i G K 7 t E D e k L P 3 q P 3 4 r 1 6 b 7 P V k j e / 2 U U / 4 L 1 / A S C S o d I = < / l a t e x i t > e m < l a t e x i t s h a 1 _ b a s e 6 4 = " q R M W 3 n B D U 4 f M l v o s u W a X b 4 Q 9 B L s = " > A A A C F n i c b V C 7 S g N B F J 2 N r x h f U U u b w S D E J u y G g N o F b C w j G B W y q 8 x O 7 s b B e S w z s 0 p Y 9 j f E T r / E T m x t / R B 7 Z 5 M U v g 5 c O J x z L / d w 4 p Q z Y 3 3 / w 6 v M z S 8 s L l W X a y u r a + s b 9 c 2 t c 6 M y T a F P F V f 6 M i Y G O J P Q t 8 x y u E w 1 E B F z u I h v j 0 v / 4 g 6 0 Y U q e 2 X E K k S A j y R J G i X V S G A p i b + I k h + J K X N c b f s u f A P 8 l w Y w 0 0 A y 9 6 / p n O F Q 0 E y A t 5 c S Y Q e C n N s q J t o x y K G p h Z i A l 9 J a M Y O C o J A J M l E 8 y F 3 j P K U O c K O 1 G W j x R v 1 / k R B g z F r H b L D O a 3 1 4 p / u c N M p s c R j m T a W Z B 0 u m j J O P Y K l w W g I d M A 7 V 8 7 A i h m r m s m N 4 Q T a h 1 N d V C C f d U C U H k M A 9 T r e J i E E R 5 H p o E 9 4 p m I 9 g v a q 6 n 4 H c r f 8 l 5 u x V 0 W k e n n U a 3 P W u s i n b Q L m q i A B 2 g L j p B P d R H F K X o A T 2 h Z + / R e / F e v b f p a s W b 3 W y j H / D e v w B / t 5 9 v < / l a t e x i t > L( , m) < l a t e x i t s h a 1 _ b a s e 6 4 = " X h j R 5 w F M x e m I r Y c v E D z / T q 5 o U l s = " r v 0 Q 9 0 5 r F t p 6 4 M L h n H u 5 9 5 4 o Y V Q b 1 / 1 0 S k v L K 6 t r 6 x v l z a 3 t n d 3 K 3 v 6 t l q n C p I s l k 6 o X I U 0 Y F a R r q G G k l y i C e M T I X T S + m P p 3 9 0 R p K s W N m S Q k 5 G g o a E w x M l b y r 2 q B p k O O T n i 9 X 6 m 6 D X c G u < l a t e x i t s h a 1 _ b a s e 6 4 = " Z 6 e D J l 1 E k probability of σ can be computed as: We first define a set of Bernoulli distributions that distribute probability mass between terminal and nonterminal rules, so that the lexical expansion model can be tied to the image model (see Section 4.2): (2) where c η is a non-terminal category, W spl ∈ R 2×h and b spl ∈ R 2 are model parameters for hidden vectors of size h, and x B,c η ∈ R h the result of a multilayered residual network (Kim et al., 2019a). The residual network consists of B architecturally identical residual blocks. For an input vector x b−1,c each residual block b performs the following computation: with base case: where δ c is a Kronecker delta function -a vector with value one at index c and zeros everywhere else -and E ∈ R d×C is an embedding matrix for each nonterminal category c with embedding size d, and are model parameters with latent representations of size h. B is set to 2 in all models following Kim et al. (2019a). Binary-branching non-terminal expansion rule probabilities for each non-terminal category c η and left and right children c η1 c η2 are defined as: where W nont ∈ R C 2 ×d and b nont ∈ R C 2 are parameters of the model. The lexical unary-expansion rule probabilities for a preterminal category c η and a word w η at node η are defined as: where w is the generated word type, and w lex ∈ R h and b lex ∈ R are model parameters. Similarly, with base case: L is a matrix of syntactic word embeddings for all words in vocabulary.

Vision model
The vision model consists of an image encoderdecoder network and a syntactic-visual projector. The image encoder-decoder network encodes an image into an image embedding and then decodes that back into the original image. This reconstruction constrains the information in the image embedding to be closely representative of the original image. The syntactic-visual projector projects word embeddings used in the calculation of lexical expansion probabilities into the space of image embeddings, building a connection between the space of syntactic information and the space of visual information.

The image encoder-decoder network
The image encoder employs a ResNet18 architecture (He et al., 2016) which encodes an image with 3 channels into a single vector. The encoder consists of four blocks of residual convolutional networks. The image decoder decodes an image from a visual vector generated by the image encoder. The image decoder used in the joint model is the image generator from DCGAN (Radford et al., 2016), where a series of transposed convolutions and batch normalizations attempts to recover an image from an image embedding. 5

The syntactic-visual projector
The projector model is a CNN-based neural network which takes a concatenated sentence embedding matrix M σ ∈ R |σ|×d as input, where embeddings in M σ are taken from L, and returns the syntactic-visual embedding e σ . The jth full lengthwise convolutional kernel is defined as a matrix K j ∈ R u j ×k j d which slides across the sentence matrix M to produce a feature map, where u j is the number of channels in the kernel, k j is the width of the kernel, and d is the height of the kernel which is equal to the size of the syntactic word embeddings. Because the kernel is as high as the embeddings, it produces one vector of length u j for each window. The full feature map F j ∈ R u j ×H j , where H j is total 5 Details of these models can be found in the cited work and the appendix. number of valid submatrices for the kernel, is: (10) Finally, an average pooling layer and a linear transform are applied to feature maps from different kernels:f All Ks, bs and Ws here are parameters of the projector.

Optimization
There are three different kinds of objectives used in the optimization of the full grounded induction model. The first loss is the marginal likelihood loss for the PCFG induction model described in Equation 1, which can be calculated with the Inside algorithm. The second loss is the syntactic-visual loss. Given the encoded image embedding e m and the projected syntactic-visual embedding e σ of a sentence σ, the syntactic-visual loss is the mean squared error of these two embeddings: L(e m , e σ ) = (e m − e σ ) (e m − e σ ).
The third loss is the reconstruction loss of the image. Given the original image represented as a vector i m and the reconstructed imageî m , the reconstruction objective is the mean squared error of the corresponding pixel values of the two images: Models with different sets of input optimize the three losses differently for clean ablation. NoIm-agePCFG, which learns from text only, optimizes the negative marginal likelihood loss (the negative of Equation 1) using gradient descent. The model with pretrained image encoders, ImagePrePCFG, optimizes the negative marginal likelihood and the syntactic-visual loss (Equation 13) simultaneously. The full grounded grammar induction model Im-agePCFG learns from text and images jointly by minimizing all three objectives: negative marginal likelihood, syntactic-visual loss and image reconstruction loss (Equation 14): L(σ, m) = −P(σ) + L(e m , e σ ) + L(m).
(15) 400 6 Experiment methods Experiments described in this paper use the MSCOCO caption data set (Lin et al., 2015) and the Multi30k dataset (Elliott et al., 2016), which contains pairs of images and descriptions of images written by human annotators. Captions in the MSCOCO data set are in English, whereas captions in the Multi30k dataset are in English, German and French. Captions are automatically parsed (Kitaev and Klein, 2018) to generate a version of the reference set with constituency trees. 6 In addition to these datasets with captions generated by human annotators, we automatically translate the English captions into Chinese, Polish and Korean using Google Translate, 7 and parse the resulting translations into constituency trees, which are then used in experiments to probe the interactions between visual information and grammar induction. Results from models proposed in this paper -NoImagePCFG, ImagePrePCFG and ImagePCFG -are compared with published results from Shi et al. (2019), which include PRPN (Shen et al., 2018), ON-LSTM (Shen et al., 2019) as well as the grounded VG-NSL models which uses either head final bias (VG-NSL+H) or head final bias and Fasttext embeddings (VG-NSL+H+F) as inductive biases from external sources. All of these models only induce unlabeled structures and have been evaluated with unlabeled F1 scores. We additionally report the labeled evaluation score Recall-Homogeneity (Rosenberg and Hirschberg, 2007;Jin and Schuler, 2020) for better comparison between the proposed models. All evaluation is done on Viterbi parse trees of the test set from 5 different runs. Details about hyper-parameters and results on development data sets can be found in the appendix. However, importantly, the tuned hyperparameters for the grammar induction model are the same across the three proposed models, which facilitates direct comparisons among these models to determine the effect of visual information on induction.

Standard set: no replication of effect for visual information
Both unlabeled and labeled evaluation results are shown in Table 1 with left-and right-branching baselines. First, trees induced by the PCFG induction models are more accurate than trees induced 6 The multilingual parsing accuracy for all languages used in this work has been validated in with all other models, showing that the family of PCFG induction models is better at capturing syntactic regularities and provides a much stronger baseline for grammar induction. Second, using the NoImagePCFG model as a baseline, results from both the ImagePCFG model, where raw images are used as input, and the ImagePrePCFG model, where images encoded by pretrained image classifiers are used as input, do not show strong indication of benefits of visual information in induction. The baseline NoImagePCFG outperforms other models by significant margins on all languages in unlabeled evaluation. Compared to seemingly large gains between text-based models like PRPN and ON-LSTM 8 and the grounded models like VG-NSL+H on French and German observed by Shi et al. (2019), the only positive gain between NoIm-agePCFG and ImagePCFG shown in Table 1 is the labeled evaluation on French where ImagePCFG outperforms NoImagePCFG by a small margin. Because the only difference between NoImagePCFG and ImagePCFG models is whether the visual information influences the syntactic word embeddings, the results indicate that on these languages, visual information does not seem to help induction. The gain seen in previous results may therefore be from external inductive biases. Finally, the Im-agePrePCFG model performs at slightly lower accuracies than the ImagePCFG model consistently across different languages, datasets and evaluation metrics, showing that the information needed in grammar induction from images is not the same as information needed for image classification, and such information can be extracted from images without annotated image classification data.

Languages with wider distribution of high-frequency word types: positive effect
One potential advantage of using visual information in induction is to ground nouns and noun phrases. For example, if images like in Figure 1 are consistently presented to models with sentences describing spaghetti, the models may learn the categorize words and phrases which could be linked with objects in images as nominal units and then bootstrap other lexical categories. However, in the test languages above, a narrow set of very high fre- quency words such as determiners provide strong identifying information for nouns and noun phrases, which may greatly diminish the advantage contributed by visual information. In such cases, visual information may even be harmful, as models may attend to other information in images which is irrelevant to induction. Korean, Polish and Chinese are chosen as representatives of languages with no definite articles, and in which statistical information provided by high frequency words is less reliable because there are more such word types. Table 2 shows the performance scores of the three proposed systems on these languages. Comparing to results in Table 1, the models with visual information in the input significantly outperform the baseline model, NoImagePCFG, on a majority of the additional test datasets. Figure 3 shows the correlation between the RH difference between the ImagePCFG model and the NoImagePCFG model on each language in an image dataset, and the distribution of high frequency words in that language, defined as the number of word types needed to account for 10% of the number of word tokens in the Universal Dependency (Nivre et al., 2016) corpus of a language. 9 The figure shows that the largest gain brought by visual information in induction is on Korean, where the number of high frequency word types is also highest. Results on Chinese and Polish 9 Korean has 41, Chinese and Polish have 5, German has 4, English has 3 and French has 2.  Figure 3: The correlation between number of word types needed to account for 10% of word tokens in a language (log # High Freq Words) and the RH gain from NoImagePCFG to ImagePCFG on different languages on the two different image datasets. also show a benefit for visual information, although the gain is much smaller and less consistent. It also shows that when there is a trend of positive correlation between the number of high frequency words and the gain brought by visual information, factors other than high frequency words are at play as well in determining the final induction outcome for each dataset in each language in the visually grounded setup, which are left for investigation in future work.

Analysis of advantages of visual information
We hypothesize three specific ways that visual information may help grammar induction. First, a strong correlation between words and objects in images can help identification and categorization  of nouns and noun phrases, especially on languages where nouns and noun phrases are not readily identifiable by neighboring high frequency words. Second, visual information may provide bottom-up information for unknown word embeddings. Languages where neighboring words can reliably predict the grammatical category of an unknown word may build robust representations of unknown word embeddings, but the construction of the UNK embedding may also benefit from bottom-up information from images, especially when sentential context is not enough to build informative UNK embeddings. Finally, semantic information inside images may be helpful in solving syntactic ambiguities like prepositional phrase attachment in languages like English. Results from experiments described below with the ImagePCFG and NoIm-agePCFG models show evidence of all three ways.

Grounding of nouns and noun phrases
The 'Noun bias' hypothesis (Gentner, 1982) postulates that visual information in the induction process may impact how words are categorized grammatically, and nouns may receive an advantage because they correspond to objects in images. However, objects in images are often described with phrases, not single words. For example, captions like a red car is parked on the street, are common in both caption datasets, where the objects in the image may associate more strongly with modifier words like red than the head noun car.
Evaluations are carried out on the parsed sentences of all languages from two caption datasets using a part-of-speech homogeneity metric (Rosenberg and Hirschberg, 2007) for measuring the partof-speech accuracy, and an unlabeled NP recall score for measuring how many noun phrases in gold annotation are also found in the induced trees. Results in Figure 4 first show that the POS homogeneity scores from different models on the same induction dataset are extremely close to each other. Given that nouns are one of the categories with the most numerous tokens, the almost identical performance of POS homogeneity across different models indicates that the unsupervised clustering accuracy for nouns across different models is also very close, in contrast to substantial RH score differences on English and Korean.
However, NP recall scores show a pattern of performance ranking that resembles the ranking observed in Tables 1 and 2. For all datasets except for the Polish Multi30k dataset, when the RH score of ImagePCFG is higher than NoImagePCFG, the NP recall score for the ImagePCFG model is also higher. Significance testing with permutation sampling shows that all performance differences are significant (p < 0.01). 10 High accuracy on noun phrases is crucial to high accuracy of other constituents such as prepositional phrases and verb phrases, which usually contain noun phrases, and eventually leads to high overall accuracy. This result suggests that the benefit contributed by visual information works at phrasal levels, most likely 403 E n g l i s h K o r e a n P o l i s h C h i n e s e G e r m a n E n g l i s h F r e n c h K o r e a n grounding phrases, not words, with objects in images.

Informativeness of the UNK embedding
The informativeness of unknown word embeddings is tested among the induction models across different languages. An UNK test set is created by randomly replacing one word in one sentence with an UNK symbol if the sentence has no unknown words present. Table 3 shows the labeled evaluation results on the multilingual datasets. 11 First, performance on the UNK test sets on all languages is lower than on the normal test sets, showing that replacing random words with UNK symbols does impact performance. The performance ranking of the models on a majority of the languages is consistent with the ranking on the normal test set. The ranking of the models on one dataset, the Chinese Multi30k, is reversed on the UNK test set, where the ImagePCFG models show significantly higher performance than the NoImagePCFG models (Chinese: p < 0.01, permutation test on unlabeled F1). This result indicates that the ImagePCFG model in which visual information is supplied during train-ing may have built more informative embeddings for the unknown word symbols, helping the model to outperform the model without visual information on a majority of datasets where UNK symbols are frequent.

Prepositional phrase attachment
Finally, visual information may provide semantic information to resolve structural ambiguities. Word quintuples such as (a) hotel caught fire during (a) storm were extracted from English Wikipedia and the attachment locations were automatically labeled either as 'n' for low attachment, where the prepositional phrase adjoins the direct object, or 'v' for high attachment, where the prepositional phrase adjoins the main verb (Nakashole and Mitchell, 2015). 168 test items are selected by human annotators for evaluation, within which 119 are sentences with high attached PPs and 49 are with low attached PPs. For evaluation of PP attachment with induced trees, one test item is labeled correct when the induced tree puts the main verb and the direct object into one constituent and it is labeled as 'v'. For example, if the induced tree has caught fire as a constituent, it counts as correct for the above example with high attachment. Low attachment trees must have a constituent with the direct object and the prepositional phrase. For example, for the sentence (a) guide gives talks about animals, the induced tree must have talks about animals. Average accuracies for all sentences as well as for sentences with high attachment or low attachment with induced grammars are shown in Figure 5. Results show that the models trained with visual information on both datasets show significantly higher performance on the PP attachment task in most of the categories, except for the low attachment category with Multi30k models where the performance from both models is not significantly different. This is in contrast to the  higher performance of the NoImagePCFG models on unlabeled F1 and labeled RH than that of the ImagePCFG models on English from both caption datasets. Results indicate that induction models use visual information for weighting competing latent syntactic trees for a sentence, which is consistent with the third hypothesized advantage of visual information for induction. This also indicates that the reason that the overall parsing performance of Im-agePCFG on English is lower than NoImagePCFG lies within other syntactic structures, which is left for future work.

Conclusion
This work proposed several novel neural networkbased models of grammar induction which take into account visual information in induction. These models achieve state-of-the-art results on multilingual induction datasets without any help from linguistic knowledge or pretrained image encoders. Further analyses isolated three hypothesized benefits of visual information: it helps categorize noun phrases, represent unknown words and resolve syntactic ambiguities.

A Details of datasets
The MSCOCO caption dataset used in Shi et al. (2019) contains 413,915 sentences in the training set, and 5000 sentences in the development and test sets respectively. 12 Every image is accompanied by 5 captions,and there are 82,783 images in total in the training set. The image embeddings of size 2048 used in Shi et al. (2019) are encoded by an image classifier with ResNet128 architecture trained with on the ImageNet classification task (Deng et al., 2009). The Multi30k caption dataset contains 29,000 sentences in the training set, and 1,014 sentences in the development and 1,000 in the test set in four different languages, all of which except Czech are used in this work thanks to the availability of high accuracy constituency parsers in these languages. 13 There are as many images as there are captions in the training set. The image embeddings of size 2048 provided with the dataset are encoded by an image classifier with ResNet50 architecture also trained with on the ImageNet classification task.
For data preprocessing, following Shi et al. (2019), the size of the vocabulary is limited to 10,000 for all languages and datasets. All raw images are resized to 3 × 64 × 64 and normalized with means [0.485, 0.456, 0.406] and standard deviations [0.229, 0.224, 0.225], calculated from images in ImageNet.

B Hyperparameters
The hyperparameters used in all proposed models are tuned with the MSCOCO English development set. For the grammar induction model, the size of word and syntactic category embeddings, as well as the size of hidden intermediary representations is 64. The size of the image embedding in the ImagePCFG system is also 64. All out-ofvocabulary words are replaced by the UNK symbol. Sentences with more than 40 words in the training set are trimmed down to 40 words. For the projector model, five different convolutional kernels, from (1,64) to (5,64), are used with 128 output channels. The trainable image encoder employs a 12 The data set can be found at https://github.com/ ExplorerFreda/VGNSL along with image embeddings encoded by pretrained image encoders. 13 The data set can be found at https://github.com/ multi30k/dataset along with image embeddings encoded by pretrained image encoders.
ResNet18 architecture, 14 and the decoder employs the decoder architecture in the DCGAN model. 15 A batch size of 2 is used in training. Adam is used as the optimizer, with the initial learning rate at 5 × 10 −4 . The loss on the validation set is checked every 20000 batches, and training is stopped when the validation loss has not been lowered for 10 checkpoints. The model with the lowest validation loss is used as the candidate model for test evaluation, where best parses are generated with the Viterbi algorithm on an inside chart.