\section{Introduction}
Motivated by the success of the \emph{pretrain-then-finetune} paradigm of BERT in natural language understanding \cite{devlin2019bert}, there has been an increasing interest in developing vision-and-language pretraining (VLP) models \cite{lu2019vilbert,tan2019lxmert,chen2020uniter,li2020oscar} to address a wide range of vision-and-language (V+L) tasks. In particular, these approaches first pretrain transformer-based models on large image-text corpus to learn task-agnostic representations, and then finetune the models on downstream V+L tasks, \emph{e.g.}, visual question answering \cite{yu2017mfb,yu2018beyond}, image text retrieval \cite{plummer2015flickr30k,lee2018stacked}, and referring expression comprehension \cite{kazemzadeh2014referitgame,yu2018rethining}. Compared to earlier methods that are only adapted to one V+L task \cite{yu2019deep,yu2018mattnet,yu2020deep}, VLP models is generalizable across multiple tasks and also achieves significantly better performance on respective tasks.

\begin{figure}
\begin{center}
\includegraphics[width=0.47\textwidth]{fig1.pdf}
\vspace{-5pt}
\caption{Schematic of the knowledge integration strategies of three VLP methods, \emph{i.e.}, OSCAR \cite{li2020oscar}, ERNIE-ViL \cite{yu2020ernievil}, and our ROSITA. OSCAR and ERNIE-ViL only exploit the intra-modal knowledge from the image and text modalities, respectively. In contrast, ROSITA simultaneously encodes the cross-modal knowledge (pink line) and intra-modal knowledge (blue line) in a unified scene graph centered at specific anchor objects, which is used to enhance the learning of fine-grained semantic alignments across modalities.}
\label{fig:example}
\end{center}
\vspace{-10pt}
\end{figure}

Learning \emph{fine-grained} semantic alignments between image regions and text words plays a key role in V+L tasks. However, manually annotating such dense alignment between regions and words is expensive and is unrealistic under the large-scale scenario. Therefore, most existing VLP approaches \cite{chen2020uniter,lu2019vilbert,li2019visualbert} use a weakly-supervised learning strategy to model the alignments implicitly. Taking the image regions and text words as inputs, they adopt multi-layer Transformers \cite{vaswani2017attention} as their backbones to learn \emph{fine-grained} semantic alignments from \emph{coarse-grained} image-text matching supervision. Moreover, the interference within and across modalities makes the learning of semantic alignments even more challenging.

To facilitate the learning of semantic alignments, two recent VLP approaches OSCAR \cite{li2020oscar} and ERNIE-ViL \cite{yu2020ernievil} introduce extra knowledge in different ways. Specifically, OSCAR additionally extracts the predicted region tags from images and uses these tags as anchor points to align with text words implicitly. ERNIE-ViL explicitly constructs a scene graph from text and puts more emphasis on the keywords (\emph{e.g.}, objects along with their attributes and relations) in the scene graph in its pretraining objectives. In terms of  knowledge source, both of them use the \emph{intra-modal} knowledge from a single modality to enhance the semantic alignments: OSCAR models the intra-modal knowledge in the image modality while ERNIE-ViL models the intra-modal knowledge in the text modality. The success of the two methods above raises a question: \emph{Is it possible to utilize the intra-modal knowledge from both modalities along with the cross-modal knowledge to further enhance the semantic alignments?}

In this paper, we present a new VLP method called ROSITA, which encodes the c\textbf{\underline{ROS}}s- and \textbf{\underline{I}}n\textbf{\underline{T}}r\textbf{\underline{A}}-modal knowledge simultaneously in a unified scene graph. As shown in Figure \ref{fig:example}, the graph consists of a set of knowledge entries, where each entry corresponds to an \emph{anchor object} along with its associated cross- and intra-modal knowledge. The intra-modal knowledge refers to the relationships between the anchor object and its intra-modal contexts (\emph{e.g.}, spatially related regions or contextually related words). The cross-modal knowledge corresponds to the relationships between the anchor object and its semantically similar objects from the opposite modality (\emph{e.g.}, the region predicted as  ``\emph{grass}'' and the word ``\emph{steppe}'').

Although we have obtained a set of knowledge entries, how to effectively use them to enhance semantic alignments is still nontrivial. We propose a novel \emph{structural knowledge masking} (SKM) strategy that can be seamlessly integrated with the masked language (region) modeling tasks, which are commonly used in existing VLP methods \cite{chen2020uniter,lu2019vilbert}. In principle, SKM determinately masks the anchor object while selectively masking its cross- and intra-modal contexts in a knowledge entry. This strategy effectively eliminates the interference information within and across modalities and enhances the semantic alignments by enforcing the model to acquire accurate information from the \emph{opposite} modality.

The contributions of this work are three-fold:
\begin{enumerate}
  \item We present a new VLP method ROSITA, which incorporates cross- and intra-modal knowledge simultaneously to enhance the semantic alignments across different modalities.
  \item We introduce a novel structural knowledge masking strategy to use the scene graph structure as a priori to be integrated with the commonly used masked language (region) modeling tasks in existing VLP methods.
  \item We achieve the best results on three typical V+L tasks over six benchmark datasets, outperforming existing state-of-the-art VLP methods.

\end{enumerate}

\section{Related work}
We briefly review previous studies on unimodal pretraining and vision-and-language pretraining, especially those studies on knowledge enhanced pretraining.
\vspace{5pt}
\\
\noindent\textbf{Unimodal Pretraining.} The pretraining technique has been widely used in computer vision (CV) tasks. Deep convolutional neural networks like VGGNet \cite{simonyan2014very} or ResNet \cite{he2016deep} pretrained on ImageNet can well generalize to various downstream tasks  \cite{ren2015faster,long2015fully,he2017mask}. In contrast to CV tasks, the popularization of pretraining in the natural language processing (NLP) community is relatively late. Based on the multi-layer Transformer architecture \cite{vaswani2017attention}, many famous pretraining approaches (\emph{e.g.}, BERT \cite{devlin2019bert}, GPT \cite{radford2018improving}, and XLNet \cite{yang2019xlnet}) have been put forward. Different from the supervised pretraining paradigm in CV tasks, the pretraining paradigm in NLP tasks is \emph{self-supervised} that aims to train a model to predict words based on their contexts without introducing human annotations. In particular, BERT introduces a novel masking language modeling (MLM) task that randomly masks the input words and predicts these masked words based on their contexts. This MLM strategy is naturally inherited by the VLP methods.
\vspace{5pt}
\\
\noindent\textbf{Vision-and-Language Pretraining (VLP).} Different from the purely self-supervised paradigm in NLP tasks, VLP models are pretrained on large-scale paired image-text corpus, \emph{e.g.},  image captioning datasets like \cite{chen2015microsoft,sharma2018conceptual,ordonez2011im2text}. Mirroring the success of BERT, recent studies naturally extend its framework to the vision-and-language domain to pretrain VLP models for a wide range of V+L tasks \cite{chen2020uniter,li2020oscar,zhang2020devlbert,lu2019vilbert,tan2019lxmert,yu2020ernievil,huang2020pixel}. ViLBERT \cite{lu2019vilbert} and LXMERT \cite{tan2019lxmert} are two pioneering works in this field, where the two-stream architectures are adopted to encode the image features and textual features with two separate Transformers and then perform multimodal fusion via a third Transformer. Recent works tend to use the single-stream architectures, where the multimodal features are directly fused using one Transformer \cite{chen2020uniter,li2019visualbert,li2020unicoder,su2019vl}. Moreover, other techniques like knowledge integration \cite{yu2020ernievil,li2020oscar}, multilingual enhancement \cite{zhou2021uc2}, contrastive learning \cite{li2020unimo}, and adversarial training \cite{gan2020large} are introduced to further improve the performance of the pretrained models.
\vspace{5pt}
\\
\noindent\textbf{Knowledge-Enhanced Pretraining.} Incorporating prior knowledge (\emph{e.g.}, external knowledge graph) to enhance model pretraining has been investigated earlier by two ERNIE methods \cite{sun2019ernie,zhang2019ernie} and widely explored in recent years \cite{liu2020kbert,wang2021kepler,wang2020k}. The introduced prior knowledge enables the model to better understand the syntactic and semantic structure of the text, thus facilitating model pretraining by an improved structural MLM task. In the VLP task, prior knowledge can be acquired from both the image and text modalities. ERNIE-ViL constructs a scene graph from text and puts more emphasis on the discovered keywords \cite{yu2020ernievil}. OSCAR exploits the predicted tags of image regions to enhance the semantic alignment across the two modalities \cite{li2020oscar}. A concurrent work UC2 utilizes off-the-shelf machine translation model to construct aligned multilingual dataset for texts and regard this extra information as prior knowledge to enhance the learning of cross-modal semantic alignment \cite{zhou2021uc2}. Despite the success of these knowledge-enhanced VLP methods, they only utilize the
intra-modal knowledge from a single modality, which restricts their effectiveness in learning semantic alignments.

To the best of our knowledge, our ROSITA is the first VLP method to integrate the cross-modal and intra-modal knowledge simultaneously in order to enhance the learning of semantic alignments across different modalities.
\begin{figure*}
\begin{center}
\includegraphics[width=0.98\textwidth]{fig2.pdf}
\vspace{-5pt}
\caption{The flowchart of knowledge extraction given an image-text pair. It consists of two main stages, namely the unified scene graph construction and knowledge representation.}
\label{fig:rosita}
\end{center}
\vspace{-10pt}
\end{figure*}
\vspace{5pt}
\section{Knowledge Extraction}
In this section, we introduce the procedure of extracting knowledge entries from an image-text pair. We first construct a unified graph to model the intra- and cross-modal knowledge from an image-text pair. On top of the established graph, we select anchor objects to obtain a set of knowledge entries. The process of knowledge extraction is illustrated in Figure \ref{fig:rosita}.
\vspace{5pt}
\\
\noindent\textbf{Unified Scene Graph Construction.}
Given an image-text pair, we resort to a unified scene graph structure $G=<V, E, S>$ to encode its intra- and cross-modal knowledge simultaneously \cite{yu2014discriminative}. The vertex set $V$ includes the words and regions from the text and image, respectively. The edge set $E$ and similarity set $S$ contain pairwise relationships and their corresponding similarities between vertices (\emph{i.e.}, edge weights), respectively.

The intra-modal knowledge within the image and text are first represented as an image scene graph and a text scene graph, respectively. For the image scene graph, regions extracted from a pretrained object detector are considered as the vertices in $V$. Inspired by \cite{li2019relation,yao2018exploring,kant2020spatially}, we calculate the similarity between each paired regions by their Intersection over Union (IoU) score. The region pairs with IoU scores larger than zero are considered to have edges in $E$ and their IoU scores are regarded as their similarities in $S$. For the text graph, we use an off-the-shelf scene graph parser provided by \cite{anderson2016spice} to obtain a text scene graph from a text. The text scene graph explicitly encodes the keywords of objects, attributes, and relations found in the text while discarding the rest of uninformative words. These mentioned keywords in the scene graph are regarded as the vertices in $V$. The word-word relationships in the scene graph (\emph{i.e.}, object-attribute or object-relation) correspond to the edges in $E$. The similarity between two vertices is the co-occurrence frequencies of the referred object-attribute (or object-relation) pair calculated on the whole dataset. Since the similarity distributions of the image and text modalities may vary widely, we normalize the similarities within each modality, respectively.

As we have modeled the intra-modal knowledge in the graph, we further integrate cross-modal knowledge to align the image regions to their semantically related words. Since such cross-modal alignment supervision is not available, we establish \emph{pseudo} semantic alignments between region-word pairs as follows. For the image regions, the predicted region tags are aligned to the object words with respect to their semantic similarities on words. We adopt a pretrained word embedding model \cite{pennington2014glove} to calculate the pairwise similarities between object tags and object words\footnote{We have tried to establish more fine-grained alignments to include the attribute words. However, the predicted attributes from image regions are too diverse that often fail to match the attribute words in the text.}. We set a minimum confidence threshold of 0.5 to the similarity scores to make a trade-off between precision and recall. The region-word pairs surpass the threshold will form cross-modal edges in $E$ and their corresponding scores represent the similarities in $S$.
\vspace{5pt}
\\
\noindent\textbf{Knowledge Representation.} Based on the constructed unified scene graph $G$, we illustrate the procedure of extracting knowledge entries from the scene graph in detail. Note that each knowledge entry is associated with an anchor object, we first select all possible anchor objects from the graph. We define an anchor object as the vertex (an image region or a text word) in the graph that is referred to by at least one cross-modal edge. Since the attribute and relation words are not directly connected to any image region, they cannot be anchor objects according to our definition.

After obtaining the anchor objects, we integrate the intra-modal knowledge and cross-modal knowledge in $G$ to obtain a knowledge entry. Given an anchor object $v\in V$, its corresponding knowledge entry is represented as a subgraph $g(v) \subseteq G$ and is obtained by the union of three subgraphs of $G$ as follows:
\begin{equation}\label{eq:rosita}
g(v)=G_\mathrm{cross}(v)\cup G_\mathrm{intra}(v)\cup G_\mathrm{intra}(G_\mathrm{cross}(v))
\end{equation}
where $G_\mathrm{cross}(v)$ contains the relationships between $v$ and its {directly connected} contexts by cross-modal edges. $G_\mathrm{intra}(v)$ models the relationships between $v$ and its directly connected contexts by intra-modal edges. $G_\mathrm{intra}(G_\mathrm{cross}(v))$ includes the relationships between the vertices in $G_\mathrm{cross}(v)$ and their corresponding intra-modal contexts. It is worth noting that the anchor object $v$ can reach every vertex in $g(v)$ within two hops.

\section{The ROSITA Framework}
Based on the extracted knowledge entries from image-text pairs, we introduce the ROSITA framework in this section. We first describe the image and text feature representations and the network architecture. Then, we introduce a structural knowledge masking (SKM) strategy,
which takes the knowledge entries as a priori to perform the masked language (region) modeling. Finally, we describe the whole pretraining objective with multi-task learning. The overall framework is illustrated in Figure \ref{fig:framework}.
\begin{figure}
\begin{center}
\includegraphics[width=0.98\columnwidth]{fig3.pdf}
\vspace{-5pt}
\caption{The flowchart of our ROSITA framework with the structural knowledge masking (SKM) strategy.}
\label{fig:framework}
\end{center}
\vspace{-10pt}
\end{figure}
\\
\noindent\textbf{Image and Text Feature Representations.} Following the commonly used strategy in existing VLP methods \cite{li2020oscar,chen2015microsoft}, the input image is represented as a set of regional features, which are extracted from a Faster R-CNN model pretrained on Visual Genome \cite{anderson2018bottom}. More specifically, we extract $m$ regions with the highest confidence probabilities from the image. For the $i$-th region, it is represented as a visual feature $f_i\in\mathbb{R}^{2048}$ and a positional feature $p_i\in\mathbb{R}^{5}$ \cite{yu2020ernievil}. The two features are fused into a $d$-dimensional image representation $x_i\in\mathbb{R}^d$ using two linear projections as follows:
\begin{equation}\label{eq:img_rep}
x_i = W_f^Tf_i + W_p^Tp_i
\end{equation}
where $W_f\in\mathbb{R}^{2048 \times d}$ and $W_p\in\mathbb{R}^{5 \times d}$. Finally, the image is represented as a feature matrix $X\in\mathbb{R}^{m\times d}$.

For its paired text, we adopt the word processing method similar to \cite{devlin2019bert}. The input text is first tokenized into words and trimmed (or padded) to a maximum of $n$ words. Each word $w_i$ and its index $i$ (\emph{i.e.}, the absolute position of $w_i$ in the text) are projected to vectors by two individual embedding layers, then added to obtain the position-aware text representation $y_i$ as follows:
\begin{equation}\label{eq:txt_rep}
y_i = \mathrm{WordEmbed}(w_i) + \mathrm{IdxEmbed}(i)
\end{equation}
where $y_i$ is $d$-dimensional to match the image representation. The text is finally represented as a feature matrix  $Y\in\mathbb{R}^{n\times d}$.
\vspace{5pt}
\\
\noindent\textbf{Network Architecture.} The image features $X=[x_1,...,x_m]$ and text features $Y=[y_1,...,y_n]$ are first concatenated before feeding to the network. We insert two special tokens to the concatenated features to obtain the multimodal input features $Z$:
\begin{equation}\label{eq:it_rep}
Z=[x_1,x_2,...,x_m, [\texttt{SEP}], y_1,y_2,...,y_n, [\texttt{CLS}]]
\end{equation}
where the \texttt{[SEP]} token marks the boundary between the image and text features. The \texttt{[CLS]} token is used to predict whether the given image and text are paired or not.

The multimodal features $Z$ are fed into a single-stream Transformer with $L$ layers \cite{devlin2019bert}. Each layer consists of a multi-head self-attention (MSA) block and a feed-forward networks (FFN) block.
\begin{equation}
\begin{aligned}\label{eq:transformer}
\hat{Z}^{\ell} &= \mathrm{LN}(\mathrm{MSA}(Z^{\ell-1})+Z^{\ell-1}),~~~&\ell=1,2,...,L \\
Z^{\ell} &= \mathrm{LN}(\mathrm{FFN}(\hat{Z}^{\ell})+\hat{Z}^{\ell}),~~~&\ell=1,2,...,L \\
\end{aligned}
\end{equation}
where $Z^0=Z$. Layer normalization \cite{ba2016layer} and residual connection \cite{he2016deep} are applied after every block, respectively.
\vspace{5pt}
\\
\noindent\textbf{Structural Knowledge Masking.} The masked language modeling (MLM) \cite{devlin2019bert} and masked region modeling (MRM) \cite{chen2020uniter} tasks are commonly used in almost all the VLP methods \cite{lu2019vilbert,chen2020uniter,yu2020ernievil}. They randomly mask the input tokens (\emph{i.e.}, words or regions) and predict these masked tokens based on their contextual tokens. Since the random masking based MLM and MRM tasks are not aware of the keywords and key regions to be aligned, their efficacy in the alignment learning is weak. To this end, we present an alternative \emph{structural knowledge masking} (SKM) strategy to selectively mask the tokens referred to by the extracted knowledge entry. Accordingly, the MLM and MRM tasks are respectively modified to the SKMLM and SKMRM tasks to adapt to the SKM strategy.

Let an image be represented as a set of regions $\mathcal{R}=\{{r}_1,...,{r}_m\}$ and a text be represented as a sequence of words $\mathcal{W}=\{{w}_1,...,{w}_n\}$, we construct a unified scene graph $G$ on top of $\mathcal{R}$ and $\mathcal{W}$, and extract a set of knowledge entries from $G$. Let $g(v_i)=<\hat{V}, \hat{E}, \hat{S}>$ be one of the knowledge entries, where $v_i\in \hat{V}$ is the anchor object. The vertices are represented as $\hat{V}=\{v_1,...,v_N\}$ and the similarities between the vertices are represented as $\hat{S}\in\mathbb{R}^{N\times N}$, where $N$ is the number of vertices in this entry.

The strategy of SKM is to determinately mask the anchor object $v_i$ while probabilistically masking its intra-modal contexts and cross-modal contexts with respect to the graph structure of the knowledge entry. Since the similarities between $v_i$ and its contexts are different, we assign \emph{independent} masking probabilities to each of the contexts with respect to their similarities to $v_i$, rather than simply using an \emph{identical} masking probability for all the contexts. To obtain the masking probabilities for the contexts, we introduce a masking strategy that satisfies the following principles: for the intra-modal contexts, a larger similarity score refers to a higher masking probability. For the cross-modal contexts, a larger score leads to a lower masking probability. The reasons behind this masking strategy will be explained hereinafter.

Note that not all contexts have direct connections to the anchor object. Therefore, we calculate the transmission probabilities $T=[t_1,...,t_N]\in[0,1]^N$ from the anchor object $v_i$ to its contexts in $\hat{V}$ based on the normalized similarities defined in $\hat{S}$. Since $v_i$ can reach all the vertices in $\hat{V}$ within two hops, $T$ is defined as follows:
\begin{equation}\label{eq:trans_prop}
T=\frac{1}{2}\hat{S}\pi(i)+\frac{1}{2}\hat{S}\hat{S}\pi(i)
\end{equation}
where $\pi(i)\in\{0,1\}^N$ is a one-hot vector with the $i$-th element to be 1. The two terms correspond to the transmission probabilities between the anchor object $v_i$ and its one-hop and two-hop contexts, respectively. After that, we convert the transmission probabilities $T$ to the masking probabilities $P=[p_1,...,p_N]\in[0,1]^N$ using the following rules to satisfy our masking strategy above:
\begin{equation}\label{eq:maskprob}
p_j =
\begin{cases}
    1, & \text{if } v_j \text{ is the anchor object}\\
    \alpha t_j, & \text{if } v_j \text{ is within the intra-modal contexts}\\
    (1-\alpha) (1-t_j), & \text{if } v_j \text{ is within the cross-modal contexts}
\end{cases}
\end{equation}
where $\alpha$ is a hyper-parameter to balance the masking probabilities of the intra-modal and cross-modal contexts. $t_j$ and $p_j$ denote the transmission and masking probability of the vertex $v_j$, respectively.

Given a knowledge entry, we use the calculated masking probabilities to obtain two groups of mask indices $M_w$ and $M_r$, indicating the words and regions to be masked, respectively. The partially masked input features are passed through the network and then fed into the SKMLM and SKMRM tasks.

In particular, if the anchor point $v_i$ refers to an object word in the text, we resort to the SKMLM task to reconstruct the masked words $\mathcal{W}_{{M}_w}$ as follows:
\begin{equation}\label{eq:skmlm}
\mathcal{L}_\mathrm{SKMLM}(\theta) = -\mathbb{E}_{(\mathcal{W}, \mathcal{R})\sim D}~\mathrm{log}~P_\theta(\mathcal{W}_{{M}_w}|\mathcal{W}_{\backslash{M}_w},\mathcal{R}_{\backslash{M}_r})
\end{equation}
where $\theta$ is the trainable parameters. Each pair $(\mathcal{W}, \mathcal{R})$ is sampled from the whole training set $D$. $\mathcal{W}_{\backslash{M}_w}$ and $\mathcal{R}_{\backslash{M}_r}$ refer to the remaining words in $\mathcal{W}$ and the remaining regions in $\mathcal{R}$ with excluding the masked tokens from their modalities, respectively.

Analogously, if $v_i$ refers to a region in the image, we resort to the SKMRM task to reconstruct the masked regions $\mathcal{R}_{{M}_r}$ as follows:
\begin{equation}\label{eq:mrm}
\mathcal{L}_\mathrm{SKMRM}(\theta) = -\mathbb{E}_{(\mathcal{W}, \mathcal{R})\sim D}~f_\theta(\mathcal{R}_{{M}_r}|\mathcal{W}_{\backslash{M}_w},\mathcal{R}_{\backslash{M}_r})
\end{equation}
where $f_{\theta}(\cdot)$ refers to some loss functions. Similar to \cite{chen2020uniter}, we use the regression-based loss and classification-based loss jointly.

The motivations of SKM can be explained as follows: (\emph{i}) The intra-modal contexts may contain interference information (\emph{e.g.}, the word ``\emph{sky}'' is frequently associated with an attribute ``\emph{blue}'' and a visual object of ``\emph{wheel}'' is usually within an object of ``\emph{car}''). Such interference information may leak out the semantics of the anchor object and reduce the difficulty of anchor object reconstruction, leading to a degradation of the pretrained model. Therefore, when masking an anchor object, its intra-modal contexts with high similarities will have high probabilities to be masked simultaneously. This operation reduces the risk of information leakage and enforces the model to acquire precise information from the opposite modality, which \emph{implicitly} enhance the semantic alignments; (\emph{ii}) The cross-modal contexts  with low similarities may contain irrelevant or noisy information. Therefore, when masking an anchor object, its cross-modal contexts with high similarities will have low probabilities to be masked at the same time, which \emph{explicitly} excludes potential noise thus benefiting the semantic alignments. As a result, the synergy of the masking operations above significantly facilitates the semantic alignments.
\vspace{5pt}
\\
\noindent\textbf{Multi-task Learning.} Similar to \cite{chen2020uniter}, we adopt a multi-task learning objective to pretrain our model. Besides the proposed SKMLM and SKMRM tasks, we also include the image-text matching (ITM) task. Moreover, since the SKMLM and SKMRM tasks only focus on the key tokens included in the knowledge entry, we still retain the original random masking-based MLM and MRM tasks to guarantee a good coverage of the remaining tokens in the image and text\footnote{We have made such an experiment that removes the MRM \& MLM tasks. The resulting model reports slight performance drop ($\sim$0.3 points) on the downstream tasks.}.

\section{Experiments}
We evaluate ROSITA on three V+L tasks and perform thorough comparative analysis to the state-of-the-art VLP methods on six datasets. Furthermore, we conduct comprehensive ablation experiments to explore its effectiveness in learning fine-grained semantic alignments.

\subsection{Pretraining Setup}
\noindent\textbf{Datasets.} Following the strategy in \cite{chen2020uniter}, we construct the pretraining dataset consisting of 9.5M train and 155K validation image-text pairs from four existing datasets, namely the COCO Captions \cite{chen2015microsoft}, Visual Genome Captions\cite{krishna2017visual}, Conceptual Captions \cite{sharma2018conceptual}, and SBU Captions \cite{ordonez2011im2text}. The four datasets are categorized into the \emph{in-domain} and \emph{out-of-domain} datasets based on whether they share the same images with the downstream tasks. The statistics of the pretraining dataset are shown in Table \ref{table:corpus}.
\vspace{5pt}
\\
\noindent\textbf{Implementation Details.} For the input image-text pairs, we extract a fixed number of 36 region features from a pre-trained Faster R-CNN model \cite{anderson2018bottom} and adopt the BPE strategy to tokenize the sentence into a maximum of 50 words following \cite{devlin2019bert}. Our ROSITA model adopts a 12-layer Transformer encoder architecture with 768 hidden units and 12 attention heads. The hyper-parameter $\alpha$ in Eq.(\ref{eq:maskprob}) is set to 0.9. The masking probabilities in the original MRM and MLM tasks are set to 15\% \cite{yu2020ernievil}. The model is initialized with the parameters from a pretrained BERT-base model \cite{devlin2019bert}, and then trained up to 40 epochs with a batch size of 512.

\begin{table}
\centering
\caption{The detailed statistics of the used datasets. Following the strategies in \cite{chen2015microsoft}, we split them into in-domain and out-of-domain splits based on the image sources. Each cell shows the number of image-text pairs.}
\label{table:corpus}
\begin{tabular}{c|cc|cc|c}
\toprule
    &  \multicolumn{2}{c|}{in-domain} & \multicolumn{2}{c|}{out-of-domain} &  \multirow{3}{*}{total}\\
\cmidrule{2-5}& \makecell{COCO\cite{chen2015microsoft}} & \makecell{VG\cite{krishna2017visual}} & \makecell {CC\cite{sharma2018conceptual}} & \makecell{SBU \cite{ordonez2011im2text}} &  \\
\midrule
    train & 533K& 5.1M& 3.0M & 869K& 9.5M \\
    val & 25K& 106K& 14K& 10K& 155K\\
\bottomrule
\end{tabular}
\vspace{-5pt}
\end{table}

\begin{table*}
\centering
\caption{Results on \textbf{downstream V+L tasks} to compare with the state-of-the-art VLP methods. For a fair comparison, all the results are archived by the base models. Most of the models are trained on the \emph{in-domain}+\emph{out-of-domain} datasets, except for those models marked with $\dag$ are trained on the \emph{out-of-domain} datasets. IR and TR denote the image retrieval and text Retrieval, respectively. For the REC task, all the results are achieved based on the detected region features from images. Dark and light grey colors highlight the top and second best results on each evaluation metric.}\label{table:mainres}
\vspace{-5pt}
\begin{tabular}{c|cl|cccccccc|c}
\toprule
task
 & \multicolumn{2}{c|}{dataset}
 &  \makecell{ViLBERT$^\dag$\\\cite{lu2019vilbert}}
 & \makecell{VLBERT$^\dag$\\\cite{su2019vl}}
 & \makecell{Unicoder-VL\\\cite{li2020unicoder}}
 & \makecell{LXMERT\\\cite{tan2019lxmert}}
 & \makecell{UNITER\\\cite{chen2020uniter}}
 & \makecell{ERNIE-ViL$^\dag$\\\cite{yu2020ernievil}}
 & \makecell{VILLA\\\cite{gan2020large}}
 & \makecell{OSCAR\\\cite{li2020oscar}}
 & \makecell{ROSITA\\(ours)}\\
\midrule
\multirow{2}{*}{VQA}& \multirow{2}{*}{VQAv2} & test-dev & 70.55 & 71.16 & - & 72.42 & 72.70 & 72.62 & \multicolumn{1}{>{\columncolor{mygray1}}c}{73.59} & 73.16 &  \multicolumn{1}{>{\columncolor{mygray2}}c}{73.91} \\
 & & test-std & 70.92 & - & - & 72.54 & 72.91 & 72.85 & \multicolumn{1}{>{\columncolor{mygray1}}c}{73.67} & 73.44 &  \multicolumn{1}{>{\columncolor{mygray2}}c}{73.97}\\
    \midrule
\multirow{8}{*}{REC}&  \multirow{3}{*}{\makecell{Ref-\\COCO}}  & val$^d$ & - & - & - & - & 81.24 & - & \multicolumn{1}{>{\columncolor{mygray1}}c}{81.65} & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{84.79}\\
&  & testA$^d$ & - & - & - & - & 86.48 & - & \multicolumn{1}{>{\columncolor{mygray1}}c}{87.40} & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{87.99}\\
&  & testB$^d$ & - & - & - & - & 73.94 & - & \multicolumn{1}{>{\columncolor{mygray1}}c}{74.48}& -& \multicolumn{1}{>{\columncolor{mygray2}}c}{78.28}\\
    \cmidrule{2-12}
& \multirow{3}{*}{\makecell{Ref-\\COCO+}}  & val$^d$ & 72.34 & 71.60 & - & - & 75.31 & {74.02} & \multicolumn{1}{>{\columncolor{mygray1}}c}{76.05} & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{76.06}\\
&  & testA$^d$ & 78.52 & 77.72 & - & - & 81.30 & {80.33} & \multicolumn{1}{>{\columncolor{mygray1}}c}{81.65} & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{82.01}\\
&  & testB$^d$ & 62.61 & 60.99 & - & - & 65.68 & {64.74} & \multicolumn{1}{>{\columncolor{mygray1}}c}{65.70} & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{67.40}\\
   \cmidrule{2-12}
& \multirow{2}{*}{\makecell{Ref-\\COCOg}}  & val$^d$ & - & - & - & - & 74.31 & - & \multicolumn{1}{>{\columncolor{mygray1}}c}{75.90} & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{78.23}\\
&  & test$^d$ & - & - & - & - & 74.51 & - & \multicolumn{1}{>{\columncolor{mygray1}}c}{75.93} & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{78.25}\\
   \midrule
\multirow{12}{*}{ITR}& \multirow{3}{*}{\makecell{IR-\\COCO}} & R@1 & - & - & 46.70 & - & 50.33 & - & - & \multicolumn{1}{>{\columncolor{mygray1}}c|}{54.00} & \multicolumn{1}{>{\columncolor{mygray2}}c}{54.40}\\
&  & R@5 & - & - & 76.00 & - & 78.52 & - & - & \multicolumn{1}{>{\columncolor{mygray1}}c|}{80.80} & \multicolumn{1}{>{\columncolor{mygray2}}c}{80.92}\\
&  & R@10 & - & - & 85.30 & - & 87.16 & - & - & \multicolumn{1}{>{\columncolor{mygray1}}c|}{88.50} & \multicolumn{1}{>{\columncolor{mygray2}}c}{88.60}\\
    \cmidrule{2-12}
& \multirow{3}{*}{\makecell{TR-\\COCO}} & R@1 & - & - & 62.30 & - & 64.40 & - & - & \multicolumn{1}{>{\columncolor{mygray1}}c|}{70.00} & \multicolumn{1}{>{\columncolor{mygray2}}c}{71.26}\\
&  & R@5 & - & - & 87.10 & - & 87.40 & - & - & \multicolumn{1}{>{\columncolor{mygray1}}c|}{91.10} & \multicolumn{1}{>{\columncolor{mygray2}}c}{91.62}\\
&  & R@10 & - & - & 92.80 & - & 93.08 & - & - & \multicolumn{1}{>{\columncolor{mygray1}}c|}{95.50} & \multicolumn{1}{>{\columncolor{mygray2}}c}{95.58}\\
   \cmidrule{2-12}
& \multirow{3}{*}{\makecell{IR-\\Flickr}}  & R@1 & 58.20 & - & 71.50 & - & 72.52 & \multicolumn{1}{>{\columncolor{mygray2}}c}{74.44} & \multicolumn{1}{>{\columncolor{mygray1}}c}{74.74} & - & {74.08}\\
&  & R@5 & 84.90 & - & 90.90 & - & 92.36 & \multicolumn{1}{>{\columncolor{mygray1}}c}{92.72} & \multicolumn{1}{>{\columncolor{mygray2}}c}{92.86} & - & {92.44}\\
&  & R@10 & 91.52 & - & 94.90 & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{96.08} & 95.94 & 95.82 & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{96.08}\\
    \cmidrule{2-12}
&\multirow{3}{*}{\makecell{TR-\\Flickr}}  & R@1 & - & - & 86.20 & - & 85.90 & \multicolumn{1}{>{\columncolor{mygray1}}c}{86.70} & 86.60 & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{88.90}\\
&  & R@5 & - & - & 96.30 & - & 97.10 & 97.80 & \multicolumn{1}{>{\columncolor{mygray1}}c}{97.90} & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{98.10}\\
&  & R@10 & - & - & 99.00 & - & 98.80 & 99.00 & \multicolumn{1}{>{\columncolor{mygray1}}c}{99.20} & - & \multicolumn{1}{>{\columncolor{mygray2}}c}{99.30}\\
\bottomrule
\end{tabular}
\vspace{-5pt}
\end{table*}

\subsection{Downstream Tasks}
After obtaining the pretrained ROSITA model, we finetune it on three downstream V+L tasks as follows.
\vspace{5pt}
\\
\noindent\textbf{Visual Question Answering (VQA)} is a task that requires the model to answer natural language questions about an image. We adopt the widely used VQAv2 dataset \cite{antol2015vqa,goyal2017making}, which is manually built on the images from the MSCOCO dataset \cite{lin2014microsoft}. The dataset is split into train (83k images and 444k questions), validation (41k images and 214k questions), and test (81k images and 448k questions) sets. Following the strategy in \cite{chen2020uniter}, we feed the representation of the $\mathsf{[CLS]}$ token to a linear classifier to predict the corresponding answer from a vocabulary of size 3129 \cite{yu2019deep}.
\vspace{5pt}
\\
\noindent\textbf{Referring Expression Comprehension (REC)} is a task that  requires to localize an image region referred to by a natural language query. We evaluate the performance on RefCOCO \cite{kazemzadeh2014referitgame}, RefCOCO+ \cite{kazemzadeh2014referitgame} and RefCOCOg \cite{mao2016generation} datasets. All the three datasets are collected from COCO images [31]. RefCOCO and RefCOCO+ are split into four subsets, including train (120k queries), validation (11k queries), testA (6k queries about people), and testB (6k queries about objects), while RefCOCOg is split into three subsets, including train (81k queries), validation (5k queries), and test (10k queries). The representation for each image region is used to predict a ranking score and a refined bounding box.
\vspace{5pt}
\\
\noindent\textbf{Image-Text Retrieval (ITR)} is a task that requires the model to calculate a similarity score between an image and a sentence and then perform cross-modal retrieval. We conduct experiments on the COCO Captions \cite{chen2015microsoft} and Flickr30K \cite{young2014image} datasets, respectively. Following the partition strategy by \cite{karpathy2015deep}, the COCO dataset is split into 82k/5k/5k train/validation/test images, while the Flickr30K dataset is split into 29k/1k/1k train/validation/test images. Similar to \cite{chen2020uniter}, we use an offline hard sample mining strategy to obtain 128 negative samples per each positive sample, and use the representation of the $\mathsf{[CLS]}$ token to predict a matching score.

\begin{table*}
\centering
\caption{Ablations of ROSITA variants without the cross- and intra-modal knowledge. All models are pretrained on the \emph{in-domain} datasets and then finetuned on specific downstream tasks. For each model, we report the accuracies on the pretraining tasks an downstream tasks, respectively. As we only have positive image-text pairs in the pretraining datasets, we use the offline hard sample mining strategy to generate an equal number of negative samples for the evaluation of the ITM task.}\label{table:aba_know}
\vspace{-5pt}
		\begin{tabular}{cl|ccc|cccc}
            \toprule
         \multirow{4}{*}{\#}   & \multirow{4}{*}{model} & \multicolumn{3}{c|}{pretraining tasks} & \multicolumn{4}{c}{downstream tasks}\\
            \cmidrule{3-9}
            && ITM& SKMLM & SKMRM & \makecell{VQAv2\\(dev)} & \makecell{RefCOCO\\(val)} & \makecell{IR-Flickr\\(test)} & \makecell{TR-Flickr\\(test)} \\
            \midrule
            1 & ROSITA (full)  &\textbf{84.34}&\textbf{67.16}&\textbf{76.50}&  \textbf{73.19} &  \textbf{84.22} &   \textbf{85.09}   & \textbf{94.33} \\
             2 & -w/o cross-modal  knowledge &83.54&63.69&72.56&   72.86&  83.85&   84.23    & 93.63 \\
             3 &  -w/o intra-modal knowledge  &83.30&63.75&73.90& 72.98  &  83.31 & 84.79  & 93.90  \\
             4 &  -w/o both types of knowledge   &82.22& 61.19&68.58& 72.47  & 82.12  &   82.11  & 92.57\\
            \bottomrule
        \end{tabular}
\vspace{-5pt}
\end{table*}

\subsection{Main Results}
We compare the proposed ROSITA model against existing state-of-the-art VLP methods. As shown in Table \ref{table:mainres}, ROSITA achieves the overall best performance on all downstream tasks, which verifies the effectiveness of the integrated cross- and intra-modal knowledge and the corresponding SKM strategy\footnote{We have conduct such an experiment that pretrains ROSITA on the \emph{out-of-domain} datasets only. The resulting model consistently outperforms the counterparts \cite{yu2020ernievil,lu2019vilbert,su2019vl}, verifying the generalization capability of our approach.}.

It is worth noting that some methods like ViLBERT, LXMERT, and ERNIE-ViL adopt the two-stream architecture, which have much more parameters (ROSITA: 116M, VilBERT: 221M, LXMERT: 183M, ERNIE-ViL: 228M). Some methods like UNITER and VILLA use a larger number of image features (up to 100 regions), which has been verified to benefit the performance at the expense of much higher computational cost. In contrast, ROSITA uses a fixed number of 36 image features. We believe the performance of ROSITA can be further improved by taking these advanced strategies above.

\subsection{Ablation Studies}
We run a number of ablations to investigate the reasons of ROSITA's effectiveness.
The results show in Table \ref{table:aba_know}-\ref{table:aba_skm} and Figure \ref{fig:attvis}-\ref{fig:comp_attvis} are discussed in detail below.
\begin{table}
\centering
\caption{Ablations of four ROSITA variants with two alternative masking strategy in SKM (\emph{i.e.}, independent probabilities and identical probability). All models are pretrained on the \emph{in-domain} datasets and finetuned on the downstream tasks.}\label{table:aba_skm}
\vspace{-5pt}
		 \begin{tabular}{c|cccc}
            \toprule
             \multirow{2}{*}{masking prob.} &  VQAv2 & {RefCOCO} & {IR-Flickr} & {TR-Flickr} \\
              & (dev) & (val)  & (test) & (test) \\
            \midrule
          independent&  \textbf{73.19} &  \textbf{84.22} &   \textbf{85.09}   & \textbf{94.33} \\
            identical ($p$=45\%)& 72.79& 83.18 &  83.70 & 93.20\\
            identical ($p$=30\%)&72.93 &    83.29 & 84.36 & 93.63\\
            identical  ($p$=15\%)& 72.75&  82.96 & 83.75 & 93.53\\
            \bottomrule
        \end{tabular}
        \vspace{-10pt}
\end{table}
\vspace{5pt}
\\
\noindent\textbf{Cross- and Intra-modal Knowledge.} In Table \ref{table:aba_know}, we show the effects of the intra-modal knowledge and cross-modal knowledge based on the performance on the pretraining and downstream tasks. Taking the full ROSITA as the reference model (Line \#1), we obtain the different variants by removing the cross-modal knowledge or the intra-modal knowledge. The variant without cross-modal knowledge (Line \#2) indicates that the model is not aware of the anchor objects and the SKM strategy is performed only on a single modality using the intra-modal knowledge. In contrast, the variant without intra-modal knowledge (Line \#3) indicates that the model is aware of the anchor objects but is not aware of the intra-modal contexts. Finally, by removing both the cross- and intra-modal knowledge, we obtain a baseline variant nearly identical to UNITER \cite{chen2020uniter} (Line \#4)\footnote{Our model has slight performance deviations compared with the original UNITER model since we use different visual features and pretraining hyper-parameters.}.

Given the pretrained models of the four variants above (\emph{i.e.}, without finetuning on downstream tasks), we evaluate their performance on three pretraining tasks. The ITM task examines the ability of semantic alignment between image-text pairs. From the results, we can see that both types of knowledge bring performance improvement to the ITM task (\#4 \emph{vs.} \#3 and \#2). Moreover, the two types of knowledge are complementary that their synergy brings 2.1 points improvement compared to the baseline model without any knowledge (\#4 \emph{vs.} \#1). Although the ITM task is the most straightforward metric for semantic alignment, it only measures the \emph{coarse-grained} alignments on the image-text level, thus cannot fully reveal the capability of ROSITA. As a complement, we resort to the SKMLM and SKMRM tasks to evaluate the \emph{fine-grained} alignments on the region-word level. Compared with the baseline model in \#4, the full ROSITA model improves the accuracies by 7.0 and 7.9 points on the SKMLM and SKMRM tasks, respectively.
\begin{figure*}
\begin{center}
\includegraphics[width=0.98\textwidth]{fig4.pdf}
\vspace{-5pt}
\caption{Visualizations of the learned cross-modal attentions (\emph{i.e.}, region-to-words attentions on the left and word-to-regions on the right) from UNITER \cite{chen2020uniter} and ROSITA. Taking the image-text pair as inputs with exactly one region (or word) being masked at a time, we extract the attention map from the last MSA block of the pretrained model. The region-to-words (word-to-regions) attentions correspond to one specific row in the bottom-left (top-right) area of the attention map, respectively.}
\label{fig:attvis}
\vspace{-10pt}
\end{center}
\end{figure*}

\begin{figure*}
\begin{center}
\includegraphics[width=0.98\textwidth]{fig5.pdf}
\vspace{-5pt}
\caption{Visualizations of the region-to-words attentions (left) and word-to-regions attentions (right) from a pretrained ROSITA model with masking multiple regions (or words) at one time.}
\label{fig:comp_attvis}
\vspace{-5pt}
\end{center}
\end{figure*}
Next, we report the performance of these variants on different downstream tasks. From the demonstrated results, we obtain similar observations to those on the pretraining tasks. The full ROSITA model consistently outperforms all the counterparts, verifying the effectiveness of the cross- and intra-modal knowledge.
\vspace{5pt}
\\
\noindent\textbf{SKM Strategy.} After extracting knowledge entries from image-text pairs, we have two alternative masking strategies in SKM, \emph{i.e.}, the independent probabilities and the identical probability. For the masking strategy with identical probability, we evaluate the choices of different probabilities within $\{15\%, 30\%, 45\%\}$. The results in Table \ref{table:aba_skm} show that the model pretrained with independent probabilities steadily outperforms all the counterparts with the identical probability. For the models pretrained with the identical probability strategy, their performance is sensitive to the choices of the predefined probability. A small masking probability (\emph{e.g.}, 15\%) may degrade the model towards the baseline without any knowledge. A large masking probability (\emph{e.g.}, 45\%) may shield the essential information that is necessary to learn the semantic alignments. In comparison, the masking strategy with independent probabilities provides a more fine-grained understanding of the knowledge structure, leading to a more robust pretrained model.
\vspace{5pt}
\\
\noindent\textbf{Cross-modal Semantic Alignments.} The effect of \emph{fine-grained} semantic alignments across modalities can be inferred from the attention maps of the learned Transformer model \cite{cao2020behind}. We visualize the learned \emph{cross-modal attentions} (\emph{i.e.}, region-to-words and word-to-regions attentions) from the pretrained UNITER \cite{chen2020uniter} and our ROSITA models, as shown in Figure \ref{fig:attvis}. Taking the image-text pair as inputs with exactly one token (a region or a word) being masked at a time, we pass the multimodal features through the pretrained model and extract the attention map from the last MSA block\footnote{We perform element-wise addition over the attention maps from different heads followed by row-wise softmax normalization to obtain one aggregated attention map.}. The region-to-words and word-to-regions attentions of the masked token correspond to one specific row in the bottom-left and top-right area of the attention map, respectively.

From the visualized cross-modal attentions, we can see that ROSITA learns significantly better semantic alignments than UNITER. ROSITA can precisely align the masked object to its reference object in the opposite modality while UNITER fails to establish such cross-modal alignments. For example, when the region of ``\emph{ramp}'' is masked, ROSITA activates the word ``\emph{ramp}'' precisely while UNITER obtains the largest attention value on the word ``\emph{skate}''. When another region of ``\emph{person}'' is masked, ROSITA precisely activates the word ``\emph{person}'' while UNITER still activates the incorrect word ``\emph{skate}''. Similar phenomena are observed in the opposite direction. ROSITA activates the accurate regions to the masked words while UNITER fails to do it.

To step further, we conduct a more challenging task as follows. We mask \emph{multiple} regions (or words) at the same time to examine whether the semantic alignments can still be achieved. The visualized results in Figure \ref{fig:comp_attvis} show that ROSITA works surprisingly well to establish accurate semantic alignment for each masked token. For example, when the regions of ``\emph{bowl}'' and ``\emph{bowl \& carrots}'' are masked simultaneously, the region of ``\emph{carrot}'' is precisely aligned to the word ``\emph{carrots}'', and the region of ``\emph{bowl \& carrots}'' is aligned to the two words ``\emph{bowl}'' and ``\emph{carrots}'' uniformly.
In the opposite direction, when the words ``\emph{man}'', ``\emph{candles}'', and ``\emph{cake}'' are masked at the same time, their corresponding regions are highlighted in the learned attentions, respectively.

\section{Conclusion}
In this paper, we present a new VLP method called ROSITA, which integrates the cross- and intra-modal knowledge in a unified scene graph to enhance the learning of cross-modal semantic alignment. We introduce a novel structural knowledge masking (SKM) strategy to perform masked language (region) modeling with respect to the knowledge entries extracted from the unified scene graph. Extensive ablations, comparative experiments, and comprehensive analysis show that ROSITA significantly outperforms existing state-of-the-art VLP approaches on three typical V+L tasks over six benchmark datasets. We hope our study will be helpful to inspire future research in the vision-and-language community and beyond.

\begin{acks}
This work was supported in part by the National Key R\&D Program of China under Grant 2018AAA0100603, and in part by National Natural Science Foundation of China under Grant 62072147, Grant 61836002, and Grant 62020106007.
\end{acks}

\newpage
\bibliographystyle{ACM-Reference-Format}


\section{Introduction}\label{intro}

When selected inclusions are integrated as fillers into soft polymeric matrices, new composite materials of extended functional application can be created \cite{stankovich2006graphene,balazs2006nanoparticle,thevenot2013magnetic}. Particularly interesting cases concern fillers that can be addressed from outside by external fields.
One such example are soft magnetic gels \cite{filipcsei2007magnetic,menzel2015tuned,odenbach2016microstructure}, where magnetic colloidal particles \cite{klapp2005dipolar,messina2014self} are embedded into soft elastic environments.
The result are soft elastic materials, the mechanical properties of which, such as the dynamic behavior \cite{jarkova2003hydrodynamics,bohlius2004macroscopic, filipcsei2007magnetic,tarama2014tunable}, elastic moduli \cite{stepanov2007effect,filipcsei2007magnetic,camp2011modeling,ivaneyko2012effects, borin2013tuning,han2013field,pessot2014structural,menzel2015tuned}, or nonlinear stress-strain behavior \cite{cremer2015tailoring},
can be reversibly tuned and switched from outside.

Another motivation has medical background and mainly concerns cancer treatment.
Magnetic colloidal particles, possibly loaded with drugs, can be guided to and embedded into cancer tissue using magnetic field gradients \cite{alexiou2006targeting,dobson2006magnetic, tietze2013efficient,zaloga2014development,matuszak2015endothelial}. During hyperthermic cancer treatment \cite{jordan1999magnetic,babincova2001superparamagnetic,lao2004magnetic,hergt2006magnetic}, the colloidal particles are
heated up. Heat is generated in the particles for instance by rapidly alternating external magnetic fields, as the ongoing remagnetization processes lead to hysteretic losses. When transmitted to the environment, the generated heat can destroy the surrounding cancer cells.

The nature of the coupling of such colloidal particles to their soft environment has been the subject of ongoing research \cite{messing2011cobalt,weeber2012deformation,gundermann2014investigation,roeder2015magnetic,huang2016buckling}. One point that should be clarified in more detail is the thermal coupling to the environment, which naturally becomes important upon heating the particles. As one aspect, one may ask whether thermophoretic effects \cite{rauch2002diffusion,rauch2003collective} become significant. Thermophoresis describes the observation that temperature gradients induce net forces on molecular and colloidal constituents.

In recent experiments, this question has been studied on relatively extreme polymeric model variants of the above situation \cite{schwaiger2011transient,schwaiger2013photothermal}. For this purpose, colloidal gold particles were embedded into a polymer solution on the one hand \cite{schwaiger2011transient} and into a strongly entangled
ultra-high-molecular-weight
polymer network on the other hand \cite{schwaiger2013photothermal}. An individual gold particle could then be selectively heated by external laser irradiation. In this way, \rev{its} surface temperature increased by more than $100$~K and thermophoretic effects could clearly be observed \cite{schwaiger2011transient,schwaiger2013photothermal}.

For the strongly entangled
ultra-high-molecular-weight polymer network \cite{schwaiger2013photothermal},
the disentanglement time was huge.
An overall elastic response of the polymer matrix on the time scale of the
experiment becomes conceivable. In this case, a very long-ranged
outward-oriented radial displacement field around the heated particle was
observed. It was tracked by the displacement of other, non-heated colloidal
particles.
\rev{These tracers are trapped in the mesh of the transient polymer network. Their motion reflects the local displacement of the polymer matrix. Approximately conserving the local volume, this displacement of the polymer is balanced by a
counter-displacement of the small solvent molecules such that no net flow of
the fluid as a whole is observed. On the one hand, in the experiments, a linear decay of the radial outward displacements around the heated particle was observed \cite{schwaiger2013photothermal}. On the other hand, the associated decay length was of the order of the sample thickness, and the displacements \textit{had} to decay with distance due to the confinement between two rigid plates of high thermal conductivity \cite{schwaiger2013photothermal}. In other words, the
confining boundaries of the sample cavity do not allow an outward
displacement on their inner surfaces \cite{schwaiger2013photothermal}. Therefore, it was conjectured that, remarkably, in an infinite system, the radial outward displacement of the polymer network is in principle constant in magnitude.} That is, the magnitude of the radial displacement
field does not decay with the distance from the center particle.

So far, this conjecture has not been tested by theoretical analysis. Previous qualitative arguments \cite{schwaiger2013photothermal} did not explicitly include the elasticity of the embedding polymer matrix. Particularly, a theoretical description of the situation would need to cover the possibility for the polymer network to move relatively to the solvent. This enables the relative displacement during the observed thermophoretic motion.

Here, we perform an according theoretical analysis in \rev{Sec.~\ref{theory_nodecay}}, employing a previously developed macroscopic two-fluid theory \cite{pleiner2004general,pleiner2004generalaip}. We confirm the experimental conjecture of an --- in principle --- non-decaying large-scale radial displacement field around the heated particle. An expression for the magnitude of this displacement field is calculated as a function of the material parameters and the heat rate. Moreover, our analysis demonstrates that elasticity is indeed the likely source for the experimentally observed behavior.
Our theoretical approach is complemented by experiments on solutions of polystyrene in toluene, see Secs.~\ref{experimental1} and \ref{experimental2}. These experiments are similar to the ones
reported in Ref.~\onlinecite{schwaiger2013photothermal} and contain new data on the heating power
dependence and the time evolution of the thermophoretic displacement of the
transient entanglement network. They highlight the role of elasticity by comparison to less entangled polymer solutions. The experimentally observed linear decay in the radial displacement field is attributed to the finite size of the sample, which is confined between rigid cavity walls. When \rev{confining} boundary conditions are imposed, \rev{a similar} linear decay \rev{is likewise found from our} theoretical results as demonstrated in Sec.~\ref{theory_finite_system}.

\section{Two-fluid description including thermophoretic effects}\label{theory_nodecay}

\rev{To perform our analysis, we employ a recently derived macroscopic two-fluid description \cite{pleiner2004general,pleiner2004generalaip}. For two-component systems, such theories explicitly allow for relative displacements and motion of one component with respect to the other \cite{pleiner2004general,pleiner2004generalaip,pleiner2013active}. In the present case, one component is a simple isotropic liquid playing the role of a solvent, while the other component corresponds to the isotropic elastic polymer network.}

\rev{We are interested in the far-field behavior of the radial displacement field $\mathbf{u}(\mathbf{r})$ of the polymer matrix.
Therefore, we assume weak elastic distortions far from the heated central
particle and thus study a linearized version of the macroscopic two-fluid equations derived in
Refs.~\onlinecite{pleiner2004general,pleiner2004generalaip}. Moreover, we
consider the case of a perfect and permanent elastic network. Due to the long
times of disentanglement for the experimentally employed
ultra-high-molecular-weight
polymer
\cite{schwaiger2013photothermal},
this represents a reasonable approximation.}

\begin{figure}
\includegraphics[width=5.0cm]{figure1.pdf}
\caption{\rev{Sketch of the system considered in Sec.~\ref{theory_nodecay}. The red wave represents the laser irradiation that heats the central particle at the origin, resulting in a spherically symmetric displacement field $\mathbf{u}(\mathbf{r})$ (indicated by the black arrows).}}
\label{fig:sketch}
\end{figure}

\rev{In the two-fluid picture, the first fluid of mass density $\rho_1$ represents the solvent, while the second fluid of mass density $\rho_2$ corresponds to the polymer. Assuming constant local density here means that the total mass density $\rho=\rho_1+\rho_2$ be constant. Then the solvent concentration $\phi$ is given by $\phi=\rho_1/\rho$ and the polymer concentration by $1-\phi=\rho_2/\rho$.
In equilibrium, i.e., in the non-heated state, the solvent concentration is given by a constant value $\phi_{\text{eq}}$. Upon heating, the concentration profile varies by $\delta\phi$, leading to $\phi=\phi_{\text{eq}}+\delta\phi$. The same applies for the entropy density, $s=s_{\text{eq}}+\delta s$.
Moreover, the displacement field $\mathbf{u}$ of the polymer matrix vanishes in equilibrium and becomes non-zero upon heating.
In summary, we characterize the state of our system by the three variables solvent concentration $\phi=\phi_\text{eq}+\delta\phi$, entropy density $s=s_\text{eq}+\delta s$, as well as displacement field $\mathbf{u}$ of the polymer matrix.
The theory in Refs.~\onlinecite{pleiner2004general,pleiner2004generalaip} provides central equations for all these quantities.}

\rev{In the following, we assume a heated colloidal particle in the center at the origin of our coordinate system and perfect spherical symmetry around it. A sketch of the geometry is depicted in Fig.~\ref{fig:sketch}.
As a consequence, we only obtain a radial variation of our variables, i.e.~a concentration field $\phi(r)=\phi_{\text{eq}}+ \delta\phi(r)$, an entropy density $s(r)=s_\text{eq}+\delta s(r)$, as well as a displacement field of the polymer matrix,
\begin{equation}\label{disp}
	\mathbf{u}(r)={}u(r)\mathbf{\hat{r}},
\end{equation}
see Fig.~\ref{fig:sketch}.
Here, $\mathbf{\hat{r}}$ is the radial unit vector in spherical coordinates. From now on, the dependence on $r$ is omitted for briefness of our notation.}

\rev{The central heat source is considered as point-like. This appears to be a reasonable simplification as the embedded particles in the experiment are of colloidal size and as we use a macroscopic continuum theory for the characterization of the set-up. As will be illustrated in more detail below, the colloidal particles are of approximately $100\,$nm in diameter, while separation distances of about $10-100\,\upmu$m were evaluated, which leads to a separation of length scales of at least two orders of magnitude.
We mentioned above that, in the experimental samples, additional non-heated colloidal particles are embedded that serve as tracer particles for the displacement field \cite{schwaiger2013photothermal}. Their possible influence on the overall behavior is likewise ignored.}

\rev{Due to the strongly entangled state of the polymer matrix, we assume that a final stationary state is reached under steady heating, which will be corroborated by the experimental results below. Therefore, in the following, all time derivatives and macroscopic velocities are equated to zero. This strongly simplifies the situation under investigation. Furthermore, as already mentioned above, we consider the composed system to  keep its local density --- commonly a reasonable assumption for polymeric gels.
As a final prerequisite, we need to introduce the linearized strain tensor \cite{landau1986elasticity}
\begin{equation}\label{lstrain}
	U_{ij}={}\frac{1}{2}(\nabla_i u_j+\nabla_j u_i).
\end{equation}
Inserting Eq.~(\ref{disp}), we obtain
\begin{equation}\label{strain}
	U_{ij}={}u'\hat{r}_i\hat{r}_j+\frac{u}{r}(\hat{\vartheta}_i\hat{\vartheta}_j+\hat{\varphi}_i\hat{\varphi}_j)
\end{equation}
in spherical coordinates with the unit vectors $\mathbf{\hat{r}}$, $\bm{\hat{\vartheta}}$, and $\bm{\hat{\varphi}}$. Moreover, $u'=\partial u(r)/\partial r$. Only the diagonal elements are non-zero.}

\rev{Under all these assumptions, we can now adopt the macroscopic two-fluid equations derived in Ref.~\onlinecite{pleiner2004general} to our situation, before we evaluate them.
We start with the macroscopic equation for the concentration field [Eq.~(120) in Ref.~\onlinecite{pleiner2004general}], which in the stationary limit and for isotropic systems becomes
\begin{equation}\label{conc}
		d\nabla^2(\mu_1-\bar{\mu}_2)+\phi(1-\phi)d^{(T)}\nabla^2 T ={} 0.
\end{equation}
\noindent Here, $d$ is the diffusion coefficient and $d^{(T)}$ denotes the thermodiffusion coefficient.
The field $T$ describes the temperature profile, while $\mu_1$ and $\bar{\mu}_2$ are the chemical potential and effective chemical potential for the mass densities $\rho_1$ and $\rho_2$, respectively.
The first term of Eq.~(\ref{conc}) corresponds to the stationary part of the diffusion equation for a concentration variation $\delta\phi$. This is revealed, when $\delta(\mu_1-\bar{\mu}_2)$ is expressed in our variables, which leads to a contribution $\sim\delta\phi$ [see Eq.~(\ref{deltamu}) below].
In our case, due to thermophoretic effects included by the second term $\sim\nabla^2 T$, nontrivial concentration profiles may arise due to the influence of spatially varying temperature fields.
Here, in the stationary limit, conventional diffusion given by the first term in Eq.~(\ref{conc}) can be balanced by thermodiffusion, described by the second term.}

\rev{Similarly, the} equation for the entropy density [Eq.~(123) in Ref.~\onlinecite{pleiner2004general}] \rev{in the stationary limit for isotropic systems} reads
\rev{\begin{equation}
\kappa\nabla^2 T + \frac{\rho_1\rho_2}{\rho}d^{(T)}\nabla^2(\mu_1-\bar{\mu}_2) ={}-q\delta_D(\mathbf{r}),\label{entr}
\end{equation}
\noindent with $\kappa$ the thermal conductivity. On} its right-hand side, Eq.~(\ref{entr}) contains the source term due to the external heating of the particle at the origin.
$\delta_D(\mathbf{r})$ denotes the Dirac delta function, \rev{treating the heated particle as point-like,} and $q$ sets the external heat rate.
\rev{Without the second term on the left-hand side, Eq.~(\ref{entr}) describes the stationary part of the conventional heat equation with a point-like heat source.
The second term on the left-hand side then includes variations in the temperature profile due to thermophoretic effects.}

Multiplying Eq.~(\ref{conc}) by $\kappa/(d^{(T)}\phi(1-\phi))$ and subtracting the result from Eq.~(\ref{entr}), the \rev{$\nabla^2 T$-terms} are eliminated and we obtain after linearization
\begin{equation}\label{nabla_deltamu_A}
	A\nabla^2\delta(\mu_1-\bar{\mu}_2)={}-q \delta_D(\mathbf{r}),
\end{equation}
with
\begin{equation}
	A={}\rho d^{(T)}\phi_{\text{eq}}(1-\phi_{\text{eq}})-\frac{d\kappa}{ d^{(T)}}\frac{1}{\phi_{\text{eq}}(1-\phi_{\text{eq}})}.
\end{equation}
Therefore,
\begin{equation}\label{deltamu_A}
	\delta(\mu_1-\bar{\mu}_2)={}\frac{q}{4\pi A}\frac{1}{r}+\tilde{A},
\end{equation}
\noindent with a constant $\tilde{A}$. As a function of $U_{ij}$, $\delta s$, and $\delta\phi$, variations $\delta(\mu_1-\bar{\mu}_2)$ can be expressed as [see Eq.~(131) in Ref.~\onlinecite{pleiner2004general}]
\begin{equation}\label{deltamu}
	\delta(\mu_1-\bar{\mu}_2)={}\frac{1}{\kappa_u}U_{kk} + \frac{1}{\rho\alpha_\phi}\delta s+\frac{1}{\rho\kappa_\phi}\delta\phi,
\end{equation}
with the expansion coefficient $\alpha_\phi$ and the compressibilities $\kappa_{\phi,u}$.
The trace of the strain tensor, see Eq.~(\ref{strain}), reads
\begin{equation}\label{trace}
	U_{kk}={}u'+\frac{2u}{r}.
\end{equation}
Inserting Eqs.~(\ref{deltamu_A}) and (\ref{trace}) into Eq.~(\ref{deltamu}), we obtain
\begin{equation}\label{GL1}
	\frac{q}{4\pi A}\frac{1}{ r}+\tilde{A} ={} \frac{1}{\kappa_u} \left(u'+\frac{2u}{r}\right) +\frac{1}{\rho\alpha_\phi}\delta s+ \frac{1}{\rho\kappa_\phi}\delta\phi.
\end{equation}

In a subsequent step, we return to Eq.~(\ref{conc}). There, we \rev{insert} Eq.~(\ref{nabla_deltamu_A}) and find
\begin{equation}
	B\nabla^2 T ={} -q\delta_D(\mathbf{r}),\label{laplace_T}
\end{equation}
with
\begin{equation}
	B ={} -A\frac{d^{(T)}\phi_{\text{eq}}(1-\phi_{\text{eq}})}{d}.
\end{equation}
Integration of Eq.~(\ref{laplace_T}) yields
\begin{equation}
	\delta T = \frac{q}{4\pi B}\frac{1}{r}+\tilde{T}\label{T},
\end{equation}
with a constant $\tilde{T}$.
Moreover, an expansion of the temperature variation is given by [Eq.~(130) in Ref.~\onlinecite{pleiner2004general}]
\begin{equation}\label{TEMP}
	\delta T={} \frac{1}{\alpha_3}U_{kk} + \frac{T_{\text{eq}}}{C_V}\delta s+\frac{1}{\alpha_\phi}\delta\phi  ,
\end{equation}
with $\alpha_3$ an expansion coefficient, $C_V$ the specific heat, and $T_{\text{eq}}$ the equilibrium temperature. Inserting Eq.~(\ref{T}) into Eq.~(\ref{TEMP}), we obtain
\begin{equation}\label{GL2}
	\frac{q}{4\pi B}\frac{1}{r}+\tilde{T} ={} \frac{1}{\alpha_3}\left(u'+\frac{2u}{r}\right)+\frac{T_{\text{eq}}}{C_V}\delta s+\frac{1}{\alpha_\phi}\delta\phi.
\end{equation}

Next, from the equation for the relative velocity between the two components [Eq.~(128) in Ref.~\onlinecite{pleiner2004general}], we find in the stationary limit
\begin{equation}\label{musigma}
	\nabla_i(\mu_1-\bar{\mu}_2)+\nabla_j\frac{1}{\rho_2}\sigma_{ij}={}0,
\end{equation}
with $\sigma_{ij}$ the stress tensor.
\rev{Summation over repeated indices is implied.}
After linearization, this equation may be rewritten as
\begin{equation}
	\nabla_j\big[\sigma_{ij} + \rho(1-\phi_{\text{eq}})\delta(\mu_1-\bar{\mu}_2)\delta_{ij} \big] ={}0.\label{sigma_gamma_grundgleichung}
\end{equation}
Thus, we may express $\sigma_{ij}$ as
\begin{equation}
	\sigma_{ij} ={} -\rho(1-\phi_{\text{eq}})\delta(\mu_1-\bar{\mu}_2)\delta_{ij}+\Gamma_{ij},\label{sigma_integriert}
\end{equation}
where we have introduced a diagonal tensor $\Gamma_{ij}$ that satisfies $\nabla_j\Gamma_{ij}=0$. The diagonal form of $\Gamma_{ij}$ is justified by the expression for the stress tensor $\sigma_{ij}$ as a function of $U_{ij}$, $\delta s$, and $\delta\phi$ [see Eqs.~(90) and (91) in Ref.~\onlinecite{pleiner2004general}], which takes the form
\begin{eqnarray}
	\sigma_{ij}&=&{}  c_{\text{tr}}\left(U_{ij}-\frac{1}{3}\delta_{ij}U_{kk}\right)+\frac{1}{3}c_{\text{l}}\delta_{ij}U_{kk} \nonumber\\
	 &&{}+\frac{1}{3\alpha_3}\delta_{ij}\delta s+\frac{1}{3\rho\kappa_u}\delta_{ij}\delta\phi.\label{stress}
\end{eqnarray}
In this expression, $c_{\text{tr}}$ and $c_{\text{l}}$ are the transversal and longitudinal elastic moduli, respectively.
From Eq.~(\ref{stress}) together with Eq.~(\ref{strain}) it follows that
$\sigma_{\vartheta\vartheta}=\sigma_{\varphi\varphi}$. An analogous relation thus applies for $\Gamma_{ij}$. $\nabla_j\Gamma_{ij}=0$ therefore yields the additional condition
\begin{equation}
	\Gamma_{rr}'+2\frac{\Gamma_{rr}}{r}-2\frac{\Gamma_{\vartheta\vartheta}}{r} ={} 0.\label{GL3}
\end{equation}
Introducing Eq.~(\ref{sigma_integriert}) into Eq.~(\ref{stress}), we obtain two different equations, namely the
$\mathbf{\hat{r}\hat{r}}$ component,
\begin{eqnarray}\label{GL4}
	\frac{q}{4\pi C}\frac{1}{r}+ \tilde{C} +\Gamma_{rr} &={} & \frac{2}{3}c_{
		\text{tr}}\left(u'-\frac{u}{r}\right)  +\frac{1}{3}c_{\text{l}} \left(u'+\frac{2u}{r}\right) \notag\\
	&{}&+ \frac{1}{3\alpha_3}\delta s+\frac{1}{3\rho\kappa_u}\delta\phi,
\end{eqnarray}
and the $\bm{\hat{\vartheta}\hat{\vartheta}}$ or $\bm{\hat{\varphi}\hat{\varphi}}$ component,
\begin{eqnarray}\label{GL5}
	\frac{q}{4\pi C}\frac{1}{r}+ \tilde{C} +\Gamma_{\vartheta\vartheta} &={} & \frac{1}{3}c_{
		\text{tr}}\left(\frac{u}{r} - u'\right)  +\frac{1}{3}c_{\text{l}} \left(u'+\frac{2u}{r}\right) \notag\\
	&{}&+ \frac{1}{3\alpha_3}\delta s+\frac{1}{3\rho\kappa_u}\delta\phi,
\end{eqnarray}
with the constants
\begin{equation}
	 C = -\frac{A}{\rho(1-\phi_{\text{eq}})}, \qquad \tilde{C}= -\rho(1-\phi_{\text{eq}})\tilde{A}.\label{C_schlange}
\end{equation}

In the macroscopic two-fluid description of Ref.~\onlinecite{pleiner2004general}, there is another equation resulting for the elastic degrees of freedom [Eq.~(124) in Ref.~\onlinecite{pleiner2004general}]. However, in the stationary case and for our assumptions, this equation is satisfied identically (we consider a perfectly elastic network, which on the considered time scale does not disentangle; moreover, vacancy diffusion \cite{martin1972unified} is neglected). Apart from that, the assumption of overall constant density sets the pressure field [see Eqs.~(127) and (129) in Ref.~\onlinecite{pleiner2004general}].

Therefore, our remaining task is to solve the system of Eqs.~(\ref{GL1}), (\ref{GL2}), (\ref{GL3}), (\ref{GL4}), and (\ref{GL5}). We use as an ansatz for $u$, $\delta s$, $\delta\phi$, $\Gamma_{rr}$, and $\Gamma_{\vartheta\vartheta}$, respectively, a power series in the radial distance~$r$,
\begin{eqnarray}
	u &={} &\sum\limits_{n=-\infty}^{\infty} u_n r^n,\label{ansatz_u}\\
	\delta s &={} &\sum\limits_{n=-\infty}^{\infty} \delta s_n r^n,\label{ansatz_s}\\
	\delta\phi &={} &\sum\limits_{n=-\infty}^{\infty} \delta\phi_n r^n,\label{ansatz_phi}\\
	\Gamma_{rr} &={} &\sum\limits_{n=-\infty}^{\infty} \Gamma_{rr,n}r^n,\\
	\Gamma_{\vartheta\vartheta} &={} &\sum\limits_{n=-\infty}^{\infty} \Gamma_{\vartheta\vartheta,n}r^n.\label{ansatz_gamma_theta}
\end{eqnarray}

Inserting this ansatz into Eqs.~(\ref{GL1}), (\ref{GL2}), (\ref{GL3}), (\ref{GL4}), as well as (\ref{GL5}), \rev{we obtain equations that can be sorted by order in $r^n$.
Solving the equations for each order of $r^n$ separately, we find the system of equations}
\begin{eqnarray}
	 0&={}&(n+2) \frac{1}{\kappa_u} u_n + \frac{1}{\rho\alpha_\phi}  \delta s_{n-1} + \frac{1}{\rho\kappa_\phi} \delta\phi_{n-1}, \label{alle_n_A}\\
	 0&={}&(n+2) \frac{1}{\alpha_3} u_n + \frac{T_{\text{eq}}}{C_V}  \delta s_{n-1} + \frac{1}{\alpha_\phi} \delta\phi_{n-1}, \label{alle_n_B}\\
	 0&={}&(n+1)\Gamma_{rr,n-1}-2\Gamma_{\vartheta\vartheta,n-1},\label{alle_n_Gamma}\\
	 0&={}& \frac{1}{3}\big(2(n-1)c_{\text{tr}} +(n+2)c_{\text{l}} \big)u_n\notag\\
	 	 &{}& + \frac{1}{3\alpha_3}\delta s_{n-1} + \frac{1}{3\rho\kappa_u}\delta\phi_{n-1}  -\Gamma_{rr,n-1} \label{alle_n_sigma1},\\
	 0 &={}& \frac{1}{3}\big((1-n)c_{\text{tr}} + (n+2)c_{\text{l}}\big) u_n \notag\\
	 	 &{} &+ \frac{1}{3\alpha_3}\delta s_{n-1} + \frac{1}{3\rho\kappa_u}\delta\phi_{n-1} -\Gamma_{\vartheta\vartheta,n-1} \label{alle_n_sigma2},
\end{eqnarray}
where $n\in\mathbb{Z}\setminus\{0,1\}$.
We note that there is no coupling between sets $\{u_n, \delta s_{n-1}, \delta\phi_{n-1},\Gamma_{rr,n-1},\Gamma_{\vartheta\vartheta,n-1}\}$ of different $n\in\mathbb{Z}\setminus\{0,1\}$. Moreover, the heat source $q$, which here is responsible for driving the system out of equilibrium, does not appear in the above equations.
Thus it follows from the equations that \rev{our} expansion coefficients for $n\in\mathbb{Z}\setminus\{0,1\}$ keep their equilibrium values equal to zero.

An exception, for which no strict statement is obtained, are the coefficients $u_{-2}$, $\Gamma_{rr,-3}$, and $\Gamma_{\vartheta\vartheta,-3}$. $u_{-2}$ cancels in Eqs.~(\ref{alle_n_A}) and (\ref{alle_n_B}), while we obtain from the remaining equations:
\begin{equation}
	u_{-2} ={} -\frac{\Gamma_{rr,-3}}{2c_{\text{tr}}}, \qquad \Gamma_{\vartheta\vartheta,-3} ={} -\frac{\Gamma_{rr,-3}}{2}. \label{eq:redundance}
\end{equation}
Thus, the precise magnitude of the coefficient $u_{-2}$, strictly speaking, remains unassigned in the present framework. It may be determined by the microscopic processes and properties on the heated particle surface, which are not captured by our macroscopic theory \cite{pleiner1996pattern}. Yet, this coefficient is insignificant for our purpose as we are interested in the far-field behavior, while the contribution $\sim r^{-2}$ decays rapidly in the far-field displacement. The far-field displacement is dominated by a \rev{long-ranged} leading order identified in the following.

For $n=1$, we find from Eqs.~(\ref{GL1}), (\ref{GL2}), (\ref{GL3}), (\ref{GL4}), and (\ref{GL5}) the expressions
\begin{eqnarray}
	 \tilde{A}&={}& \frac{3}{\kappa_u} u_1 + \frac{1}{\rho\alpha_\phi}  \delta s_{0} + \frac{1}{\rho\kappa_\phi} \delta \phi_{0},\label{u_1_a} \\
	 \tilde{T}&={}& \frac{3}{\alpha_3} u_1 + \frac{T_{\text{eq}}}{C_V}  \delta s_{0} + \frac{1}{\alpha_\phi} \delta \phi_{0}, \\
	 0&={}&\Gamma_{rr,0}-\Gamma_{\vartheta\vartheta,0}, \label{redundant}\\
	 \tilde{C}&={}& c_{\text{l}} u_1  + \frac{1}{3\alpha_3}\delta s_{0} + \frac{1}{3\rho\kappa_u}\delta \phi_{0}  -\Gamma_{rr,0}.\label{u_1_e}
\end{eqnarray}
One of the two relations in Eqs.~(\ref{alle_n_sigma1}) and (\ref{alle_n_sigma2}) becomes redundant due to Eq.~(\ref{redundant}).
Thus, additional conditions are necessary to fix the magnitude of the expansion coefficients  $u_1$, $\delta s_{0}$, $\delta \phi_{0}$, and $\Gamma_{rr,0}$.
At $r\rightarrow\infty$ the magnitude of displacement should not diverge. This sets $u_1=0$. In addition to that, we assume that for $r\rightarrow\infty$ the temperature and the (effective) chemical potentials keep their equilibrium values, e.g., via coupling to external heat and substance reservoirs. From Eqs.~(\ref{deltamu_A}), (\ref{T}), and (\ref{C_schlange})
it then follows that $\tilde{A}=\tilde{T}=\tilde{C}=0$ and $\delta s_0=\delta\phi_0=\Gamma_{rr,0}=\Gamma_{\vartheta\vartheta,0}=0$.

Finally, we find for the remaining order $n=0$ \rev{the following system of linear equations:}
\begin{eqnarray}
	 \frac{q}{4\pi A}&={}& \frac{2}{\kappa_u} u_0 + \frac{1}{\rho\alpha_\phi}\delta  s_{-1} + \frac{1}{\rho\kappa_\phi} \delta\phi_{-1}, \label{n_0_gl_a}\\
	 \frac{q}{4\pi B}&={}& \frac{2}{\alpha_3} u_0 + \frac{T_{\text{eq}}}{C_V} \delta s_{-1} + \frac{1}{\alpha_\phi} \delta\phi_{-1},\\
	 \frac{q}{4\pi C}&={}& \frac{2}{3}(c_{\text{l}}-c_{\text{tr}}) u_0+ \frac{1}{3\alpha_3}\delta s_{-1} \notag\\
	 	 	 &{}& + \frac{1}{3\rho\kappa_u}\delta\phi_{-1}  -\Gamma_{rr,-1}, \\
	 \frac{q}{4\pi C} &={}& \frac{1}{3}(c_{\text{tr}} + 2c_{\text{l}}) u_0 + \frac{1}{3\alpha_3} \delta s_{-1} \notag\\
	 	 	 &{} &+ \frac{1}{3\rho\kappa_u}\delta\phi_{-1} -\Gamma_{\vartheta\vartheta,-1}, \\
	 0&={}&\Gamma_{rr,-1}-2\Gamma_{\vartheta\vartheta,-1}.\label{n_0_gl_e}
\end{eqnarray}
Here, the heat source $q$ directly enters and drives the system out of equilibrium.
\rev{These equations can be solved explicitly, which leads to expressions for the coefficients $u_0$, $\delta s_{-1}$, $\delta\phi_{-1}$, $\Gamma_{rr,-1}$, and $\Gamma_{\vartheta\vartheta,-1}$.
They are rather lengthy and therefore are listed in the appendix. The important point is that we have now obtained an analytical solution of the macroscopic equations, which reveals the overall long-ranged response of the system.}

In \rev{summary}, the \rev{results for the} entropy density and the concentration \rev{profile are obtained as}
\begin{eqnarray}
	 s(r) &=&{}  s_{\text{eq}}+\frac{\delta s_{-1}}{r},\label{s_final} \\
	 \phi(r) &=&{} \phi_{\text{eq}}+\frac{\delta\phi_{-1}}{r}. \label{phi_final}
\end{eqnarray}
\rev{Very illustratively, in these expressions the $1/r$ dependence of the entropy and concentration deviations can be viewed as a direct consequence of the spherical symmetry with the heated source in the center. From the heat equation, see Eq.~(\ref{entr}), we find the $1/r$ temperature profile given by Eq.~(\ref{T}). This well-known result survives the thermophoretic coupling to the concentration field described by Eqs.~(\ref{conc}) and (\ref{entr}). With the $1/r$ temperature profile at hand, already the expansion of the temperature fluctuations in our variables in Eq.~(\ref{TEMP}) suggests the $1/r$ dependence of the entropy and concentration deviations found in Eqs.~(\ref{s_final}) and (\ref{phi_final}), see also Fig.~\ref{fig:th_infinite}.}

\rev{Thus, the irradiated heat drives the system out of equilibrium, which is directly reflected by the modified entropy density and, via thermophoretic coupling, by the modified concentration field.
Changing the local concentration in our spherically symmetric situation is only possible by radial displacement of the elastic polymer network. However, radial displacements imply strains, see also Eq.~(\ref{trace}), which directly contribute to the chemical potential and temperature variations, see Eqs.~(\ref{deltamu}) and (\ref{TEMP}). Naturally, opposing stresses arise from these distortions, see Eq.~(\ref{stress}), which counteract the effect. Overall,}
the general solution for the \rev{radial} displacement field reads
\begin{equation}\label{u_final}
	u(r)={}u_0+\frac{u_{-2}}{r^2},
\end{equation}
\rev{and contains a \textit{non-decaying} component $u_0$.}
As discussed above, the precise magnitude of the coefficient $u_{-2}$ remains unassigned in the present framework. Yet, its impact decays rapidly as a function of distance and does not influence the leading-order \rev{far-field} behavior connected with the coefficient $u_0$.
\rev{The leading-order deviations of $u(r)$, $s(r)$, and $\phi(r)$ from static equilibrium are illustrated in Fig.~\ref{fig:th_infinite} as functions of $r$.}

\begin{figure}
\includegraphics[width=7.5cm]{figure2.pdf}
\caption{\rev{Plots of the rescaled leading-order deviations from static equilibrium for $u(r)$, $s(r)$, and $\phi(r)$ as functions of the distance $r$ from the heated center, see Eqs.~(\ref{s_final})--(\ref{u_final}). The rapidly decaying $r^{-2}$ term in Eq.~(\ref{u_final}) is neglected. Overall, our macroscopic solutions highlight the long-ranged far-field behavior, see also the discussion in the text.}}
\label{fig:th_infinite}
\end{figure}

\rev{At first glance, one might wonder how a non-decaying radial outward displacement of constant magnitude is energetically possible throughout the system. After all, the amount of displaced material diverges with increasing distance $r$ from the center. For the moment, let us consider spherical shells of thickness $\mathrm{d}r$ around the center. Then, the number of displaced volume elements on each shell diverges with its volume as $\sim r^2\mathrm{d}r$. These volume elements are stretched along the directions tangential to the shell surface during radial outward displacement, which costs elastic energy. However, the corresponding local strain energy density\cite{landau1986elasticity} following via the strain given by Eq.~(\ref{trace}) decreases as $\sim 1/r^2$. Overall, we obtain a constant strain energy $\sim\mathrm{d}r$ per shell. Illustratively, the reason for this result is the following. There are many more strained volume elements on the shells for large $r$. However, the curvature of these shells decreases with increasing $r$. As a consequence, the outward displacement with increasing $r$ locally more and more resembles a rigid outward translation of the locally nearly flat parts of the shell surfaces. Rigid translations do not cost elastic energy. Still, in total, the non-decaying radial displacement field requires an infinite input of strain energy in our infinite system. Yet, it is provided via the infinite input of heat energy (already the $1/r$ temperature profile resulting for the simple decoupled static heat equation corresponds to an infinite energy input in an infinite system).}

Overall, \rev{with our central result in Eq.~(\ref{u_final})}, we here confirmed in a two-fluid approach that it is indeed the elastic response of one of the two components that can induce \rev{a} \textit{non-decaying} radial \rev{outward} displacement \rev{$u_0$}, as conjectured in Ref.~\onlinecite{schwaiger2013photothermal}. Moreover, we derived an expression for the magnitude of $u_0$, \rev{see Eq.~(\ref{sol_u}) in the appendix}, as a function of the macroscopic material parameters, \rev{where, however,} several numerical values of these macroscopic parameters are not known at present.

\section{Experimental setup and sample preparation}\label{experimental1}

To further elaborate on the role of elasticity during the temperature-induced
deformation process and to further study the influence of a finite system
size, we have performed additional experimental investigations of the
thermophoretic effect around laser-heated gold nanoparticles (GNPs) in
transient polymer networks.

GNPs (BBInternational) of approximately $100\,\text{nm}$ diameter show a
strong optical absorption in their plasmon resonance.  They serve as tunable
heat sources that can individually be addressed by a tightly focused laser
beam ($\lambda = 525\,\text{nm}$, Coherent Verdi V5), see Fig.~\ref{fig:setup} for an illustration of our set-up.
The non-heated GNPs, which are randomly distributed over the sample, serve as
tracers to visualize the deformation of the transient network.  Their
displacement is monitored with an optical microscope (Olympus IX71) equipped
with an sCMOS camera (Tucsen Discovery MH15). Phase contrast imaging is used
for the observation of smaller particles.

\begin{figure}
\includegraphics[width=8.5cm]{figure3.pdf}
\caption{Experimental set-up to visualize thermophoretically induced
displacements. A focused laser beam heats a single gold nanoparticle (GNP),
indicated in yellow. Other GNPs (grey) are trapped as tracers in an entangled
polymer network and follow its displacement. The sample consists of a thin
film of about $200\,\upmu\text{m}$ in thickness, confined between rigid
cuvette walls. }
\label{fig:setup}
\end{figure}

Samples were prepared by first dissolving polystyrene (PS) of different
molecular weight ($M_w = 16\,800\,\text{kg}\,\text{mol}^{-1}$, $M_w =
500\,\text{kg}\,\text{mol}^{-1}$, PSS) in toluene (Sigma Aldrich, 99.9\%) at a
weight concentration of 6\%, yielding a highly entangled polymer solution.
It is then slowly stirred with 20 rpm for at least 10 days. A small droplet
of the polymer solution is filled into a detachable cuvette (Hellma 106-QS).
After evaporation of the solvent, the thin left-over polymer film is overlayed
with a droplet of the aqueous GNP dispersion. The GNPs are given approximately
10 minutes to sediment and attach to the polymer layer, before the remaining
GNP dispersion is removed by flushing with deionized water (Millipore).
After drying, the remaining volume is filled with polymer
solution. The cuvette is closed and sufficient time is given for the
polymer solution to redissolve the lower polymer layer and to homogenize,
resulting in a transient entangled polymer network with randomly embedded
GNPs.
The final polymer concentration in the cuvette is approximately
twice the one of the stock solution.
Sedimentation of the GNPs is very slow and can be compensated by occasionally
turning over the cuvette.

Several samples were examined
using laser powers between $2\,\text{mW}$ and $20\,\text{mW}$. Due to the
small cross section of the GNPs and the much wider laser focus, the absorbed
laser power is several orders of magnitude smaller.  Particle displacement was
determined
by fitting a two dimensional Gaussian function, which allows for a position
accuracy of roughly $\pm$ $50\,\text{nm}$.

\section{Measurements}\label{experimental2}

\rev{We have recorded the displacements of the tracer particles in the plane parallel to the cell windows, which we define as the $xy$ plane. The optical axis was perpendicular to this plane.}
The displacements of GNPs within a distance of $10\,\upmu\text{m}$ to
$120\,\upmu\text{m}$ to the heated center have been tracked and are plotted
in Fig.~\ref{fig:displacement}.
\begin{figure}
\includegraphics[width=8.5cm]{figure4.pdf}
\caption{Distance-dependent radial displacements of GNPs for different
heating powers as indicated on the top right. Linear fits illustrate an approximately linear decay with radial distance from the heated center. All data points were acquired at a time of $2\,\text{s}$ after switching on the heating.
(The initial concentration of the polymer was $c=0.06$.)}
\label{fig:displacement}
\end{figure}
These displacements
show a behavior similar to the
 ones reported by Schwaiger et al.\ \cite{schwaiger2013photothermal},
however, with a somewhat extended range.
Most likely, this is a consequence of
the higher polymer concentration of the stock solution ($c=0.06$ as compared
to $c=0.03$), but no systematic study of the concentration dependence has been
carried out so far.

As indicated in Fig.~\ref{fig:displacement}, the amplitude of the long-ranged
radial displacements shows an approximately linear decay
with the distance from the heated center.  This linear decay is
attributed to the finite size of the sample, which is confined by rigid
cuvette walls \rev{of sevenfold higher thermal conductivity} that enforce vanishing displacements at the cell boundaries
\cite{schwaiger2013photothermal}.
Thus, our experimental data do not contradict our theoretical results in
Sec.~\ref{theory_nodecay}. There, an infinitely extended system has been
considered, while a rigidly confined system will be discussed below in
Sec.~\ref{theory_finite_system}. Apart from that, a sublinear dependence of
the displacement
amplitude
on the heating power was observed.

All displacements in Fig.~\ref{fig:displacement} were measured at a fixed time
of $2\,\text{s}$ after turning on the heating.  We plot the time evolution of
the displacement for two different examples of lower laser power in
Fig.~\ref{fig:time_evolution}.
\begin{figure}
\includegraphics[width=8.5cm]{figure5.pdf}
\caption{Time dependence of the displacement of two different GNPs at
different distances from the heated center as indicated on the top left. The fits were performed using an exponential with a single relaxation time, yielding $(0.54\pm0.04)\,\text{s}$ and $(1.27\pm0.19)\,\text{s}$ for the two distances,
respectively \cite{supplemental}.
A laser power of $2.25\,\text{mW}$ was applied.}
\label{fig:time_evolution}
\end{figure}
As can be inferred from Fig.~\ref{fig:time_evolution},
the characteristic time scale
to reach a \mbox{(quasi-)}stationary displacement increases with radial
distance from the heated GNP. The fits
correspond to
an exponential with a single relaxation time. As we can
see, relaxation is to a
large
extent completed at our measurement time of
$2\,\text{s}$.
We have checked that the linear decay in Fig.~\ref{fig:displacement} is still
observed when plotting the extrapolated asymptotic displacements instead of
the transient values after $2\,\text{s}$ \cite{supplemental}.

Finally, we underline the role of the elastic response of the strongly
entangled polymer network. In accord with our theoretical considerations in
Sec.~\ref{theory_nodecay}, it is the most likely source for the observed
long-ranged displacements. For this purpose, we performed identical
measurements using, however, polymers of significantly lower molecular weight
of $M_w =
500\,\text{kg}\,\text{mol}^{-1}$ at the same concentration. This hardly entangled polymer solution does not feature a long-time elastic response.

Typical example trajectories of tracer GNPs in the high- and low-molecular-weight samples relatively close to the heated center are shown in Figs.~\ref{fig:trajectory_high} and \ref{fig:trajectory_low}, respectively.
\begin{figure}
\includegraphics[width=8.5cm]{figure6.pdf}
\caption{Example trajectory of a tracer GNP at an initial distance of approximately
$15\,\upmu\text{m}$ from the heated center in a $c = 0.06$ solution
of molecular weight $M_w = 16\,800\,\text{kg}\,\text{mol}^{-1}$.
\rev{The heated center is located at the origin.}
A protocol of sudden stepwise increase and subsequent decrease of the laser power $0\,\text{mW}\rightarrow2.25\,\text{mW}\rightarrow6.75\,\text{mW}\rightarrow11.25\,\text{mW}\rightarrow16.75\,\text{mW}\rightarrow2.25\,\text{mW}$ is performed. The numbers on the trajectory mark switches to the new (indicated) laser power, from where relaxation to the new steady state occurs.
The coincidence of the point of switching $2.25\,\text{mW}\rightarrow6.75\,\text{mW}$ and of the end point at $2.25\,\text{mW}$ highlights the reversible character of the displacements.}
\label{fig:trajectory_high}
\end{figure}
\begin{figure}
\includegraphics[width=8.5cm]{figure7.pdf}
\caption{Example trajectory of a tracer GNP at an initial distance of approximately
$15\,\upmu\text{m}$ from the heated center in a $c = 0.06$ solution
of $M_w =500\,\text{kg}\,\text{mol}^{-1}$.
\rev{The heated center is located at the origin.}
Here, the GNP is subject to visible
Brownian motion due to the relatively low viscosity compared to the highly entangled
solution in Fig.~\ref{fig:trajectory_high}. The laser is switched on only at the point marked by the arrow at about
$1.5\,\text{s}$ into the recorded trajectory. Turning on the laser does not have any visible effect.
} \label{fig:trajectory_low}
\end{figure}
During a heating protocol of switching on and off the laser, the GNP in the high-molecular-weight sample is undergoing
a directed and reversible displacement. In contrast to that, the displacements of the GNP in the lower-molecular-weight sample are dominated by Brownian motion. In this lower-molecular-weight case, Brownian motion is more pronounced because of the lower viscosity of the solution. Other than for the entangled system, switching on the laser does not lead to correlated displacements.

\section{Theoretical description of rigidly confined systems}\label{theory_finite_system}

Our generalized theoretical derivation in Sec.~\ref{theory_nodecay}
has been
performed for an infinitely extended system, while in the experiments, by
construction, samples of finite size were investigated. Most importantly, the
experimental samples are confined by rigid cuvette walls and an approximately
linear decay in the outward displacement field was observed, see
Fig.~\ref{fig:displacement}. We explain in the following that \rev{similar results can be found within} our theoretical \rev{framework}, \rev{when modified}
boundary conditions \rev{are applied}.

To address finite size, we consider the system to be confined within a spherical shell of effective radius $R$ \cite{schwaiger2013photothermal}. This \rev{geometry} keeps the problem within our analytical framework of Sec.~\ref{theory_nodecay}. Assuming the shell to be rigid, the outward displacement must vanish on its surface:
\begin{equation}\label{finite_u_bc}
	u(r=R) ={} 0.
\end{equation}
\rev{Naturally, the rigid spherical shell does not exactly match the experimental configuration of rigid confinement between two parallel plates. However, the presence of the confining cuvette windows in the experiments sets a dominant length scale for the overall response of the system. The cuvette walls have a sevenfold higher thermal conductivity, which tends to bend the direction of the heat fluxes towards the plate surfaces. As a consequence, both geometries, the experimental situation and the rigid sphere, are strongly influenced by the one dominating length scale set by the confinement.}

Returning to Sec.~\ref{theory_nodecay}, we find that our results and conclusions from Eqs.~(\ref{alle_n_A})--(\ref{eq:redundance})\rev{, (\ref{n_0_gl_a})--(\ref{n_0_gl_e}), and (\ref{sol_u})--(\ref{sol_abb})}, particularly our expression for $u_0$, remain unchanged \rev{upon imposing Eq.~(\ref{finite_u_bc})}. Modifications only concern our conclusions from Eqs.~(\ref{u_1_a})--(\ref{u_1_e}). Previously, the coefficient $u_1$ only vanished from our requirement that $u(r\rightarrow\infty)$ \rev{should not diverge}. Under the modified boundary condition Eq.~(\ref{finite_u_bc}), we instead obtain
\begin{equation}
u_1 = -\frac{u_0}{R},
\end{equation}
where we now directly neglected a possible --- but within the present framework undetermined --- coefficient $u_{-2}$. Its contribution $\sim r^{-2}$ vanishes rapidly with increasing distance from the heated center.
Altogether, this leads to
\begin{equation}\label{u_finite}
	u(r) ={} u_0\left(1-\frac{r}{R}\right)
\end{equation}
and thus \rev{coincides with} the linear decay observed in the experiments.

Let us briefly discuss the implications for the other variables in
Eqs.~(\ref{u_1_a})--(\ref{u_1_e}). For instance, one may maintain the
equilibrium temperature $T(r=R)=T_{\text{eq}}$ on the system boundary by
coupling to an external heat bath. This sets the constant $\tilde{T}=-q/4\pi B
R$. Assigning a value to $\tilde{A}$ is difficult as it, in principle,
requires knowledge about chemical interaction details with the shell surface.
In an actually closed spherical cavity, the overall confined material should
be conserved. The constant $\tilde{A}$ can be used to tune $\delta\phi_0$ to
achieve this overall goal within the framework of our solution.  It should be
recalled, however, that in our experiment the system is not enclosed in a
sphere and exchange of material in the lateral direction is possible.

\section{Conclusions}

In this work, we have analyzed the situation of
an elastic two-fluid system subjected to thermophoretic effects. The two-fluid environment consists of an elastic matrix suspended in a fluid solvent. These two components can be displaced relatively with respect to each other by externally heating a colloidal particle embedded in the system center. Recent experiments reported here and in Ref.~\onlinecite{schwaiger2013photothermal} suggest a long-ranged \textit{non-decaying} component of the resulting thermophoretically induced radial displacement field \rev{in an idealized infinite system}. The elastic response of the polymer matrix seems to play a central role in these observations.
Using a macroscopic two-fluid description \cite{pleiner2004general,pleiner2004generalaip}, our theoretical analysis confirms this experimental conjecture \rev{in the linearized regime}.
Moreover, for rigidly confined systems, a \textit{linear decay} in the radial displacement field is experimentally observed. \rev{Similar results are found from} theoretical considerations for rigidly bounded spherical systems.

\rev{We should recall at this point that our analysis was performed for small deviations from equilibrium only, using a macroscopic theory. That is, we lost several couplings between different variables during the linearization of the equations. For example, Eqs.~(\ref{conc}) and (\ref{entr}) initially contain nonlinear couplings to the concentration deviations. In dynamic situations, further nonlinearities are present due to the convective terms but also due to nonlinear contributions to the reversible stress, see Ref.~\onlinecite{pleiner2004general}. Predominantly, deviations from the linearized regime are of course obtained close to the heated center. Yet, in the very vicinity of the center, the macroscopic description loses its significance in any case and microscopic processes need to be taken into account. Most immediately, our macroscopic approach serves to characterize the far-field behavior, which was the central scope of our investigation.}

Concerning future investigations, we remark that our present analysis is restricted to the case where a final steady state is attained. As a next step of significantly higher complexity,
dynamic situations could be addressed. On the one hand, this applies for the dynamic path towards a final steady state. \rev{In nonlinear situations, the final state may even depend on the chosen dynamic path towards it.} On the other hand, a final steady state might not be reached at all \rev{on the considered time scales}, if the polymer matrix is not perfectly elastic but shows disentanglement or vacancy diffusion \cite{martin1972unified,pleiner2004general,pleiner2004generalaip}. Moreover, both theoretically and experimentally, situations of more than one heated particle could be addressed. Finally, the significance of the observed effect in similar situations but different systems should be analyzed. A related example with medical background is biological tissue exposed to hyperthermic cancer treatment \cite{jordan1999magnetic,babincova2001superparamagnetic,lao2004magnetic,hergt2006magnetic}.