Towards Emergent Language Symbolic Semantic Segmentation and Model
  Interpretability
  Recent advances in methods focused on the grounding problem have resulted in
techniques that can be used to construct a symbolic language associated with a
specific domain. Inspired by how humans communicate complex ideas through
language, we developed a generalized Symbolic Semantic ($\text{S}^2$) framework
for interpretable segmentation. Unlike adversarial models (e.g., GANs), we
explicitly model cooperation between two agents, a Sender and a Receiver, that
must cooperate to achieve a common goal. The Sender receives information from a
high layer of a segmentation network and generates a symbolic sentence derived
from a categorical distribution. The Receiver obtains the symbolic sentences
and co-generates the segmentation mask. In order for the model to converge, the
Sender and Receiver must learn to communicate using a private language. We
apply our architecture to segment tumors in the TCGA dataset. A UNet-like
architecture is used to generate input to the Sender network which produces a
symbolic sentence, and a Receiver network co-generates the segmentation mask
based on the sentence. Our Segmentation framework achieved similar or better
performance compared with state-of-the-art segmentation methods. In addition,
our results suggest direct interpretation of the symbolic sentences to
discriminate between normal and tumor tissue, tumor morphology, and other image
characteristics.
