Ways to Encode context for text classification? I have a binary classification task which has the following specification:

Input: Chunk of text (not more than a few sentences, mostly a sentence).
Additional Input: For each input sample there is additional information available (which is also some text of similar length, max 2-3 sentences).
Problem: Classify text content using additional context
Problem Type: Binary classification

Essentially the task boils down to classifying the content conditioned on the context or p(content|context).

I was thinking of effective ways to encode the text for classification using a deep neural network. I searched for recent works but the existing literature mostly use the technique mentioned below.

I was wondering if there is a better way to encode context? Which has shown to be effective in some domain as compared to just concatenation.

Encode then concatenate: This involves encoding the context using ways similar to the content and then concatenating the feature representation before classification. This is the widely used technique. There are a lot of variants of this technique (different ways to encode text e.g. using tfidf rep, word embedding, LSTMs, CNNs ) which are widely explored.

Are there any better ways to bring context information during classification?

P.S: Recursive RNN is something that I have on my TODO.
A Recursive Neural Network (RNN) is best suited to solve your problem. And if you want to take a step further try Recursive Neural Tensor Network. You can generate compositions for entire sentences using RNN both for your content and context data. This will help you map your sentences to a higher dimensional space and as an output you will get numerical values (or embeddings) both for your content and context. Feeding these values into a simple logistic classifier will do your task. For a brief overview on RNNs take a look at this lecture by Richard Socher.