Explaining what a neural network is learning In more traditional statistical learning methods, such as logistic regression, the coefficients on the pre-selected features can be interpreted as increasing or decreasing the log-odds of success. For example, say we want to use logistic regression with a single predictor variable studying_hours to predict whether a student will pass the exam. We can estimate the parameters in the model:

$log(Y) = \beta_0 + \beta_1$studying_hours

where $Y = 1$ if the student passes the exam and 
$Y = 0$ otherwise, and 

studying_hours is a variable with values 1, 2, ..., 100 hours.

We can say that a one-unit increase (a one hour increase) in the number of hours studying for the exam increases the log-odds of passing the exam by $\beta_1$.

It stands to reason that we should be able to make a similar argument when the features are learned instead of pre-selected. But then we would need to understand exactly what the learned feature is. In my example above, instead of pre-selecting studying_hours to be the feature we want to use to predict whether a student passes the course, we would learn some informative feature instead. 

I've heard of Hinton diagrams, and of visualizing the feature map for CNNs, but I don't think either of those techniques really works here. A Hinton diagram would just tell me the magnitude and sign of the weights. A feature map would not really be informative. I don't expect to understand what all of the features are, because the whole point is that the algorithm can do a better job of learning features than I can do designing features. But even so, at least some of the features should be human-intepretable, like a 'straight line detector' in a CNN.

How can we interpret the features being learned by a multilayer perceptron for logistic regression? 

Not really sure what to tag here, so please, if there's an edit you would recommend, go ahead!

EDIT

This is my attempt at starting an answer, but clarification/other opinions would be appreciated.

Let's take a case of binary classification, and start from the simplest possible case: one scalar input, one hidden unit with a relu activation, and one output unit. Assume the hidden unit has a bias of 0. We can express the function that describes this network as:

$log(Y) = \beta_0 + \beta_1 max\{wx, 0\}$,

where


$\beta_0$ is the bias on the output unit
$\beta_1$ is the weight on the output unit
$w$ is the weight from the input to the hidden unit


Then we can expand that function to:

$log(Y) = \beta_0 + \beta_1wx$, if $|wx| &gt; 0$ 

$log(Y) = \beta_0$ otherwise.

Which means that if $|wx| &gt; 0$, then the hidden unit is giving us a factor by which to multiply the slope of the line separating the two classes. We can say that a one-unit increase in $\beta_1 w$ implies a one-unit increase in the log-odds of success. 

If $|wx| &lt;= 0$, then the best separating line between the two classes has 0 slope, regardless of the magnitude of the weight (I think this is related to the dying relu problem).

If we expand this to include a bias on the output, we have:

$log(Y) = \beta_0 + \beta_1(wx + b)$, if $|wx + b| &gt; 0$ 

$log(Y) = \beta_0$ otherwise.

Through arithmetic, that first part can be expressed as:

$log(Y) = \beta_0 + \beta_1b + \beta_1wx$, if $|wx + b| &gt; 0$

which means to me that the weight $w$ from the input to the hidden unit tells us both how to modify the y-intercept (which is now $\beta_0 + \beta_1b$) and how to modify the slope. 

Does that make sense?
That's a popular question, since many use cases require to understand what's going on in the model.

I suggest to do just some simulation with the resulting NN. Like in your example you could have a test-dataset and den modify the 'studying_hours' by +5%, +10%, +15%, ... and then plot the result over this modification factor. This way you can see by an simulation what the NN is thinking of the 'studying_hours' and how it influences the result.