Loss is bad, but accuracy increases?
If you have an imbalanced dataset, you may want to take steps to resample. When evaluating your results of your algorithm, it may be a good idea to choose a metric that is better suited for class imbalance problems. 

Accuracy is not a good metric for evaluating the performance of class imbalance problems because accuracy is rewarded for predicting the most commonly occurring label. Accuracy is the measurement of the the number of labels predicted correctly over the number of all the observations. For example, if you had 99% of the data with label 'a' and 1% of the data with label 'b', and you then labeled the entire dataset with label 'a', your accuracy would be 99%. However, your ability to label 'b' correctly was 0%. Log loss is also not particularly good for class imbalance problems. You may want to consider weighted log loss instead. 

Here are some other metrics that you can investigate.

Precision and Recall

Precision is the ratio of correctly predicted positive labels to the total predicted positive observations. Recall is the ratio of correctly predicted positive observations to the all observations in the given class. You can read more about these metrics here. For a Binary Classification problem, you can view the predicted class vs the actual class in a confusion matrix.



F1 Score

F1 Score is the weighted average of precision and recall. F1 score takes false negatives and false positives into account. So, it is more informative than accuracy when evaluating class imbalance problems. 



F1 Score exists as a Python package in sklearn. It also exists as a function in R.

Area Under Precision Recall Curve

You can also investigate Area under precision recall curve (PR), which the measurement under the area of precision recall curve plot. It can be used to evaluate large class imbalance problems. PR Curve exists as a python package in sklearn and an R package.