Is it a sign of overfitting when validation_loss dips and then goes up with increasingly bigger swings?
Yes this looks a lot like overfitting. The clue is in the low and slowly decreasing training loss compared to the large increases in validation loss.

One simple fix would be to stop training around epoch 50, taking the best cross validation result to select the most general network at that point. However, anything that works to improve stable generalisation could help here - more training data, more regularisation, simpler model. 

The tricky part is finding the best combination for generalisation. Typically if you overuse regularisation to make the NN completely stable, it will lose some accuracy, so you need to run multiple experiments and measure things carefully. As you have a small dataset here and seem capable of running 1000's of epochs, I would suggest k-fold cross validation for improved measurement of cross validation loss.