Loading a saved model in Keras with a custom layer and prediction results are different?
I finally found the answer.
I have implemented a custom Lambda layer to handle reshaping. This layer has difficulties loading. Specifically, it reshapes one dimension into two dimensions on an arbitrary interval. The interval was defaulting to one specific value every time I loaded the model, even though it was incorrect. When I force it to be the correct interval, it works.
My network was achieving 96% accuracy on my dataset (edit: on predicting 9 classes). I saved the entire model for every epoch (weights included) whenever I ran it. I ran it 3 times, each time testing different hyperparameters, each time achieving around 96% accuracy.
When I try to load any of these tests now and run it again, it achieves around 50% accuracy. I'm extremely confident I'm running it on the same dataset. 
Here's the interesting thing: if I train a new network instance of the exact same architecture, same size, shape, etc., it only ever reaches a max of 85% accuracy. Additionally, saving and loading these new training models works correctly, as in the model will reach the same 85% accuracy.
So, there's no problem with loading and no problem with my dataset. The only way this could happen is if something is wrong in my custom layer, or something else is happening.
Unfortunately I haven't been committing all of my changes to the custom layer to git. While I can't guarantee that my custom layer is the exact same, I'm almost completely confident it is.
Any ideas to what could be causing this discrepancy?
Edit: To add more context, the layer is ripped from the ConvLSTM2d class but I replaced the call() function to simply be a vanilla RNN that uses convolution instead of the dot product. I am confident that the call() function is the same as before, but I am not confident that the rest of the class is the same. Is there anything else in the class that could affect performance? I've checked activation function already.