How to properly convert pytorch LSTM to keras CuDNNLSTM?
I am trying to hand-convert a Pytorch model to Tensorflow for deployment. ONNX doesn't seem to natively go from Pytorch LSTMs to Tensorflow CuDNNLSTMs so that's why I'm writing it by hand.

I've tried the code below:
This is running in an Anaconda environment running Python 2.7, Pytorch 1.0, tensorflow 1.12, cuda9. I'm running this with no bias in the Pytorch layer as it follows a batchnorm, but since Keras does not provide that option I'm simply assigning a 0 bias. 

import torch
import tensorflow as tf
import numpy as np
from tensorflow.keras.layers import CuDNNLSTM, Bidirectional
from tensorflow.keras.models import Sequential, Model

input_size = 80
hidden_size = 512
with torch.no_grad():
    rnn1 = torch.nn.LSTM(input_size=input_size, hidden_size=hidden_size, bidirectional=True, bias=False, batch_first=True).cuda()

model = Sequential()
model.add(Bidirectional(CuDNNLSTM(hidden_size, return_sequences=True),  input_shape=(None, input_size), name='rnn'))

bias_size = rnn1.weight_hh_l0.detach().cpu().numpy().T.shape[1] * 2
keras_format_weights = [
                    rnn1.weight_ih_l0.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0.detach().cpu().numpy().T,
                    np.zeros(bias_size,),
                    rnn1.weight_ih_l0_reverse.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0_reverse.detach().cpu().numpy().T,
                    np.zeros(bias_size,),
                  ]


model.layers[0].set_weights(keras_format_weights)

random_test = np.random.rand(1, 1, 80)

res1, _ = rnn1.forward(torch.FloatTensor(random_test).cuda())
res1 = res1.detach().cpu().numpy()
res2 = model.predict(random_test)

print(np.allclose(res1, res2, atol=1e-2))
print(res1)
print(res2)


False
[[[ 0.01265562  0.07478553  0.0470101  ... -0.02260824  0.0243004
   -0.0261014 ]]]
[[[-0.05316251 -0.00230848  0.03070898 ...  0.01497027  0.00976444
   -0.01095549]]]


Now, this does work with the generic Keras LSTM:

model = Sequential()
model.add(Bidirectional(LSTM(hidden_size, recurrent_activation='sigmoid', return_sequences=True),  input_shape=(None, input_size), name='rnn'))

bias_size = rnn1.weight_hh_l0.detach().cpu().numpy().T.shape[1]
keras_format_weights = [
                    rnn1.weight_ih_l0.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0.detach().cpu().numpy().T,
                    np.zeros((bias_size,)),
                    rnn1.weight_ih_l0_reverse.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0_reverse.detach().cpu().numpy().T,
                    np.zeros((bias_size,))
                  ]


But I need the speed advantages of the CuDNNLSTM, and Pytorch is using the same backend anyway.

Update: the solution was to convert the torch model into a keras base LSTM model, then call

base_lstm_model.save_weights('1.h5')
cudnn_lstm_model.load_weights('1.h5')

