Recurrent neural network multiple types of input Keras
Kera's functional API allows arbitrary models to be combined like you are describing. 

One option is to define a model very similar to the documentation linked above. First, define a "main input" that is categorical, and then, define an "auxiliary input" that is numerical. Both of these sub-models can be merged into a single network with additional layers.