Prediction: plugin a corelation table (neuron) into a Time-Series Neuron in Keras/ TF i am adding more details

I have a time series of Babies (1,2,3) showing how many problem they have each week (Born week 1 to week 80) and in which organ (14 organ).

There is a separate numeric time-free correlation tables for babies/organs. showing Xtype blood cells per organ per baby -> Causes problems

My aim is to predict how many problem will NewBaby (Baby4) have, in which organ, per week

Predict this: Week1: 1 problem | Stomach Week1: 1 problem | Liver Week2: 1 problem | Stomach ..

TIME_SERIES_INPUT ------> LSTM (RNN?) -------\
-------------------------------------------------------------------> MERGE --->
AUXILIARY_INPUTS --> [do something] ----------/*

How can I model this problem with Keras/TF? To be specific firstly should I modify the dataset to include correlation tables. Or can I plug a model into LSTM? I have seen some examples on this board but they are mostly CNN that does not apply to my data set Ps I can insert whole dataset if needed

Thanks

DataSet Looks like 

Week Baby   Age Organ   Date    IssueFound Problem Type
2   Baby2   A0  Organ2  06/24/16    1   F
2   Baby3   A0  Organ11 05/01/17    1   F
2   Baby3   A0  Organ11 05/02/17    1   F
2   Baby3   A0  Organ13 05/03/17    1   F
2   Baby3   A0  Organ13 05/05/17    1   F
3   Baby2   A0  Organ12 06/29/16    1   F
3   Baby2   A0  Organ2  07/01/16    1   F
4   Baby3   A0  Organ13 05/18/17    1   F
4   Baby3   A0  Organ2  05/19/17    1   F


time-free Co relation Table looks like

New X-Type Blood Cell per Organ per baby

Organ   Baby1   Baby2   Baby3   NewBaby
Bladder Organ1  288 247 97  253
Brain   Organ2  2548    1972    992 251
Ears    Organ3  32  254 162 155
Eyes    Organ4  111 0   459 -89
Heart   Organ5  42  1214    0   142
Kidney  Organ6  4   188 186 -87
Kidneys Organ7  806 1764    796 153
Liver   Organ8  658 657 -20 0
Lung    Organ9  0   484 0   0
Nose    Organ10 3328    9940    4435    3223
Pancreas    Organ11 11  389 207 194
Skin    Organ12 664 766 164 -145
Stomach Organ13 246 -71 0   -70
Tongue  Organ14 1403    0   33  0
If you just have a normal correlation table (Which I am not sure about) you can do it like this: For every row you can add a correlation of the organ in a row to other organs. So you will have 

Week Baby   Age Organ   Date    IssueFound Problem Type Organ1 Organ2 Organ3 ...
2   Baby2   A0  Organ2  06/24/16    1      F            0.5    0      0.3    ...  


If you do not have a row for a new baby you should then format the data this way together with other parameters. The Corr_Organ1 an aggregation of all correlations that lead to Organ1 failure.

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         Yes    06/24/16    No     nan         0.2         0           ...


Then you make iterations: First forecast everything with zero correlations. Then when you calculate the first forecast and Organ 1 failed you update all other correlations.

Starting data:

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         nan    nan         nan    nan         0           0           ...


After first iteration:

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         yes    xxx         no     nan         0           0.8         ...


Then you predict again based on correlations and get the following result:

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         yes    xxx         yes     nan         0.2         0.8         ...


You repeat those iterations until you do not change anything.

The aggregation of correlations could be just a sum or something more complicated.