Ways to combine embeddings I am building a recommender for an online shop and I have categorical inputs that belong to one of the following categories:


user current session features (e.g. current_product_brand, current_product_id) 
user previous activity, i.e. latest 10 sessions
(e.g previous_product_id, previous_product_brand) 
user
previous activity 2D features, i.e. (e.g. product_semantics_embedding, product_tags_embedding - assume that each product has 20 tags/semantics)


Indicative part of the summary of the model:

_________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
current_product_brand (InputLaye   (None, 1)            0                                            
__________________________________________________________________________________________________
current_product_id (InputLayer)    (None, 1)            0                                            
__________________________________________________________________________________________________
previous_product_id (InputLayer)   (None, 10)           0                                            
__________________________________________________________________________________________________
previous_product_brand (InputLayer)(None, 10)           0                                            
__________________________________________________________________________________________________
previous_product_semantics (InputL (None, 200)          0                                            
__________________________________________________________________________________________________
previous_product_tags (InputLayer) (None, 200)          0                                            
__________________________________________________________________________________________________
product_brand_embedding (Embeddi   (None, 10, 60)       60       current_product_brand[0][0]       
                                                                 previous_product_brand[0][0]      
__________________________________________________________________________________________________
product_id_embedding (Embedding)   (None, 10, 60)       98820    current_product_id[0][0]            
                                                                 previous_product_id[0][0]
__________________________________________________________________________________________________
product_semantics_embedding (Embed (None, 200, 60)      104880   previous_product_semantics[0][0]    
__________________________________________________________________________________________________
product_tags_embedding (Embedding) (None, 200, 60)      2760     previous_product_tags[0][0]         
__________________________________________________________________________________________________      


Features like product brand that appear both in current and previous sessions are embedded in the same space. 

Note that the output of all embeddings is constant (in this case 60).

Now, I want to combine all the embeddings into a single tensor in order to feed them into another layer, e.g. a Dense. 
I think my options are the following:


Concat all embeddings: I cannot use axis 1 since the product_semantics and product_tags have different shape. Does it makes sense to concat them on axis 2? 
Concat them per group, i.e concat product_brand_embedding with product_id_embedding and product_semantics_embedding with product_tags_embedding, apply global average pooling to each results and then concat the  2 outputs of the global average pooling nodes.


Which is the right way to go ? Are there any other options?
I have tackled this exact issue using GlobalAveragePooling1D to flatten the output from those multivariate Embeddings and then concatenated them all along with the 1D embeddings later, which is how Youtube treats such things in their own recommendation engine:



So in your place, I would go with your second approach. When I did it, the model structure looked something like this:

Model: "model_8"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
history-input (InputLayer)      (None, 3063)         0                                            
__________________________________________________________________________________________________
manufacturer-history-input (Inp (None, 3063)         0                                            
__________________________________________________________________________________________________
history-embedding (Embedding)   (None, 3063, 32)     538144      history-input[0][0]              
__________________________________________________________________________________________________
manufacturer-history-embedding  (None, 3063, 32)     32224       manufacturer-history-input[0][0] 
__________________________________________________________________________________________________
average-history-embedding (Glob (None, 32)           0           history-embedding[0][0]          
__________________________________________________________________________________________________
manufacturer-average-history-em (None, 32)           0           manufacturer-history-embedding[0]
__________________________________________________________________________________________________
numeric-inputs (InputLayer)     (None, 49)           0                                            
__________________________________________________________________________________________________
concatenate (Concatenate)       (None, 113)          0           average-history-embedding[0][0]  
                                                                 manufacturer-average-history-embe
                                                                 numeric-inputs[0][0]             
__________________________________________________________________________________________________
dense-512 (Dense)               (None, 1024)         116736      concatenate[0][0]                
__________________________________________________________________________________________________
dropout_13 (Dropout)            (None, 1024)         0           dense-512[0][0]                  
__________________________________________________________________________________________________
dense-256 (Dense)               (None, 512)          524800      dropout_13[0][0]                 
__________________________________________________________________________________________________
dropout_15 (Dropout)            (None, 512)          0           dense-256[0][0]                  
__________________________________________________________________________________________________
target (Dense)                  (None, 16817)        8627121     dropout_15[0][0]                 
==================================================================================================
Total params: 9,839,025
Trainable params: 9,839,025
Non-trainable params: 0