Character level CNN - 1D or 2D
I want to implement a character-level CNN in Pytorch. 

My input has 4 dimensions:


  (batch_size, seq_length, padded_character_length, embedding_dim)


I'm wondering whether I should merge two dimensions and use a Conv1D-layer or instead use a Conv2D-layer on the existing dimensions.

Given the dimensions of the input both would technically work fine, I also have seen implementations for both versions. So I'm wondering which method to prefer.

Does one of the two methods have particular advantages over the other?

I agree with Venkatesh that 1D might make more sense for your implementation. Instead of merging, I typically use the TimeDistributed layers that are found in Keras. This takes one layer and applies it across time dimensions. The advantage is that you keep the features from each dimension separate until you want to merge them. 

If you are using padding (as you mentioned) then it makes more sense to apply the same layer across the time dimensions instead of merging the layers and creating awkward padding space in between actual characters. TimeDistributed layers solves this.

I googled for just a moment and found someone who has tried implementing this feature in PyTorch, that might at least get you started.

PyTorch TimeDistributed

To expand on my usage a bit.. My latest model has 5, 12 hour periods, each 12 hour period has sporadic activity so it is padded to reach a standard length of 30, so the final input shape is (?, 5, 30, embedding_size). I generate features within a single period using a TimeDistributed 1D CNN, then I max pool those features and concatenate to create a new shape of (?, 5, n_feats) where there are now 5 sets of feature maps. I again run that data over with a different 1D CNN layer which looks across the 5, 12 hour periods. The padding in each period is independent of each other, so I can't simply use a 2D CNN as elements at the same index wouldn't represent the same time across periods.

Edit:

I think the Keras implementation is a little more sophisticated but that should be close. Their documentation says "This wrapper applies a layer to every temporal slice of an input." If getting there requires to merge then restore after then their may be some considerations around the feature map. For example, if the filter size is 2 then the last item in the first feature map (after reshaping) will include the last feature and the first feature between two time slices.

Here is one more link on discussion around this capability in PyTorch that might be helpful. 
