Concatenate convolution layers with different strides in tensorflow. I am trying to do an inception layer, but it only works if the convolution strides, pool strides and pool size are the same, otherwise I get an error in 


  tf.concat


that Dimesion 1 is not the same. So If I change something in the last three tuples, I get the error.

conv1 = conv2d_maxpool(x, 64, (5, 5), (1, 1), (2, 2), (2, 2)) 
conv2 = conv2d_maxpool(x, 64, (4, 4), (1, 1), (2, 2), (2, 2)) 
conv3 = conv2d_maxpool(x, 32, (2, 2), (1, 1), (2, 2), (2, 2)) 
conv4 = conv2d_maxpool(x, 32, (1, 1), (1, 1), (2, 2), (2, 2)) 
conv = tf.concat([conv1, conv2, conv3, conv4], 3)


For example, this is the error I get if I change the 5x5 filter to have strides 3:

conv1 = conv2d_maxpool(x, 64, (5, 5), (3, 3), (2, 2), (2, 2))



  Dimension 1 in both shapes must be equal, but are 6 and 16 for
  'concat' (op: 'ConcatV2') with input shapes: [?,6,6,64], [?,16,16,64],
  [?,16,16,32], [?,16,16,32], [].


This is the conv2d_maxpool function:

def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    # TODO: Implement Function
    weights = tf.Variable(tf.truncated_normal(
        shape = [*conv_ksize, int(x_tensor.get_shape().dims[3]), conv_num_outputs], 
        mean = 0.0, 
        stddev=0.1, 
        dtype=tf.float32))
    bias = tf.Variable(tf.zeros(conv_num_outputs)) 

    conv_layer = tf.nn.conv2d(x_tensor, weights, strides=[1, *conv_strides, 1], padding='SAME')
    conv_layer = tf.nn.bias_add(conv_layer, bias)
    conv_layer = tf.nn.relu(conv_layer)

    conv_layer_max_pool = tf.nn.max_pool(conv_layer, ksize=[1, *pool_ksize, 1], strides=[1, *pool_strides, 1], padding='SAME')

    return conv_layer_max_pool


How can I combine convolution filters with different strides and/or different pooling to create an inception layer?
Not 100% sure, but the problem is that when you work with different strides, the size of the convolved image change, so you should ensure, that all the convolved images have the same shape before concatenating the output. You can fill with 0s, or considering that the image is periodical in time, so filling with reflections of the image