Should there be a flat layer in between the conv layers and dense layer in YOLO? Should there be a flat layer in between the conv layers and dense layer in YOLO?

It's something not specified in the paper, but I see most implementations of YOLO on github do this. In my implementation, I do not flatten the 7*7*1024 feature map and directly add a Dense(4096) layer after it (I'm using keras with tensorflow backend).

Code in question:

x = Conv2D(filters=1024, kernel_size=(3,3), padding='same', activation=leaky_relu)(x)
# x = Flatten()(x) # not using this right now
x = Dense(4096, activation=leaky_relu)(x)
x = Dense(30)(x)


My output of model.summary is as follows:

conv2d_294 (Conv2D)          (None, 14, 14, 1024)      9438208   
_________________________________________________________________
conv2d_295 (Conv2D)          (None, 7, 7, 1024)        9438208   
_________________________________________________________________
conv2d_296 (Conv2D)          (None, 7, 7, 1024)        9438208   
_________________________________________________________________
conv2d_297 (Conv2D)          (None, 7, 7, 1024)        9438208   
_________________________________________________________________
dense_14 (Dense)             (None, 7, 7, 4096)        4198400   
_________________________________________________________________
dense_15 (Dense)             (None, 7, 7, 30)          122910    


I feel that this implementation is more correct than the flat one (unless otherwise specified by the author) because on flattening the output we lose the spatial information needed to predict bounding boxes and confidence scores for each element in the grid.
It doesn't matter, with or without flattening, a Dense layer takes the whole previous layer as input. The spatial structure information is not used anymore. Some Neural Network implementations might not be able to map a spatial structure directly into a dense layer, which is why you would need the Flatten in between. Mathematically it is exactly the same in this case.
EDIT: As I expected, in the source of Keras it mentions that if the rank is higher like with convolutions, they implicitly Flatten it first.