title: matplotlib Gridspec subplots unexpected different size
body: <p>I am trying to create a grid of images using matplotlib.
  The first row and column define the input to a function and the rest of the grid is the output.
  Here's someone else's reference of how I would like it to look: <a href="https://i.stack.imgur.com/TWlSm.png" rel="nofollow noreferrer">reference</a>.</p>
  
  <p>Especially note that lines seperating the first row and column from everything else.</p>
  
  <p>I was trying for the last couple of hours to make it work. The best I've come so far is using Gridspec to divide the image into four groups and construct the image using PIL.
  However, for a reason I cannot understand the shapes of the different subplots don't match.</p>
  
  <p>Attaching a minimal code and it's output.</p>
  
  <pre><code>import matplotlib.pyplot as plt
      import matplotlib.gridspec as gridspec
      import PIL
  
      f = plt.figure(figsize=(20, 20))
  
      resolution = 256
      num_images = 6
      h = w = num_images
      main_grid = gridspec.GridSpec(h, w, hspace=0, wspace=0)
  
      col = f.add_subplot(main_grid[0, 1:])
      row = f.add_subplot(main_grid[1:, 0])
      mid = f.add_subplot(main_grid[1:, 1:])
      corner = f.add_subplot(main_grid[0, 0])
  
      corner_canvas = PIL.Image.new('RGB', (resolution, resolution), 'gray')
      mid_canvas = PIL.Image.new('RGB', (resolution * w, resolution * h), 'yellow')
      col_canvas = PIL.Image.new('RGB', (resolution * w, resolution), 'blue')
      row_canvas = PIL.Image.new('RGB', (resolution, resolution * h), 'red')
  
      corner.imshow(corner_canvas)
      col.imshow(col_canvas)
      row.imshow(row_canvas)
      mid.imshow(mid_canvas)
  
      plt.savefig('fig.png')
  </code></pre>
  
  <p>As you can see <a href="https://i.stack.imgur.com/c4ec3.png" rel="nofollow noreferrer">here</a>, the shapes don't match which make the grid not aligned.
  Any solution producing an image in the style of the reference would be great !</p>