imagemagick's convert jpeg to pdf rotates even pages 180°
It seems like the scanner I used to scan the documents I was trying to make PDFs of was saving the images with an orientation property. These were the two properties:

Orientation: 1 (Normal)

Orientation: 3 (Rotated 180°)

So, although it looked as though the images were upright, every second image would technically be flipped 180°, which is the property that imagemagick would use. 

Removing this property from multiple images should be possible on macOS using a tool like ExifTool.