Best way to connect two adjacent PCBs by their outermost hole rows?
I would consider connecting them with 90° angled male and female header pins, while on each end of those add 2 "spacers" and use small (such as M3) screws to hold in place, as shown on photo below.

Different shade of green shows on 2 different PCBs, while on one side there are female and on the lighter green side they are male pin headers. I've drawn straight instead of 90° angled, but you get the point.
On each side, you have those small grey "boxes", which are aluminum parts with 2 drilled holes for your screws (as I've said, such as M2, M2.5 or M3, depends on size of your PCB) and the red are the screw heads. Needles to say, use washers and nuts on the bottom of your PCBs.
I think this will be quiet a good, permanent connection between them.



EDIT: 3 new options came to my mind;
Both ones start with setting one PCB on top of another, so the holes match.
One option is to fully cast both holes together with the solder. I wouldn't recomend this option very much, as other 2 are more stable.

Second one is to stack holes again, one on top of another and push through copper wire, as thick as possible, that goes through the holes, then solder it on both edges (on top of top PCB and bottom of bottom PCB).

The third one is very simmilar to the second one, but use pins for this application, such as these: