Finding serial port on generic IP Camera PCB
Well I'd start with the datasheet for the ralink.  He's clearly the controller in this system.  You could look at its pin list, find the location of serial or jtag pins, and then search that corner of the chip on the board.  It will take a lot of poking and measuring but maybe you'll get lucky and their boot loader puts a message out at startup and you can see that on a scope.

Reverse engineering is an exercise in patience and trial and error, followed by more trial and error.

On the other hand the  Linux files are not stored in the ralink but rather externally in a little flash part.   The datasheet suggests it supports boot from spi-flash, so look up the other part numbers on the chips on your board and see if one is a spi-flash.   You could then just pull it off and program it with a little programmer (they're not that expensive).   That might be an easier path if you know what needs to be programmed in.  Even better if you have another camera and you could just clone its flash.

It does mention nand flash support but I didn't see boot from nand flash.