Why port B5 on ATMEGA64A-AU mailfunction after enabling USART function?
The register map of Mega64 and Mega103 are different, for example the Mega103 does not have UBRRH and UCSRC registers while the Mega64 does, so writing to these register addresses in Mega103 mode may control something else that hangs the processor.
To find out exacty what is the main reason, you need to debug it, but SRAM starts earlier on Mega103, so writing to the Mega64 extended register addresses maps to SRAM and it might overwrite data in RAM, but there is not much SRAM usage in the code, and stack should not be in that area.
So more likely is that the stack is at the end of the SRAM, and on the Mega103 the memory ends earlier due to not having the extended registers, so running Mega64 code in Mega103 mode puts the stack into non-existing memory area, so stack is unavailable. Calling subroutines will fail to push the return address and returning from a subroutine pops the wrong address.