Questions regarding design problem
1) Antennae can both transmit and receive, so you can use another of the same type, in principle. Sometimes you want one to be directional and the other to be non-directional, so they are generally different; a broadcaster often wants to be heard from every direction, so would choose a different transmit antenna.

2) Yes they are constrained by the antenna and the frequency range is given in the datasheet. Limits on the amplitude are not given because they are determined by conductors heating and electrical breaktown but unless you're using vacuum tubes for the amplifiers you are unlikely to approach them.

HOWEVER both frequency and amplitude are tightly constrained by national and international laws. You must learn your local laws and choose frequencies within your antenna's range that you are permitted to use, and restrict amplitude to the permitted transmit power (AND ERP which is transmitted power * antenna gain)

3) You have used a very unusual way of expressing an AM waveform. Normally it is expressed as the PRODUCT of two sinewaves : a carrier frequency and a modulating frequency (which is the desired output signal). The frequency you would normally consider for loss is the carrier, which in your case is (f1 + f2) / 2.

Note that f1 and f2 are usually very close together; the modulating frequency (baseband signal) can be expressed as either (carrier - f1) or (f2 - carrier) or (f2 - f1) / 2. F1 and f2 are called the "sidebands" around the carrier. Your chosen frequencies in step 2 must both be within the same legal "channel" which may be only a few kHz wide.

4) You probably want to study the Friis equation