Why pets are allowed only on select flights?
The USDOT has a list off things related to this matter here but there are many factors.

International Travel: Since flights often go between countries that also means they cross jurisdictions. Many countries have varying laws on importing and exporting different animals. As such airlines may not always allow you to carry animals since it may not be legal.
Contamination: Animals can carry a wide variety of disease and other things. An airline my chose to not carry animals to avoid any potential issues. Even if in the cargo hold it can not be guaranteed that the animals fur wont come in contact with another passengers baggage. If that passenger had an allergy to that type of fur they may later have an issue.
Food: Placing an animal in the baggage hold prevents anyone from effectively feeding the animal which the DOT does seem to have some regulations on for longer flights.
Temperature: Although it is pressurized the baggage compartment is by no means warm. This prevents a potentially dangerous situation for animals as well.

As for bringing an animal on the plane (aside from service animals which are not considered animals). It would seem that the FAA leaves this specific matter up to the airlines. According to this document when it comes to

What are the FAA rules regarding traveling with pets in the passenger cabin?
The Federal Aviation Administration (FAA) allows each airline to decide if they will allow you to travel with your pet in the passenger cabin. If an airline does allow you to bring your pet into the cabin, we consider your pet container to be carry-on baggage and you must follow all carry on baggage rules (14 CFR part 121, section 121.589):