Is it possible to run an arbiter on a machine that also hosts a secondary member?
Is it possible to run an arbiter on system that also hosts a secondary member?


There is no technical restriction on running multiple processes on the same host, as long as they are listening to different port numbers. For example, you could start an arbiter listening to port 27018 (instead of the default 27017). However, you do need to consider the operational implications of colocating processes.


  Even if it is not advisable, but if it is possible, how can I install an arbiter on the secondary server for when the primary go down the secondary become the primary?


This is the standard replica set behaviour. Assuming you have a preferred primary (which will have a higher priority value), your replica set configuration would look similar to:

{
   _id : "myreplset",
   members: [
      { _id: 0, host: "machine1.example.net:27017", priority: 2 },
      { _id: 1, host: "machine2.example.net:27017" },
      { _id: 2, host: "machine2.example.net:27018" }
   ]
}


With three voting members in your replica set, the strict majority required to elect (or maintain) a primary is  two votes. If machine1 is unavailable, the members on machine2 can still elect a primary.

However, because machine2 has the majority of the voting members there are a few operational consequences:


If machine2 is unavailable, machine1 cannot elect or sustain a primary without manual intervention to force reconfigure the replica set.
In the event of a network split, machine2 will become primary.


If the arbiter (or another voting member) was on a third machine, both of these scenarios would be avoided. 

You may want to consider colocating the arbiter on machine1 , so in the event of a network split the primary will remain on machine1 (reversing the operational scenarios noted above so that failover to machine2 would be a manual reconfiguration).

For a production deployment I would also recommend using a third secondary (instead of an arbiter). When your Primary-Secondary-Arbiter deployment is down a member (so effectively Primary-Arbiter) there is no longer any active replication and your deployment will not be able to acknowledge majority writes. It is definitely not ideal to have multiple data-bearing members competing for resources on the same host machine, but data redundancy and availability of majority writes is often a more important factor.


  Once installed, how can I configure the replica set for when the original primary returns and make a copy of the secondary?


Assuming you have configured the member on machine1 with a higher priority as per the above example, the expected outcome is that when machine1 is available again it will automatically resume syncing from the current primary on machine2 (as a secondary). Once the member on  machine1 catches up and is eligible to be primary, it will initiate an election based on a higher priority member being available (referred to as a priority takeover election).