sharding and replication on mongodb?
My understanding is that each shard is a physical machine. holding below two sets of data(primary and secondary) and host two separate mongod server/instance. 


No, each replica-set member is a physical machine. In the above graphic, each shard consists of two physical machines. While you can technically run multiple members of a replica-set on the same machine, there is nothing to gain from that.


  . One will hold primary data and second will hold secondary data which is back up of primary data on another shard


No, the secondary will back up the data from the other member(s) of the same shard. But if you want to build the server-level equivalent of a mirroring RAID, you could of course put two members of different shards on the same physical machine.


  i believe i need to start only 1 mongos instance


You only need one, but to avoid a single point of failure, you might want to have more than one.


  Similarly I need to have single mongo arbiter for all 3 shards instead of 3 on each shard?


An arbiter can only be member of a single replica-set. So you need 3 arbiters. However, the arbiters are very lightweight. They don't hold any data and don't do anything unless there is a primary election. So you do not need a dedicated server for the arbiter processes.