Raspberry Pi, too much load on GPIO
To day I use two power sources and a relay between RPI and the remote
  control(green card) since I don't want to accidentally fry my RPI.


That's a good policy. Most cell-phone 'chargers' can't actually deliver their rated current reliably - if at all. Using a dedicated supply for the Pi makes it much less likely to suffer from brownouts or surges caused by an overloaded power supply.


  Is it safe to share one power source for both RPI and my remote
  control circuit, regardless if I add more power hungry crap on the
  non-RPI side of the relay?


IMO, no. The advantage of your current setup is that it maintains galvanic isolation between the Pi and all the 'power hungry crap' that might damage it. Using a common supply will make the system less safe.