Can I use Opto Isolator or Opto Coupler instead of Relay to operate AC Source?
If your load is AC AND low power then some opto-isolators will work such as the MOC3021. It has a triac output capable of driving a low power load in the order of a few mA but, more likely you would want to drive a more powerful device such as this: -



The external triac would typically be a BT136 and there are a pile of google images here that show this type of configuration. The BT136 comes in a T0-220 package that isn't that small but maybe some space saving can be achieved. A lower power BT131-600 could handle 100 watt loads but RMS current is limited to below 1 amp. It is available in a T0-92 package.

You could also consider using a solid-state-relay although a lot of them will contain pretty much the above Triac circuit or back-to-back MOSFETs: -

http://www.bristolwatch.com/ele2/acdcssr/4a.png