In daily casual conversations in Mandarin are silent initials often ommitted or spoken?
Pinyin is just a guide and cannot be taken literally. Most initials should be taken fairly straightforwardly though. 

Y on the other hand shouldn’t.

If you look at the ipa for 意:


  /i⁵¹/


there is only an: i.

Y basically equals i.

Next look at the ipa for 因:


  /in⁵⁵/


Again y basically equals i.

So, yi is easily said that the y is silent but if you take something where the final doesn’t start with i like 羊/阳/洋 ipa becomes:


  /i̯ɑŋ³⁵/


It still maintains this i sounding initial.