What semantic notions underlie 草 (grass) with 拈花惹草 (philandering)?
The focus in 拈花惹草 is not 拈花 (touch flower), but 拈野花 (touch wildflower) and 惹草 (contact grass) actually mean (touch wild grass).
野花野草 (wildflower and  wild grass) is a collective noun for 'wild plants' (outside women) in contrast to 'homegrown plant' (your wife)