How safe is IFR in class E?
As with any flight, "It's as safe as you make it".

The vast majority of the General Aviation fleet (piston singles) don't even have an autopilot, and the ones that do may only be single-axis units (wing levelers), so the pilot has to have some level of attention on flying the airplane.
If we assume these are the folks flying around VFR we can also generally assume they should be looking out the window the vast majority of the time (like Skip said, "Put down your iPads and do your job.")

Adding to that, "See and avoid" is every pilot's responsibility in VMC - whether operating under Visual Flight Rules or Instrument Flight Rules - if you can see, you are expected to avoid (and if necessary deal with ATC freaking out about you deviating from a clearance later).



A number of other steps have been taken to enhance safety in Class D and E airspace:

In the US FAR 91.159 prescribes specific VFR cruising altitudes, and FAR 91.179 is the IFR equivalent.
The cruising altitudes in these regulations result in an automatic 500-foot vertical separation of VFR and IFR traffic (at least between 3,000 feet MSL and 18,000 feet MSL - which is where VFR and IFR traffic are likely to be mixing in cruise. ).

Additionally IFR traffic is usually receiving radar separation services (there are some non-radar areas in the US, but not many). Nominally this separation is "IFR-to-IFR" in Class E airspace, but if a controller sees a VFR target squawking 1200 and behaving unpredictably they can vector IFR traffic around it.
Similarly if a controller sees a "primary target only" (something without a transponder, and not reporting altitude information) they will often vector IFR traffic around it until they can establish what it is and what it's doing.

Within class D airspace the level of service you'll get depends somewhat on the equipment the controllers have.  A "purely visual" tower with no radar can only give traffic advisories for aircraft they can see by looking out their window, but if they see a conflict they're going to say something. A tower with radar (either local or slaved from a nearby airport) may give VFR-to-VFR or VFR-to-IFR separation advisories based on that data as well, sometimes including advisories to aircraft inbound who are outside their airspace.