What are the drawbacks of being a helicopter and fixed-wing pilot at the same time?
In case of an engine failure, the fixed wing pilots natural reaction is to push forward to lower the nose, then look for the correct glide airspeed. A helicopter pilot in most engine failures should pull back (although not agressively) to load the disc and restore, or increase rotor RPM.  Maintaining rotor RPM is the only way to get to the ground safely so flaring is an instinctive reaction trained in.  You don't look for airspeed or rotor RPM.  You dump the collective and flare, then have a look at airspeed and RPM.  Pushing forward will reduce rotor RPM which in some flight phases and some helicopters might not kill you but in many helicopters at altitude may well kill you.  In addition, a stall buzzer will prompt the fixed wing pilot to push the stick forward and increase power, which in a helicopter means pulling the collective.  In a helicopter the low rotor RPM buzzer sounds just like a stall warning but pushing the stick forward and pulling the collective will almost certainly kill you.

Another action that has to be instinctive is to stick in a boot full of right pedal (counter clockwise rotor) which is totally alien to a single engine fixed wing pilot in an engine failure and completely the opposite of what a twin engined fixed wing pilot might do, who will naturally "step on the failed engine".  In a helicopter, that can get you into a very bad attitude rapidly from which it might be impossible to recover.

Negative G in an aircraft is not a problem. In a 2 bladed helicopter, you must gently pull back to reload the disc.  Pushing forward has a real risk of a blade chopping off the tail or hitting the canopy. It might also cause "mast bumping" which can cause the rotor to separate from the aircraft.  Too many helicopter pilots have been killed this way. Most helicopter pilots avoid -ve G.  The main point here is that you do not push the stick forward to level out from a climb.  The instinctive reaction for a helicopter pilot is to to pullback as soon as they feel the slightest hint of rising out of their seat.

In a helicopter, to accelerate, you push the stick forward then increase power to maintain altitude or climb.  In an aircraft, you increase power, then push the stick forward to maintain altitude.

A running landing, and not a hover landing, requires you to push the stick forward to level the skids.  A fixed wing tricycle gear pilot wants to pull the stick back to flare which again, might chop off the tail.

Keeping trim (balance) is a function of power and not direction of turn.  It is perfectly possible to make a properly executed balanced turn in a helicopter with the opposite pedal pushed in.

Helicopters can slow down with a nose down attitude and you take off by pushing the stick forward to accelerate.

I'm sure there are others which I've forgotten but I think these are the main things and, most importantly, as I've shown, trying to fly a helicopter like a fixed wing can be deadly.

In summary, flying helicopters is very different to flying fixed wing and you need different responses which need to be intuitive, especially in emergencies.

My gut reaction response is that it is easier to transition from helicopter to fixed wing and that the lower time pilots will find it easier because the years of experience and muscle memory are not there.