sensor to detect rope position, as it nears the end of the line retreval
There are several types of sensor you could try.

Tie a knot in the rope. Put a lever operated microswitch next to the rope inlet on the winch. 

Wind some iron wire round the rope (assuming it's not steel rope). Perhaps use a sleeve of heatshrink over the top to hold it in place. Then arrange a magnetic circuit with a permanent magnet and a linear Hall effect sensor, such that the output of the sensor changes as the iron changes the reluctance of the magnet path. Use an opamp or comparator to sense the change in Hall output.

Wind some copper wire round the rope, and solder the start to the end to make a shorted turn (assuming it's not a steel rope, but with care it might even be got to work on steel). Use a high frequency oscillator and a search coil close to the rope, to sense the shorted turn. Sense either a drop in level of the search coil, or coupling to another coil.

For both of these schemes, sensitivity and rejection of false triggers will be improved by using a pair of sensors side by side, sharing the same exciter in the middle, and sensing them differentially. Common mode interference and drift will be rejected as the sensors are differential. As the rope comes in, first the outboard sensor will react, then the inboard. Look for this sequential signature. 

I assume that reflective sensors would quickly be blinded by mud.