Is the value of this resistor critical and if so why?
Note that the pull-ups are pulling to 3.3 V, the same as the MCU power supply in your linked schematic. Meanwhile the sensors are powered from +5V so their output will be feeding 5 V into a 3.3 V input. It looks like the series resistors on the input are a hack to limit the input current to (5 - 3.3 - 0.6)/1k = 1.1 mA to each input and out to the 3.3 V supply via the chip's input protection diode (which is where the 0.6 V comes in).
This wouldn't be considered high-class design but it probably works. It relies on the chip being able to absorb all the input currents via its supply pin as it probably can't be absorbed back into the PSU. With only three inputs giving up to 3.3 mA it should be OK as the CPU will consume more than that. (Otherwise the 3.3 V rail would rise up towards 5 V.)

My question is about R17. What is its role and is its value critical? Would it make a difference if R17 was 100 ohm?

Yes. 100 Ω would give ten times the input current and I would think that it may overload the input, possibly burn out the protection diode and then the chip is in trouble.

On my board the value of R4 and R13 are both 1k but R17 is 100 ohms and it is the Z- signal that seems to misbehave.

Add 1k in series with the input and see if that helps. It may be too late though. If you think the chip is still working you could make a potential divider on the input with 1k8 and 3k3 resistor to drop the input voltage to less than 3.3 V max.



simulate this circuit &ndash; Schematic created using CircuitLab
Figure 1. Voltage divider for the Z-limit input.