Why am I receiving a Modem Status (0x8A) on broadcast messages (XBee)?
It seems as if you did not send a correct API broadcast package. You sent

7e 0 10 17 0 0 0 0 0 0 0 ff ff ff fe 1 43(C) 42(B) 1 66

Where the first byte is the start delimiter for an API packet (0x73) - correct

the next two bytes are the length, which is 16 (0x0010) - correct

The next byte is the API identifier value for AT command request (0x17) - correct

The frame ID is also correct as already mentioned in your post - correct

The 64 bit destination address (0x000000000000FFFF) - correct

Destination network address (0xFFFE) - correct

The command options must either be 0x02 or 0x00. All other bits must be 0. You have 1 which means that bit 0 is set which should not be the case. I assume the reset is due to this wrong value. Change that to 0x02 and you will be fine.