How to find pinout of DC geared motor with encoder?
The two large solder blobs either side of the encoder wheel are the motor terminals. These appear to be going to the black and red wires via B2 and B3 (which are probably low value inductors used to block brush noise). Connect your multimeter to these wires and measure resistance, which should be a few ohms. Then measure DC volts while spinning the motor by hand, which should produce a small voltage whose polarity depends on the motor direction.
The other 3 wires are the Hall effect ground, power supply, and output signal. You have already determined that the brown wire is ground, which makes sense because it has a thick trace going off it. B1 and C5 are probably a power supply filter, so put your meter on continuity test measure between B1 and the Orange and blue wires. One of these should be a direct connection to B1. I'm betting on Orange because the blue wire appears to go to R1, which would then make it the signal wire.