AccelStepper library advanced codes!
The setOutputPins() function sets the pins for a pulse for a single stepper. Depending on the used stepper motor a different number of pins has to be set (normal stepper drivers have 2 pins, the ULN2003 and its siblings have 4) in a different way. For that the function first determines the number of pins to use via the _interface member variable of the class.
The mask parameter contains the states of the pins. Each bit represents one pin. If the corresponding bit is set to 1, the pin will be set to an active state. If not, it will be set to an inactive state. Normally an active state (like I have called it here) is HIGH, an inactive state is LOW. But when working with different stepper drivers, some pins might work with an inverted logic. Thus the value (LOW or HIGH) is inverted, if the pin is marked as inverted in the _pinInverted array. This logic is in the line
digitalWrite(_pin[i], (mask &amp; (1 &lt;&lt; i)) ? (HIGH ^ _pinInverted[i]) : (LOW ^ _pinInverted[i]));

The idiom condition ? expression : expression represents an embedded if condition. If the condition evaluates to true, the first expression is used, else the second expression. With mask &amp; (1 &lt;&lt; i) we are checking, if the ith bit in mask is set. If yes, we use the next expression: HIGH ^ _pinInverted[i]. Here the ^ bitwise XOR operator is used to invert the value, if _pinInverted[i] is set to 1 (and thus marked the pin as inverted). If _pinInverted[i] is zero, the value will stay as it is. The handling of the LOW value is similar.
For understanding that better, you might want to read more about bitwise operators, if you haven't already done so.

why is it [the mask parameter] using uint8_t?

uint8_t is the smallest type, that could be used: 1 byte. As each bit represents one pin, and the library supports steppers with up to 4 pins, only 4 bits are needed at max. Thus using the smallest type with 8 bits is sufficient.

Is this line sending a step signal to multiple stepper?

No, it is setting the pins of a single stepper motor. Each instance of the AccelStepper class represents a single stepper motor. Thus every function in it also only talks to one single motor. You haven't really explained, how you want to add the feedback functionality. But you should not change the AccelStepper class to handle multiple motors. That does not make sense to me. Each motor has to be controlled individually and this class is there to capsule the work behind that. Thats the idea behind making a class for this.
I guess you want to drive multiple motors, each with it's own feedback loop. So then you first need to change the AccelStepper class for supporting that feedback loop. How this is done depends very much on your type of feedback loop. Also the setOutputPins() method is not a good place for that. It is a very low level function. You need to start further up in the call chain, maybe in the step() method or the runSpeed() method (this really depends on how you want to handle the feedback). Also you need to write methods to set up and handle the encoders input. You may need to configure timers and/or interrupts for reading them correctly.
For then supporting multiple stepper motors, you could still look into the MultiStepper class. You could change it to fit with your changes in AccelStepper or write your own wrapping class. That mostly depends on how you want the multiple steppers to behave.