Does input and output size vary by address type? For instance, we have two outputs, output X sends to a P2SH address, and output Z sends to a P2PKH address.

Would output X have a different transaction size than output Z? What about the size of the inputs, is that affected by the address type?
Both outputs and inputs differ in size depending on the address format.

P2PKH outputs are 34 bytes and P2SH outputs are 32 bytes.

P2PKH inputs are 148 bytes (180 bytes for uncompressed keys), and P2SH inputs are of variable size depending on the redeem script they need to fulfill. For example, a 2-of-3 multisig P2SH input (a very common P2SH use) has 297 bytes.