The complement of s (when s < curve order / 2) Bitcoinlib refers to the requirement that if s &gt; ORDER / 2, then the complement of s should be used instead since it's one byte shorter

This sounds like it's referring to the inverse or negative s value, but I also know complement can be a computer science term. 

The relevant code can be found here (perhaps someone adept with the Bitcoin Core code can chime in on what the code is doing), so, what is the complement of s referring to?
In ECDSA s can be on either side of the curve and the signature will still be valid. The term "low s" is referring to the value literally being being below the curve order (0x7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF5D576E7357A4501DDFE92F46681B20A0). The requirement of low-s is arbitrary (high s would have worked just fine as well), and is one of the new BIP62 validity rules that are intended to combat transaction malleability.

Absent this rule, any person is able to take a Bitcoin transaction, flip s in any of its signatures, and push the transaction out again with a different TXID. Being able to do this only changes the hash of the transaction, and does not alter its validity in any way. Being able to mutate transactions breaks a number of potentially interesting transaction types in Bitcoin like payment channels, where chains of transactions will suddenly be invalidated by a parent being mutated and an alternate form included in a block. 

By forcing valid transactions to always have low s this ability is removed, though a person with the private key for a transaction is still able to mutate their own transactions by resigning them with a new nonce.