Why isn't Replace by fee default for all transactions? I know it's a wallet implementation and a flag, but when it was introduced why wasn't this change a rule for all transactions so that anyone could RBF if they wanted to instead having had to set the mandatory flag?
It's been over six years and I've done entirely too little research to state the below with confidence, but what I can roughly piece together quickly between cursory research and from what I remember:
There was a big debate about the merits of RBF since at least 2013, when Peter Todd and John Dillon provided a Full RBF patch. Gavin Andresen described it as anti-social and among others pushed for a First-seen-safe RBF variant which would require a replacement transaction to include all outputs of the replaced transaction. It was pointed out that FSS RBF would only protect transactions that only spent confirmed inputs (since transactions that spent unconfirmed inputs could be invalidated by replacing the parent) and that each replacement would add another change output to the transaction adding UTXO bloat.
Regardless, the full RBF proposal saw strong push-back, especially by a number of services built around zero confirmation payments (IIRC e.g. Satoshi Dice, BitPay). The 2015 compromise was then to require transactions to opt-in to Full RBF and to require that replacements pay not only more fees than the replaced transactions but more than their entire descendant tree to curb DOS attacks. Standard transactions were left unchanged in their behavior.
Obviously, RBF is just a mempool policy and cannot be enforced, so nothing would prevent nodes (or even miners) from individually switching to running a Full RBF mempool policy.