Would removing the branch delay slots change the instructions set architecture?
The instructions set architecture (ISA) is the contract between the hardware designer and the software designer.  Anything that changes the contract, changes the ISA.

The question you have to answer is: given every possible program written with this particular ISA, do any of them have different behavior (give a different answer) if we remove the branch delay slots.