Why are special instructions used to implement system calls? Why not use normal subroutine calling instructions?
System calls could be done, and have been done, with normal (normal enough) call instructions. However, the operation wouldn't be a normal call: the privilege level changes, and it is unsafe to allow calling arbitrary offsets (which might not be the entrypoint of a subroutine) with a lot of privileges. So there should be some mechanism that both limits the possible destinations and changes the privilege level. That mechanism is similar in function to how a dedicates system call instruction would work, but it could be triggered by a normal subroutine call instead.
Call gates on x86 are an example of that, a mechanism that allows system calls to some locations that have been set up in advance, by using relatively normal far calls. 32bit code does not normally use far calls though, so it's not really the same thing as a completely standard subroutine call. Far calls were more usual in the 16bit era, and the 80286 which introduced call gates was still a 16bit processor, but it seems that call gates did not see much use on the 80286. Similar mechanisms existed on some other old processors.