Can I use OS X Server 4 to remotely manage an OS X Server 3.2.1?
No you can't. 


Server 3 won't run on Yosemite.
So management of remote Server 3 is not possible with Yosemite client.
Server 4 client on Yosemite alerts that it "will only connect to remote OS X Server running on OS X 10.10"
In other words: both Server client app and Server server need to be at Server 4 on OS X 10.10 if either end switches over to Yosemite.