How to replace Microsoft Small Business Server 2003 with Mac Mini Server (Mountain Lion)
Yes, in OSX 10.8 Server the alternative to AD is called Open Directory.  There's a fantastic resource here;

https://help.apple.com/advancedserveradmin/mac/10.8/

You'll want to read up on the Manage Users section, but in short you can import all of your data from AD into Open Directory so that both Macs and PCs can authenticate against your OSX Server.  As you already have a NAS you can also store all user directories there too (home folders / my documents).

Its an involved task to do this, so its probably too long to answer here, but that resource from Apple gives you everything you need to know.