How can I build a custom stair case in a small space?
First of all, since it sounds like this could be considered "occupied space", you better check your local building codes and/or with your local inspector to see what is allowed. Most codes have specific requirements for stairs, and in some jurisdictions strange things like spiral stair cases may not be allowed or may require specific approval because of their narrow treads.



I'd definitely look to see if there is any way to get more space at all. For example, what is behind the closet? If you can lose a bit of ceiling height there, you could get a bit more room for the staircase. Likewise, is it acceptable to have the stairs coming out into the room at all? This too, will provide a bit more space. Using this calculator, if you can do 10 rises of 8" (this may be steep though), you need 6'8" of total run space. 



Depending on the adjoining space, you might also be able to make use of the room under the stairs.



If you can't get any extra space, you might consider a "ladder staircase" (kind of a hybrid of ladders and stairs):



There are some interesting designs for these, including alternating treads:

 

And some really funky variations if you want to be really creative: