How to perform model selection for One-Class Classification?
Have you read Janssens' dissertation "Outlier Selection and One-Class Classification"? He has a chapter on evaluation which may be of use. Have you thought about artificial generation of negative instances? 

I had to deal with OCC evaluation awhile back and never came upon an entirely satisfactory solution. As I recall, the basic problem boils down to the fact that you don't really have labelled instances of the negative class. Without these instances there is no way of computing precision = TP/(TP+FP) which relies upon the count of incorrectly accepted negative instances (FP). This just leaves you with recall = TP/(TP + FN) as a computable metric - which by itself is next to useless in evaluating classifier performance i.e. it's trivial to write a classifier that get's 100% recall, just predict the positive class for every instance in your test data... you will correctly identify every true positive instance, but you will also incorrectly predict every negative instance as a positive instance. What is typically required is a balance or trade-off between precision and recall.

You didn't mention a particular problem domain in which you are working, but perhaps you can leverage this domain knowledge as a work-around to the problem of OCC evaluation. This particular problem I will relate is technically cast as a PU Learning problem (learning from Positive and Unlabelled instances). I'll try to describe some solutions that I explored while tackling this challenge - I'll give you fair warning that they are not entirely satisfactory solution, but I believe they are logically sound and somewhat defensible. From a purely pragmatic perspective, they are definitely better than the alternative.

Implicit Negative Class

Scenario: imagine a membership-based incentive program, in which people can pay an annual fee to become a member of Foobar Inc. Members, but not non-members, are entitled buy widgets at Foobar's retail outlets.

You are in charge of a marketing drive to increase Foobar's membership. You have limited funds and reach, so you need to carefully select people from the general population that you believe will most likely become members if you target them with a promotional campaign. In this scenario you have a database of known Foobar member's (the labelled, positive class data, which we label POS) and a database of known non-members from the general public. This is your unlabelled data that contains people who may or may not become members if targeted. All we have is POS instances and UL instances -- there are no labelled NEG instances. 

Depending upon your particular problem, context, and domain, you may be able to transform the OCC/PU problem into a more classical problem with standard performance metrics. In the situation I just described, there are at least three ways of creating an implicit negative class:


Use the spy technique to generate "reliable" negative instances
that can be placed in your test set; 
If historical campaign data is available you could identify those
individuals who were targeted multiple times for campaign
membership, but who rejected it every time -- these could be
"consistently negative" instances; 
If your known membership pool consists of members of varying
value to the business, you could divide members into good/bad, and
use the bad instances as a proxy for NEG instances.


One final area you may want to explore is evaluation in collaborative filtering research. Collaborative filtering on unary responses (e.g., Facebook "like") 


  can be considered the matrix completion analog of the
  positive-unlabelled (PU) learning problem in classification and
  regression modelling. (Aggarwal, 2016)


Perhaps this field has developed some novel approaches to OCC evaluation that you can leverage. Without knowing the details of your particular case, however, its hard to provide a more detailed solution. Please update this thread with your eventual solution -- there is so much more research that needs to be done in this particular problem domain.