How can I create a classifier using the feature map of a CNN? I intend to make a classifier using the feature map obtained from a CNN. Can someone suggest how I can do this?

Would it work if I first train the CNN using +ve and -ve samples (and hence obtain the weights), and then every time I need to classify an image, I apply the conv and pooling layers to obtain the feature map? The problem I find in this, is that the image I want to classify, may not have a similar feature map, and hence I wouldn't be able to find the distance correctly. As the order of the features may by different in the layer.
You can use restricted boltman machine to obtain feature map, and then build a CNN with this architecture : conv layer -> pooling->conv layer pooling-> ....->fully connected layer.   Add conv layer and pooling layer until the matrix size reduced into $1\times 1$, then treat them as input for your fully connected layer, then train the fully connected layer