Autoencoder for anomaly detection from feature vectors
Does that mean that my model (or indeed my approach of using an AE) is ineffective


Well, it depends. Auto Encoder are a quite broad field, there are many hyperparameters to tune, width, depth, loss function, optimizer, epochs. 


  How should I proceed?


From my gut feeling I would say that you don't have enough data to train the AE properly. Keep in Mind, the MNIST database contains 50,000 image. And you need enough variance in order to not overfit your training data. Tree based approaches are, at least in my experience, easier to train. If you like to stick at the anomaly detection part, which I recommend since you don't know what anomalies you will face, try the Isolation Forest Algorithm. But for a solid recommendation I would need to know how your data looks. 

Btw, A good metric to use in such a case is  the ROC score, which basically tells you how likely it is that your model will classify new data points correctly. Check out the link for an visual explanationROC explained 

So Baseline is, try less complex approaches until you a certain that they are not sufficient enough.