Tensorflow classification - maximize the accuracy of certain classes
The best approach for this problem would be to change the value of X. For example, if you are using -%2 and %2 as the classification limit such as  sell(&lt;-%2), buy(>%2) and no action otherwise; you can reduce this to %1, which will in turn reduce the number of samples that fall into this class while increasing number of samples in buy and sell classes.

If you want to keep that value, the way to go is undersampling: when training you take equal number of samples from each class, meaning you ignore some portion of the data from no action class. So you training and validation sets will be balanced and you can use accuracy as a proper metric.

If this also is not a good option for you, another way would be to try changing the classification threshold for each output so that their possible outcomes are roughly equal. But this might be a little complicated to implement as you have 3 classes (for 2 classes this can be done in a relatively simple way by changing classification threshold from 0.5).

If none of these suit you, you can find a more general approach to imbalance problems here. In case you wish to use a different metric, the most popular ones are listed and discussed in detail here. You probably don't need to define a custom metric because there is a one for pretty much every choice.