Deep Learning: Feed Forward for Unbalanced Classes Using Tensor Flow In theory, Deep Learning NN can predict a class with very few observations.  My problem, I have a class that happens less than 4% of the time.  Feeding the network data with distribution intact (96 one class and 4% the other class) results in the network predicting mostly the most common class.  Using under sampling result in a model that is able to predict 90% of the less common class, but still have a high False Positive.
I know that using F Score (with beta >= 2) in the training phase of the model will allow me to improve significantly the model performance. My question, how or where do I set my own performance metric in Tensor Flow?
Any suggestions are appreciated.
By the way, I did tune all the parameters (learning rate, momentum, etc..) performance increases but still not at the level I want.
My network has 20 inputs 5 hidden layers with 60 unites
My batch is 100 (tried up to 500), and I tried epoch from 100 to 1,200  

The code is in Mathematica.

 netDrop1 = 
     NetGraph[&lt;| "i1" -&gt; BatchNormalizationLayer[], 
    "l1" -&gt; CatenateLayer[], "l12" -&gt; DropoutLayer[0.2], "l2" -&gt; 60, 
    "l21" -&gt; DropoutLayer[0.5], "l3" -&gt; Tanh, "l22" -&gt; 60, 
    "l221" -&gt; DropoutLayer[0.5], "l33" -&gt; Tanh , "l23" -&gt; 60, 
    "l234" -&gt; BatchNormalizationLayer[0.5] , "l34" -&gt; LogisticSigmoid, 
    "l24" -&gt; 60, "l2435" -&gt; DropoutLayer[0.5], "l35" -&gt; Ramp, 
    "l45" -&gt; 60, "l435" -&gt; DropoutLayer[0.5], "l36" -&gt; Tanh, 
    "l55" -&gt; 60, "l37" -&gt; Tanh, "l4" -&gt; 2, 
    "l5" -&gt;  SoftmaxLayer[]|&gt;, {NetPort["Input1"] -&gt; 
     "i1" -&gt; "l1" -&gt; 
      "l12" -&gt; 
       "l2" -&gt; "l21" -&gt; 
         "l3" -&gt; "l22" -&gt; 
           "l221" -&gt; 
            "l33" -&gt; 
             "l23" -&gt; 
              "l234" -&gt; 
               "l34" -&gt; 
                "l24" -&gt; 
                 "l2435" -&gt; 
                  "l35" -&gt; 
                   "l45" -&gt; 
                    "l435" -&gt; 
                    "l36" -&gt; 
                    "l55" -&gt;  "l37" -&gt; "l4" -&gt; "l5", {{NetPort[
        "MaritalStatus"], NetPort["Gender"], NetPort["Input"]} -&gt; 
       "l1"}}, "Input" -&gt; 11, "Input1" -&gt; 2, 
       "MaritalStatus" -&gt; NetEncoder[{"Class", marital, "UnitVector"}],
       "Gender" -&gt; NetEncoder[{"Class", {"M", "F"}, "UnitVector"}],
       "Output" -&gt; NetDecoder[{"Class", {"no", "yes"}}]]


From reading publication about NN, I have decided to create a Dropout with = 0.2 at the input level, and at all subsequent levels to have Dropout at 0.5.
I have used both Tangent and ReLu (but found little differences in performance between them)
As you notice, a simple FeedForward Network

I do under sampling, the under sampling with great success:  the model is able to recall most of the yes, but still predict many negative samples as positive.

  &lt;|{"yes", "no"} -&gt; 972, {"no", "no"} -&gt; 1082, {"yes", "yes"} -&gt; 
  81, {"no", "yes"} -&gt; 3|&gt;


By the way, I have used these same data,and used RandomForest with a utility function with great results http://femvestor.blogspot.com/search?q=geico
To Improve the performance of the network, I added a Batch Normalization layer, and other variables that I had to remove when using Random Forest.  One of these variables is a categorical variable with 51 levels.  I made sure to add an embedding layer that will represent the values by a vector of size 10.
As the original dataset had 15 numerical variables and 9 categorical variables, I have decided to also perform dimension reduction using auto-encoding.