figure - fully connected deep neural network model for sentiment classification we call this fully connected deep neural network (dnnbecause neurons or units in each pair of adjacent layers are fully pairwise connected these networks are also known as deep artificial neural networks (annsor multi-layer perceptrons (mlpssince they have more than one hidden layer the following function leverages keras on top of tensorflow to build the desired dnn model def construct_deepnn_architecture(num_input_features)dnn_model sequential(dnn_model add(dense( activation='relu'input_shape=(num_input_features,))dnn_model add(dropout( )dnn_model add(dense( activation='relu')dnn_model add(dropout( )dnn_model add(dense( activation='relu')dnn_model add(dropout( )dnn_model add(dense( )dnn_model add(activation('softmax')dnn_model compile(loss='categorical_crossentropy'optimizer='adam'metrics=['accuracy']return dnn_model from the preceding functionyou can see that we accept parameter num_input_featureswhich decides the number of units needed in the input layer ( for word vec and for glove featureswe build sequential modelwhich helps us linearly stack our hidden and output layers we use units for all our hidden layers and the activation function relu indicates rectified linear unit this function is typically defined as relu max , where is typically the input to neuron this is popularly known as the ramp function also in electronics and electrical engineering this function is preferred now as compared to the previously popular sigmoid function because it tries to solve the vanishing gradient problem this problem occurs when and as increasesthe gradient from sigmoids becomes really small (almost vanishingbut relu prevents this from happening besides thisit also helps with faster convergence of gradient descent we also use regularization in the network in the form of dropout layers by adding dropout rate of it randomly sets of the input feature units to at each update during training the model this form of regularization helps prevent overfitting the model