Calculating the Number of flops for a given Neural Network?
FLOPs are the floating-point operations performed by a model. It is usually calculated using the number of multiply-add operations that a model performs. Multiply-add operations, as the name suggests, are operations involving multiplication and addition of 2 or more variables. For example, the expression, a * b + c * d, has 2 flops while a * b + c * d + e * f + f * h has 4 flops.
Let's now take a simple linear regression model for example. Assume this model has 4 parameters w1, w2, w3, and w4 and a bias b0. Inference on an input data, X = [x1, x2, x3, x4] results in output = x1 * h1 + x2 * h2 + x3 * h3 + x4 * h4 + b0. This operation has 4 flops
The FLOPs measurement in CNNs involves knowing the size of the input tensor, filters and output tensor for each layer. Using this information, flops are calculated for each layer and added together to obtain the total flops. Let's look at the first layer in VGG16 with input tensor of size 224x224x3, 64 filters of size 3x3x3 and output size of 224x224x64. Each element in the output results from an operation involving (3x3x3) multiply-add between the filter and the input tensor. Hence the number of flops for the first layer of VGG16 is (3x3x3)x(224x224x64)= 86,704,128
I have a neural network(ALEXnet or VGG16) written with Keras for Image Classification and I would like to calculate the Number of floating point operations for a network. The size of the images in the dataset could varry.
Can a generlized code be written in python which could calculate flops automatically ? or is there any libraray avaialble.
I am working with spyderAnaconda and the defined network is a sequential model.
Thank you.