Show HN: NN-512 – Generate standalone C code for neural nets
NN-512
IntroductionNN-512 is a compiler that generates C99 code for neural net inferenceIt takes as input a simple text description of a convolutional neural net inference graphIt produces as output a stand-alone C99 implementation of that graphThe generated C99 code uses AVX-512 vector instructions to perform inferenceThe generated C99 code is human-readable and should be compiled with GCC 9.1 or laterEarlier versions of GCC may also be used, yielding slightly inferior object codeThe generated C99 code has no dependencies outside the C POSIX libraryNN-512 is a Go program with no dependencies outside the Go standard libraryThe NN-512 compiler executable is stand-aloneNN-512 performs a variety of inference graph optimizationsFusion of elementwise operations into adjacent operationsFusion of similar convolutions (as needed for, e.g., ResNet)Removal of concatenations (as needed for, e.g., DenseNet)End-to-end planning of memory layoutNN-512 generates specialized code for each tensor operationGuided by a description of the target CPU cache hierarchyThread-level parallelism is maximized while limiting synchronization costsSimplified code is generated for tensor edges, exploiting tile/vector overhangComplete knowledge of memory layout simplifies addressingMain FeaturesGenerates efficient code for convolutions with arbitrary filter shape, stride, and dilationThe input data tensor is split into disjoint subtensors modulo the stride, while being packed for the cacheThe weight tensor is split similarly, taking dilation into accountCorresponding subtensors are multiplied, with accumulation at heightwise offsetsThe output data tensor is formed by combining accumulators at widthwise offsetsExample: Filter1x7 Stride1x4 Dilation1x1Example: Filter3x6 Stride3x4 Dilation3x1Example: Filter8x6 Stride4x1 Dilation1x1Example: Filter4x4 Stride2x2 Dilation3x3Example: Filter7x7 Stride4x3 Dilation2x1Generates very efficient code for Stride2x2 convolutions (in particular, Filter7x7 Stride2x2)Fourier convolution with a 16x16 tile; four 8x8 FFTs per tile, interleaved modulo the strideFourier-domain data is packed for persistence in L2 cacheFourier-domain weights are streamed in half-precision to conserve memory bandwidthThe 16x16 tiles are deinterleaved, multiplied, and accumulated (yielding 8x8 tiles)The inverse transform (IFFT) operates on 8x8 tiles, four at a timeExample: Filter3x3 Stride2x2Example: Filter4x3 Stride2x2Example: Filter5x5 Stride2x2Example: Filter7x7 Stride2x2Example: Filter8x9 Stride2x2Generates very efficient code for Filter3x3 convolutionsWinograd-Cook-Toom-Lavin convolution with an 8x8 tile4-way tile transforms fully utilize 512-bit vector registers (and amortize transposition costs)Winograd-domain data is packed for persistence in L2 cacheWinograd-domain weights are streamed in half-precision to conserve memory bandwidthExample: Filter3x3 small tensorExample: Filter3x3 medium tensorExample: Filter3x3 large tensorGenerates very efficient code for Filter1x1 convolutions (including those that are not Stride1x1)Single-precision matrix multiplication (making full use of the large vector register file)The input data tensor is subsampled and packed for persistence in L2 cacheThe weight tensor is packed and streamed for broadcast multiplicationExample: Filter1x1 Stride1x1Example: Filter1x1 Stride2x2Example: Filter1x1 Stride3x3Example: Filter1x1 Stride4x4Generates efficient code for various other tensor operationsExample: Fully connected (half-precision weights)Example: Max pooling (3x3 window, 2x2 stride)Example: Avg pooling (3x3 window, 2x2 stride)Example: Max pooling (2x2 window, 2x2 stride)Example: Avg pooling (2x2 window, 2x2 stride)Example: Global max poolingExample: Global avg poolingExample: Softmax (small channel)Example: Softmax (large channel)Integrates elementwise operations directly into the code generated for more complex tensor operationsBatch normalization is implemented by modifying weights and biases during packing (if possible)Remaining elementwise operations are applied to data already present in registers (during inference)Example: Convolution (Filter1x1 Stride1x1)Example: Convolution (Filter3x3 Stride1x1)Example: Convolution (Filter7x7 Stride2x2)Example: Convolution (Filter4x4 Stride1x1)Example: Pooling (3x3 window)Example: Pooling (2x2 window)Example: Pooling (global)Example: Fully connectedDownload (Version 35)NN-512 compiler source code: NN-512.src.tar.gz (go1.16.6)NN-512 compiler executable: NN-512.bin.tar.gz (linux/amd64)Browse Source CodeSource code for the NN-512 compiler (go1.16.6)Generated code for ResNet50 (C99)Generated code for DenseNet121 (C99)Generated code for ResNeXt50 (C99)DocumentationSyntax and semantics of the NN-512 graph languageEmail37ef.ced3@gmail.comLicenseNN-512 (https://NN-512.com)Copyright (C) 2019 [    37ef ced3 3727 60b4    3c29 f9c6 dc30 d518    f4f3 4106 6964 cab4    a06f c1a3 83fd 090e]All rights reserved.Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditionsare met:1. Redistributions of source code must retain the above copyright   notice, this list of conditions and the following disclaimer.2. Redistributions in binary form must reproduce the above copyright   notice, this list of conditions and the following disclaimer in   the documentation and/or other materials provided with the   distribution.THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOTLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FORA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHTHOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOTLIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANYTHEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USEOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.