they're geometric transformations repeatedly applied inside for loop the temporal for loop is itself hardcoded by human developersit' built-in assumption of the network naturallyrnns are still extremely limited in what they can representprimarily because each step they perform is differentiable geometric transformationand they carry information from step to step via points in continuous geometric space (state vectorsnow imagine neural network that' augmented in similar way with programming primitives--but instead of single hardcoded for loop with hardcoded geometric memorythe network includes large set of programming primitives that the model is free to manipulate to expand its processing functionsuch as if brancheswhile statementsvariable creationdisk storage for long-term memorysorting operatorsadvanced data structures (such as listsgraphsand hash tables)and many more the space of programs that such network could represent would be far broader than what can be represented with current deep-learning modelsand some of these programs could achieve superior generalization power we'll move away from havingon one handhardcoded algorithmic intelligence (handcrafted softwareandon the other handlearned geometric intelligence (deep learninginsteadwe'll have blend of formal algorithmic modules that provide reasoning and abstraction capabilitiesand geometric modules that provide informal intuition and pattern-recognition capabilities the entire system will be learned with little or no human involvement related subfield of ai that think may be about to take off in big way is program synthesisin particular neural program synthesis program synthesis consists of automatically generating simple programs by using search algorithm (possibly genetic searchas in genetic programmingto explore large space of possible programs the search stops when program is found that matches the required specificationsoften provided as set of input-output pairs this is highly reminiscent of machine learninggiven training data provided as input-output pairswe find program that matches inputs to outputs and can generalize to new inputs the difference is that instead of learning parameter values in hardcoded program ( neural network)we generate source code via discrete search process definitely expect this subfield to see wave of renewed interest in the next few years in particulari expect the emergence of crossover subfield between deep learning and program synthesiswhere instead of generating programs in generalpurpose languagewe'll generate neural networks (geometric data-processing flowsaugmented with rich set of algorithmic primitivessuch as for loops and many others (see figure this should be far more tractable and useful than directly generating source codeand it will dramatically expand the scope of problems that can be solved with machine learning--the space of programs that we can generate automaticallygiven appropriate training data contemporary rnns can be seen as prehistoric ancestor of such hybrid algorithmic-geometric models licensed to