# embeddings_queue

created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4b2-6db4-11ee-88cf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-67", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "can be managed by one thread. However, processing requests individually is inef\ufb01cient computationally, for two main reasons. Firstly, when processing requests individually, the processor must\nload all the weights of the network for each request. This lowers the arithmetic intensity of the workload, and tends to make the computation memory bandwidth bound, as it is dif\ufb01cult to effectively\nuse on-chip caches when requests are presented individually. Secondly, the amount of parallelism\nthat can be exploited to classify one request is limited, making it dif\ufb01cult to exploit SIMD or multicore parallelism. RNNs are especially challenging to deploy because evaluating RNNs sample by\n20\n0 1 2 3 4 5 6 7 8 9 10 11\nBatch size0.00.10.20.30.4Probability10 streams\n20 streams\n30 streamsFigure 5: Probability that a request is processed in a batch of given size\nsample relies on sequential matrix vector multiplications, which are bandwidth bound and dif\ufb01cult\nto parallelize.\nTo overcome these issues, we built a batching scheduler called Batch Dispatch that assembles\nstreams of data from user requests into batches before performing forward propagation on these\nbatches. In this case, there is a tradeoff between increased batch size, and consequently improved"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4b3-6db4-11ee-81f5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-68", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "streams of data from user requests into batches before performing forward propagation on these\nbatches. In this case, there is a tradeoff between increased batch size, and consequently improved\nef\ufb01ciency, and increased latency. The more we buffer user requests to assemble a large batch, the\nlonger users must wait for their results. This places constraints on the amount of batching we can\nperform.\nWe use an eager batching scheme that processes each batch as soon as the previous batch is completed, regardless of how much work is ready by that point. This scheduling algorithm has proved\nto be the best at reducing end-user latency, despite the fact that it is less ef\ufb01cient computationally,\nsince it does not attempt to maximize batch size.\nFigure 5 shows the probability that a request is processed in a batch of given size for our production\nsystem running on a single NVIDIA Quadro K1200 GPU, with 10-30 concurrent user requests. As\nexpected, batching works best when the server is heavily loaded: as load increases, the distribution\nshifts to favor processing requests in larger batches. However, even with a light load of only 10\nconcurrent user requests, our system performs more than half the work in batches with at least 2\nsamples.\n0 10 20 30 40\nNumber of concurrent streams050100latency (ms)50%ile\n98%ile"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4b4-6db4-11ee-9e05-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-69", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "samples.\n0 10 20 30 40\nNumber of concurrent streams050100latency (ms)50%ile\n98%ile\nFigure 6: Median and 98 percentile latencies as a function of server load\nWe see in Figure 6, that our system achieves a median latency of 44 ms, and a 98 percentile latency\nof 70 ms when loaded with 10 concurrent streams. As the load increases on the server, the batching\nscheduler shifts work to more ef\ufb01cient batches, which keeps latency low. This shows that Batch\nDispatch makes it possible to deploy these large models at high throughput and low latency.\n21\n1 2 3 4 5 6 7 8 9 10\nBatch size0.00.10.20.30.40.5TeraFLOP/snervana\nbaiduFigure 7: Comparison of kernels that compute Ax=bwhere Ais a matrix with dimension 2560\u00022560 , and\nxis a matrix with dimension 2560\u0002Batch size, where Batch size 2[1, 10] . All matrices are in half-precision\nformat.\n7.2 Deployment Optimized Matrix Multiply Kernels\nWe have found that deploying our models using half-precision (16-bit) \ufb02oating-point arithmetic does\nnot measurably change recognition accuracy. Because deployment does not require any updates to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4b5-6db4-11ee-82bf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-70", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "not measurably change recognition accuracy. Because deployment does not require any updates to\nthe network weights, it is far less sensitive to numerical precision than training. Using half-precision\narithmetic saves memory space and bandwidth, which is especially useful for deployment, since\nRNN evaluation is dominated by the cost of caching and streaming the weight matrices.\nAs seen in Section 7.1, the batch size during deployment is much smaller than in training. We\nfound that standard BLAS libraries are inef\ufb01cient at this batch size. To overcome this, we wrote our\nown half-precision matrix-matrix multiply kernel. For 10 simultaneous streams over 90 percent of\nbatches are for N\u00144, a regime where the matrix multiply will be bandwidth bound. We store the A\nmatrix transposed to maximize bandwidth by using the widest possible vector loads while avoiding\ntransposition after loading. Each warp computes four rows of output for all Noutput columns. Note\nthat forN\u00144theBmatrix \ufb01ts entirely in the L1 cache. This scheme achieves 90 percent of peak\nbandwidth for N\u00144but starts to lose ef\ufb01ciency for larger Nas theBmatrix stops \ufb01tting into the\nL1 cache. Nonetheless, it continues to provide improved performance over existing libraries up to\nN= 10 ."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4b6-6db4-11ee-b869-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-71", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "L1 cache. Nonetheless, it continues to provide improved performance over existing libraries up to\nN= 10 .\nFigure 7 shows that our deployment kernel sustains a higher computational throughput than those\nfrom Nervana Systems [44] on the K1200 GPU, across the entire range of batch sizes that we use\nin deployment. Both our kernels and the Nervana kernels are signi\ufb01cantly faster than NVIDIA\nCUBLAS version 7.0, more details are found here [20].\n7.3 Beam Search\nPerforming the beam search involves repeated lookups in the n-gram language model, most of which\ntranslate to uncached reads from memory. The direct implementation of beam search means that\neach time-step dispatches one lookup per character for each beam. In Mandarin, this results in over\n1M lookups per 40ms stride of speech data, which is too slow for deployment. To deal with this\nproblem, we use a heuristic to further prune the beam search. Rather than considering all characters\nas viable additions to the beam, we only consider the fewest number of characters whose cumulative\nprobability is at least p. In practice, we have found that p= 0.99 works well. Additionally, we limit\nourselves to no more than 40 characters. This speeds up the Mandarin language model lookup time"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4b7-6db4-11ee-b5ff-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-72", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "ourselves to no more than 40 characters. This speeds up the Mandarin language model lookup time\nby a factor of 150x, and has a negligible effect on the CER (0.1-0.3% relative).\n7.4 Results\nWe can deploy our system at low latency and high throughput without sacri\ufb01cing much accuracy.\nOn a held-out set of 2000 utterances, our research system achieves 5.81 character error rate whereas\nthe deployed system achieves 6.10 character error rate. This is only a 5% relative degradation for\n22\nthe deployed system. In order to accomplish this, we employ a neural network architecture with low\ndeployment latency, reduce the precision of our network to 16-bit, built a batching scheduler to more\nef\ufb01ciently evaluate RNNs, and \ufb01nd a simple heuristic to reduce beam search cost. The model has\n\ufb01ve forward-only recurrent layers with 2560 hidden units, one row convolution layer (Section 3.7)\nwith\u001c= 19 , and one fully-connected layer with 2560 hidden units. These techniques allow us to\ndeploy Deep Speech at low cost to interactive applications.\n8 Conclusion\nEnd-to-end deep learning presents the exciting opportunity to improve speech recognition systems\ncontinually with increases in data and computation. Indeed, our results show that, compared to the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4b8-6db4-11ee-a2ee-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-73", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "8 Conclusion\nEnd-to-end deep learning presents the exciting opportunity to improve speech recognition systems\ncontinually with increases in data and computation. Indeed, our results show that, compared to the\nprevious incarnation, Deep Speech has signi\ufb01cantly closed the gap in transcription performance with\nhuman workers by leveraging more data and larger models. Further, since the approach is highly\ngeneric, we\u2019ve shown that it can quickly be applied to new languages. Creating high-performing\nrecognizers for two very different languages, English and Mandarin, required essentially no expert\nknowledge of the languages. Finally, we have also shown that this approach can be ef\ufb01ciently\ndeployed by batching user requests together on a GPU server, paving the way to deliver end-to-end\nDeep Learning technologies to users.\nTo achieve these results, we have explored various network architectures, \ufb01nding several effective\ntechniques: enhancements to numerical optimization through SortaGrad and Batch Normalization,\nevaluation of RNNs with larger strides with bigram outputs for English, searching through both\nbidirectional and unidirectional models. This exploration was powered by a well optimized, High\nPerformance Computing inspired training system that allows us to train new, full-scale models on\nour large datasets in just a few days."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4b9-6db4-11ee-990e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-74", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "Performance Computing inspired training system that allows us to train new, full-scale models on\nour large datasets in just a few days.\nOverall, we believe our results con\ufb01rm and exemplify the value of end-to-end Deep Learning methods for speech recognition in several settings. In those cases where our system is not already comparable to humans, the difference has fallen rapidly, largely because of application-agnostic Deep\nLearning techniques. We believe these techniques will continue to scale, and thus conclude that the\nvision of a single speech system that outperforms humans in most scenarios is imminently achievable.\nAcknowledgments\nWe are grateful to Baidu\u2019s speech technology group for help with data preparation and useful conversations. We would like to thank Scott Gray, Amir Khosrowshahi and all of Nervana Systems for\ntheir excellent matrix multiply routines and useful discussions. We would also like to thank Natalia\nGimelshein of NVIDIA for useful discussions and thoughts on implementing our fast deployment\nmatrix multiply.\nReferences\n[1] O. Abdel-Hamid, A.-r. Mohamed, H. Jang, and G. Penn. Applying convolutional neural networks concepts to hybrid nn-hmm model for speech recognition. In ICASSP , 2012."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ba-6db4-11ee-912b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-75", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "[2] D. Bahdanau, K. Cho, and Y . Bengio. Neural machine translation by jointly learning to align and translate.\nInICLR , 2015.\n[3] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y . Bengio. End-to-end attention-based large\nvocabulary speech recognition. abs/1508.04395, 2015. http://arxiv.org/abs/1508.04395.\n[4] J. Barker, E. Marxer, Ricard Vincent, and S. Watanabe. The third \u2019CHiME\u2019 speech separation and recognition challenge: Dataset, task and baselines. 2015. Submitted to IEEE 2015 Automatic Speech Recognition\nand Understanding Workshop (ASRU).\n[5] S. Baxter. Modern GPU. https://nvlabs.github.io/moderngpu/ .\n[6] Y . Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In International Conference\non Machine Learning , 2009.\n[7] H. Bourlard and N. Morgan. Connectionist Speech Recognition: A Hybrid Approach . Kluwer Academic\nPublishers, Norwell, MA, 1993.\n23"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4bb-6db4-11ee-afa3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-76", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "[7] H. Bourlard and N. Morgan. Connectionist Speech Recognition: A Hybrid Approach . Kluwer Academic\nPublishers, Norwell, MA, 1993.\n23\n[8] W. Chan, N. Jaitly, Q. Le, and O. Vinyals. Listen, attend, and spell. abs/1508.01211, 2015.\nhttp://arxiv.org/abs/1508.01211.\n[9] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer. cuDNN:\nEf\ufb01cient primitives for deep learning.\n[10] T. Chilimbi, Y . Suzue, J. Apacible, and K. Kalyanaraman. Project adam: Building an ef\ufb01cient and scalable\ndeep learning training system. In USENIX Symposium on Operating Systems Design and Implementation ,\n2014.\n[11] K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP , 2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4bc-6db4-11ee-8ada-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-77", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "[12] J. Chorowski, D. Bahdanau, K. Cho, and Y . Bengio. End-to-end continuous speech recognition using\nattention-based recurrent nn: First results. abs/1412.1602, 2015. http://arxiv.org/abs/1412.1602.\n[13] C. Cieri, D. Miller, and K. Walker. The Fisher corpus: a resource for the next generations of speech-totext. In LREC , volume 4, pages 69\u201371, 2004.\n[14] A. Coates, B. Carpenter, C. Case, S. Satheesh, B. Suresh, T. Wang, D. J. Wu, and A. Y . Ng. Text detection\nand character recognition in scene images with unsupervised feature learning. In International Conference\non Document Analysis and Recognition , 2011.\n[15] A. Coates, B. Huval, T. Wang, D. J. Wu, A. Y . Ng, and B. Catanzaro. Deep learning with COTS HPC. In\nInternational Conference on Machine Learning , 2013.\n[16] G. Dahl, D. Yu, and L. Deng. Large vocabulary continuous speech recognition with context-dependent\nDBN-HMMs. In Proc. ICASSP , 2011."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4bd-6db4-11ee-87f4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-78", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "DBN-HMMs. In Proc. ICASSP , 2011.\n[17] G. Dahl, D. Yu, L. Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large\nvocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing , 2011.\n[18] J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker,\nK. Yang, and A. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing\nSystems 25 , 2012.\n[19] D. Ellis and N. Morgan. Size matters: An empirical study of neural network training for large vocabulary\ncontinuous speech recognition. In ICASSP , volume 2, pages 1013\u20131016. IEEE, 1999.\n[20] E. Elsen. Optimizing RNN performance. http://svail.github.io/rnn_perf . Accessed: 2015-11-24.\n[21] M. J. F. Gales, A. Ragni, H. Aldamarki, and C. Gautier. Support vector machines for noise robust ASR.\nInASRU , pages 205\u20132010, 2009."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4be-6db4-11ee-9f27-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-79", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "InASRU , pages 205\u20132010, 2009.\n[22] A. Graves, S. Fern\u00e1ndez, F. Gomez, and J. Schmidhuber. Connectionist temporal classi\ufb01cation: Labelling\nunsegmented sequence data with recurrent neural networks. In ICML , pages 369\u2013376. ACM, 2006.\n[23] A. Graves and N. Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML ,\n2014.\n[24] A. Graves, A.-r. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In\nICASSP , 2013.\n[25] H. H. Sak, A. Senior, and F. Beaufays. Long short-term memory recurrent neural network architectures\nfor large scale acoustic modeling. In Interspeech , 2014.\n[26] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta,\nA. Coates, and A. Y . Ng. Deep speech: Scaling up end-to-end speech recognition. 1412.5567, 2014.\nhttp://arxiv.org/abs/1412.5567."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4bf-6db4-11ee-86cb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-80", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "http://arxiv.org/abs/1412.5567.\n[27] A. Y . Hannun, A. L. Maas, D. Jurafsky, and A. Y . Ng. First-pass large vocabulary continuous speech\nrecognition using bi-directional recurrent DNNs. abs/1408.2873, 2014. http://arxiv.org/abs/1408.2873.\n[28] K. Hea\ufb01eld, I. Pouzyrevsky, J. H. Clark, and P. Koehn. Scalable modi\ufb01ed Kneser-Ney language model\nestimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics ,\nSo\ufb01a, Bulgaria, 8 2013.\n[29] G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V . Vanhoucke, P. Nguyen,\nT. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE\nSignal Processing Magazine , 29(November):82\u201397, 2012."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4c0-6db4-11ee-b934-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-81", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "T. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE\nSignal Processing Magazine , 29(November):82\u201397, 2012.\n[30] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation , 9(8):1735\u20141780,\n1997.\n[31] N. Jaitly and G. Hinton. V ocal tract length perturbation (VTLP) improves speech recognition. In ICML\nWorkshop on Deep Learning for Audio, Speech, and Language Processing , 2013.\n[32] R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network architectures.\nInICML , 2015.\n24\n[33] O. Kapralova, J. Alex, E. Weinstein, P. Moreno, and O. Siohan. A big data approach to acoustic model\ntraining corpus selection. In Interspeech , 2014.\n[34] K. C. Knowlton. A fast storage allocator. Commun. ACM , 8(10):623\u2013624, Oct. 1965."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4c1-6db4-11ee-a58f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-82", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "[34] K. C. Knowlton. A fast storage allocator. Commun. ACM , 8(10):623\u2013624, Oct. 1965.\n[35] T. Ko, V . Peddinti, D. Povey, and S. Khudanpur. Audio augmentation for speech recognition. In Interspeech , 2015.\n[36] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classi\ufb01cation with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 , pages 1106\u20131114, 2012.\n[37] C. Laurent, G. Pereyra, P. Brakel, Y . Zhang, and Y . Bengio. Batch normalized recurrent neural networks.\nabs/1510.01378, 2015. http://arxiv.org/abs/1510.01378.\n[38] Q. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. Corrado, J. Dean, and A. Ng. Building high-level\nfeatures using large scale unsupervised learning. In International Conference on Machine Learning , 2012.\n[39] Y . LeCun, F. J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4c2-6db4-11ee-b21d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-83", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "[39] Y . LeCun, F. J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance\nto pose and lighting. In Computer Vision and Pattern Recognition , volume 2, pages 97\u2013104, 2004.\n[40] A. Maas, Z. Xie, D. Jurafsky, and A. Ng. Lexicon-free conversational speech recognition with neural\nnetworks. In NAACL , 2015.\n[41] Y . Miao, M. Gowayyed, and F. Metz. EESEN: End-to-end speech recognition using deep rnn models and\nwfst-based decoding. In ASRU , 2015.\n[42] A. Mohamed, G. Dahl, and G. Hinton. Acoustic modeling using deep belief networks. IEEE Transactions\non Audio, Speech, and Language Processing , (99), 2011.\n[43] A. S. N. Jaitly, P. Nguyen and V . Vanhoucke. Application of pretrained deep neural networks to large\nvocabulary speech recognition. In Interspeech , 2012.\n[44] Nervana Systems. Nervana GPU. https://github.com/NervanaSystems/nervanagpu . Accessed:\n2015-11-06."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4c3-6db4-11ee-863c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-84", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "[44] Nervana Systems. Nervana GPU. https://github.com/NervanaSystems/nervanagpu . Accessed:\n2015-11-06.\n[45] J. Niu, L. Xie, L. Jia, and N. Hu. Context-dependent deep neural networks for commercial mandarin\nspeech recognition applications. In APSIPA , 2013.\n[46] V . Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain\naudio books. In ICASSP , 2015.\n[47] R. Pascanu, T. Mikolov, and Y . Bengio. On the dif\ufb01culty of training recurrent neural networks.\nabs/1211.5063, 2012. http://arxiv.org/abs/1211.5063.\n[48] P. Patarasuk and X. Yuan. Bandwidth optimal all-reduce algorithms for clusters of workstations. J.\nParallel Distrib. Comput. , 69(2):117\u2013124, Feb. 2009.\n[49] R. Raina, A. Madhavan, and A. Ng. Large-scale deep unsupervised learning using graphics processors.\nIn26th International Conference on Machine Learning , 2009."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4c4-6db4-11ee-9f27-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-85", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "In26th International Conference on Machine Learning , 2009.\n[50] S. Renals, N. Morgan, H. Bourlard, M. Cohen, and H. Franco. Connectionist probability estimators in\nHMM speech recognition. IEEE Transactions on Speech and Audio Processing , 2(1):161\u2013174, 1994.\n[51] T. Robinson, M. Hochberg, and S. Renals. The use of recurrent neural networks in continuous speech\nrecognition. pages 253\u2013258, 1996.\n[52] T. Sainath, O. Vinyals, A. Senior, and H. Sak. Convolutional, long short-term memory, fully connected\ndeep neural networks. In ICASSP , 2015.\n[53] T. N. Sainath, A. rahman Mohamed, B. Kingsbury, and B. Ramabhadran. Deep convolutional neural\nnetworks for LVCSR. In ICASSP , 2013.\n[54] H. Sak, A. Senior, K. Rao, and F. Beaufays. Fast and accurate recurrent neural network acoustic models\nfor speech recognition. abs/1507.06947, 2015. http://arxiv.org/abs/1507.06947."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4c5-6db4-11ee-9011-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-86", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "for speech recognition. abs/1507.06947, 2015. http://arxiv.org/abs/1507.06947.\n[55] H. Sak, O. Vinyals, G. Heigold, A. Senior, E. McDermott, R. Monga, and M. Mao. Sequence discriminative distributed training of long shortterm memory recurrent neural networks. In Interspeech , 2014.\n[56] B. Sapp, A. Saxena, and A. Ng. A fast data collection and augmentation procedure for object recognition.\nInAAAI Twenty-Third Conference on Arti\ufb01cial Intelligence , 2008.\n[57] M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal\nProcessing , 45(11):2673\u20132681, 1997.\n[58] F. Seide, G. Li, and D. Yu. Conversational speech transcription using context-dependent deep neural\nnetworks. In Interspeech , pages 437\u2013440, 2011.\n[59] J. Shan, G. Wu, Z. Hu, X. Tang, M. Jansche, and P. Moreno. Search by voice in mandarin chinese. In\nInterspeech , 2010."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4c6-6db4-11ee-9b84-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-87", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "Interspeech , 2010.\n[60] H. Soltau, G. Saon, and T. Sainath. Joint training of convolutional and non-convolutional neural networks.\nInICASSP , 2014.\n25\n[61] I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of momentum and initialization in\ndeep learning. In 30th International Conference on Machine Learning , 2013.\n[62] I. Sutskever, O. Vinyals, and Q. V . Le. Sequence to sequence learning with neural networks. 2014.\nhttp://arxiv.org/abs/1409.3215.\n[63] C. Szegedy and S. Ioffe. Batch normalization: Accelerating deep network training by reducing internal\ncovariate shift. abs/1502.03167, 2015. http://arxiv.org/abs/1502.03167.\n[64] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. 2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4c7-6db4-11ee-8100-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-88", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "[65] R. Thakur and R. Rabenseifner. Optimization of collective communication operations in mpich. International Journal of High Performance Computing Applications , 19:49\u201366, 2005.\n[66] K. Vesely, A. Ghoshal, L. Burget, and D. Povey. Sequence-discriminative training of deep neural networks. In Interspeech , 2013.\n[67] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang. Phoneme recognition using time-delay\nneural networks,\u00e2 \u02d8A\u02d9I acoustics speech and signal processing. IEEE Transactions on Acoustics, Speech\nand Signal Processing , 37(3):328\u2013339, 1989.\n[68] R. Williams and J. Peng. An ef\ufb01cient gradient-based algorithm for online training of recurrent network\ntrajectories. Neural computation , 2:490\u2013501, 1990.\n[69] T. Yoshioka, N. Ito, M. Delcroix, A. Ogawa, K. Kinoshita, M. F. C. Yu, W. J. Fabian, M. Espi, T. Higuchi,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4c8-6db4-11ee-90b5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-89", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "S. Araki, and T. Nakatani. The ntt chime-3 system: Advances in speech enhancement and recognition for\nmobile multi-microphone devices. In IEEE ASRU , 2015.\n[70] W. Zaremba and I. Sutskever. Learning to execute. abs/1410.4615, 2014. http://arxiv.org/abs/1410.4615.\nA Scalability improvements\nIn this section, we discuss some of our scalability improvements in more detail.\nA.1 Node and cluster architecture\nThe software stack runs on a compute dense node built from 2Intel CPUs and 8NVIDIA Titan\nX GPUs, with peak single-precision computational throughput of 53teraFLOP/second. Each node\nalso has 384GB of CPU memory and an 8TB storage volume built from two 4 TB hard disks in\nRAID-0 con\ufb01guration. We use the CPU memory to cache our input data so that we are not directly\nexposed to the low bandwidth and high latency of spinning disks. We replicate our English and\nMandarin datasets on each node\u2019s local hard disk. This allows us to use our network only for weight\nupdates and avoids having to rely on centralized \ufb01le servers.\nGPUGPUGPUGPUGPUGPUGPUGPUPLXPLXPLXCPUCPUPLX"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4c9-6db4-11ee-b139-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-90", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "updates and avoids having to rely on centralized \ufb01le servers.\nGPUGPUGPUGPUGPUGPUGPUGPUPLXPLXPLXCPUCPUPLX\nFigure 8: Schematic of our training node where PLX indicates a PCI switch and the dotted box includes all\ndevices that are connected by the same PCI root complex.\nFigure 8 shows a schematic diagram of one our nodes, where all devices connected by the same PCI\nroot complex are encapsulated in a dotted box. We have tried to maximize the number of GPUs\nwithin the root complex for faster communication between GPUs using GPUDirect. This allows us\nto use an ef\ufb01cient communication mechanism to transfer gradient matrices between GPUs.\n26\nAll the nodes in our cluster are connected through Fourteen Data Rate (FDR) In\ufb01niband which is\nprimarily used for gradient transfer during back-propagation.\nA.2 GPU Implementation of CTC Loss Function\nThe CTC loss function that we use to train our models has two passes: forward and backward, and\nthe gradient computation involves element-wise addition of two matrices, \u000band\f, generated during\nthe forward and backward passes respectively. Finally, we sum the gradients using the character in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ca-6db4-11ee-8fae-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-91", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "the gradient computation involves element-wise addition of two matrices, \u000band\f, generated during\nthe forward and backward passes respectively. Finally, we sum the gradients using the character in\nthe utterance label as the key, to generate one gradient per character. These gradients are then backpropagated through the network. The input to the CTC loss function are probabilities calculated by\nthe softmax function which can be very small, so we compute in logprobability space for better\nnumerical stability.\nThe forward pass of the CTC algorithm calculates the \u000bmatrix, which has Srows and Tcolumns,\nwhere S= 2(L+ 1) . The variable Lis the number of characters in the label and Tis the number\nof time-steps in the utterance. Our CPU-based implementation of the CTC algorithm assigns one\nthread to each utterance label in a minibatch, performing the CTC calculation for the utterances in\nparallel. Each thread calculates the relevant entries of the matrix sequentially. This is inef\ufb01cient for\ntwo reasons.\nFirstly, since the remainder of our network is computed on the GPU, the output of the softmax\nfunction has to be copied to the CPU for CTC calculation. The gradient matrices from the CTC\nfunction then has to be copied back to the GPU for backpropagation. For languages like Mandarin"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4cb-6db4-11ee-aebf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-92", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "function then has to be copied back to the GPU for backpropagation. For languages like Mandarin\nwith large character sets, these matrices have hundreds of millions of entries, making this copy\nexpensive. Furthermore, we need as much interconnect bandwidth as possible for synchronizing the\ngradient updates with data parallelism, so this copy incurs a substantial opportunity cost.\nSecondly, although entries in each column of the \u000bmatrix can be computed in parallel, the number\nof entries to calculate in each column depends both on the column and the number of repeated\ncharacters in the utterance label. Due to this complexity, the CPU implementation does not use\nSIMD parallelism optimally, making the computation inef\ufb01cient.\nWe wrote a GPU-based implementation of CTC in order to overcome these two problems. The key\ninsight behind our implementation is that we can compute all elements in each column of the \u000b\nmatrix, rather than just the valid entries. If we do so, Figure 9 shows that invalid elements either\ncontain a \ufb01nite garbage value ( G), or\u00001 (I), when we use a special summation function that adds\nprobabilities in logspace that discards inputs that are \u00001. This summation is shown in Figure 9\nwhere arrows incident on a circle are inputs and the result is stored in the circle. However, when we"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4cc-6db4-11ee-ac53-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-93", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "where arrows incident on a circle are inputs and the result is stored in the circle. However, when we\ncompute the \ufb01nal gradient by element-wise summing \u000band\f, all \ufb01nite garbage values will be added\nwith a corresponding \u00001 value from the other matrix, which results in \u00001, effectively ignoring the\ngarbage value and computing the correct result. One important observation is that this element-wise\nsum of\u000band\fis a simple sum and does not use our summation function.\nTo compute the gradient, we take each column of the matrix generated from element-wise addition\nof\u000band\fmatrices, and do a key-value reduction using the character as key, using the ModernGPU\nlibrary [5]. This means elements of the column corresponding to the same character will sum up\ntheir values. In the example shown in Figure 9, the blank character, B, is the only repeated character\nand at some columns, say for t= 1 oft= 2, both valid elements (gray) and \u00001 correspond to\nit. Since our summation function in logspace effectively ignores the \u00001 elements, only the valid\nelements are combined in the reduction.\nIn our GPU implementation, we map each utterance in the minibatch to a CUDA thread block ."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4cd-6db4-11ee-ad8f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-94", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "elements are combined in the reduction.\nIn our GPU implementation, we map each utterance in the minibatch to a CUDA thread block .\nSince there are no dependencies between the elements of a column, all of them can be computed in\nparallel by the threads in a threadblock. There are dependencies between columns, since the column\ncorresponding to time-step t+ 1cannot be computed before the column corresponding to time-step\nt. The reverse happens when computing the \fmatrix, when column corresponding to time-step\ntcannot be computed before the column corresponding to time-step t+ 1. Thus, in both cases,\ncolumns are processed sequentially by the thread block.\nMapping the forward and backward passes to corresponding CUDA kernels is straightforward since\nthere are no data dependencies between elements of a column. The kernel that does the backward\npass also computes the gradient. However, since the gradients must be summed up based on the label\n27\nIIIIATCIIIIIGGGGGGGG\nG1234T-3T-2T-1T\nGGGGATCGGGGGIIIIIIII\nI\u21b5\u0000BBBBBBBBFigure 9: Forward and backward pass for GPU implementation of CTC. Gray circles contain valid values,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ce-6db4-11ee-92d3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-95", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "GGGGATCGGGGGIIIIIIII\nI\u21b5\u0000BBBBBBBBFigure 9: Forward and backward pass for GPU implementation of CTC. Gray circles contain valid values,\ncircle with Icontain\u00001 and circle with Gcontain garbage values that are \ufb01nite. Bstand for the blank character\nthat the CTC algorithm adds to the input utterance label. Column labels on top show different time-steps going\nfrom1toT.\nvalues, with each character as key, we must deal with data dependencies due to repeated characters\nin an utterance label. For languages with small character sets like English, this happens with high\nprobability. Even if there are no repeated characters, the CTC algorithm adds L+ 1blank characters\nto the utterance label. We solve this problem by performing a key-value sort, where the keys are the\ncharacters in the utterance label, and the values are the indices of each character in the utterance.\nAfter sorting, all occurrences of a given character are arranged in contiguous segments. We only\nneed to do the sort once for each utterance. The indices generated by the sort are then used to\nsequentially sum up the gradients for each character. This sum is done once per column and in\nparallel over all characters in the utterance. Amortizing the cost of key-value sort over Tcolumns is"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4cf-6db4-11ee-af01-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-96", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "parallel over all characters in the utterance. Amortizing the cost of key-value sort over Tcolumns is\na key insight that makes the gradient calculation fast.\nOur GPU implementation uses fast shared memory and registers to achieve high performance when\nperforming this task. Both forward and backward kernels store the \u000bmatrix in shared memory .\nSince shared memory is a limited resource, it is not possible to store the entire \fmatrix. However,\nas we go backward in time, we only need to keep one column of the \fmatrix as we compute the\ngradient, adding element-wise the column of the \fmatrix with the corresponding column of the\n\u000bmatrix. Due to on-chip memory space constraints, we read the output of the softmax function\ndirectly from off-chip global memory .\nDue to inaccuracies in \ufb02oating-point arithmetic, especially in transcendental functions, our GPU and\nCPU implementation are not bit-wise identical. This is not an impediment in practice, since both\nimplementations train models equally well when coupled with the technique of sorting utterances\nby length mentioned in Section 3.3.\n28"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4d0-6db4-11ee-9329-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-0", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "Published as a conference paper at ICLR 2017\nRECURRENT BATCH NORMALIZATION\nTim Cooijmans, Nicolas Ballas, C\u00e9sar Laurent, \u00c7a \u02d8glar G\u00fcl\u00e7ehre & Aaron Courville\nMILA - Universit\u00e9 de Montr\u00e9al\nfirstname.lastname@umontreal.ca\nABSTRACT\nWe propose a reparameterization of LSTM that brings the bene\ufb01ts of batch normalization to recurrent neural networks. Whereas previous works only apply batch\nnormalization to the input-to-hidden transformation of RNNs, we demonstrate that\nit is both possible and bene\ufb01cial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps.\nWe evaluate our proposal on various sequential problems such as sequence classi\ufb01cation, language modeling and question answering. Our empirical results show\nthat our batch-normalized LSTM consistently leads to faster convergence and improved generalization.\n1 I NTRODUCTION\nRecurrent neural network architectures such as LSTM (Hochreiter & Schmidhuber, 1997) and\nGRU (Cho et al., 2014) have recently exhibited state-of-the-art performance on a wide range of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4d1-6db4-11ee-9ee7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-1", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "GRU (Cho et al., 2014) have recently exhibited state-of-the-art performance on a wide range of\ncomplex sequential problems including speech recognition Amodei et al. (2015), machine translation (Bahdanau et al., 2015) and image and video captioning (Xu et al., 2015; Yao et al., 2015).\nTop-performing models, however, are based on very high-capacity networks that are computationally intensive and costly to train. Effective optimization of recurrent neural networks is thus an\nactive area of study (Pascanu et al., 2012; Martens & Sutskever, 2011; Ollivier, 2013).\nIt is well-known that for deep feed-forward neural networks, covariate shift (Shimodaira, 2000; Ioffe\n& Szegedy, 2015) degrades the ef\ufb01ciency of training. Covariate shift is a change in the distribution\nof the inputs to a model. This occurs continuously during training of feed-forward neural networks,\nwhere changing the parameters of a layer affects the distribution of the inputs to all layers above it.\nAs a result, the upper layers are continually adapting to the shifting input distribution and unable\nto learn effectively. This internal covariate shift (Ioffe & Szegedy, 2015) may play an especially\nimportant role in recurrent neural networks, which resemble very deep feed-forward networks."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4d2-6db4-11ee-b1bc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-2", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "important role in recurrent neural networks, which resemble very deep feed-forward networks.\nBatch normalization (Ioffe & Szegedy, 2015) is a recently proposed technique for controlling the\ndistributions of feed-forward neural network activations, thereby reducing internal covariate shift. It\ninvolves standardizing the activations going into each layer, enforcing their means and variances to\nbe invariant to changes in the parameters of the underlying layers. This effectively decouples each\nlayer\u2019s parameters from those of other layers, leading to a better-conditioned optimization problem.\nIndeed, deep neural networks trained with batch normalization converge signi\ufb01cantly faster and\ngeneralize better.\nAlthough batch normalization has demonstrated signi\ufb01cant training speed-ups and generalization\nbene\ufb01ts in feed-forward networks, it is proven to be dif\ufb01cult to apply in recurrent architectures (Laurent et al., 2016; Amodei et al., 2015). It has found limited use in stacked RNNs, where the normalization is applied \u201cvertically\u201d, i.e. to the input of each RNN, but not \u201chorizontally\u201d between\ntimesteps. RNNs are deeper in the time direction, and as such batch normalization would be most"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4d3-6db4-11ee-ac76-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-3", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "timesteps. RNNs are deeper in the time direction, and as such batch normalization would be most\nbene\ufb01cial when applied horizontally. However, Laurent et al. (2016) hypothesized that applying\nbatch normalization in this way hurts training because of exploding gradients due to repeated rescaling.\nOur \ufb01ndings run counter to this hypothesis. We show that it is both possible and highly bene\ufb01cial to\napply batch normalization in the hidden-to-hidden transition of recurrent models. In particular, we\ndescribe a reparameterization of LSTM (Section 3) that involves batch normalization and demonstrate that it is easier to optimize and generalizes better. In addition, we empirically analyze the\n1arXiv:1603.09025v5  [cs.LG]  28 Feb 2017\nPublished as a conference paper at ICLR 2017\ngradient backpropagation and show that proper initialization of the batch normalization parameters\nis crucial to avoiding vanishing gradient (Section 4). We evaluate our proposal on several sequential problems and show (Section 5) that our LSTM reparameterization consistently outperforms the\nLSTM baseline across tasks, in terms of both time to convergence and performance.\nLiao & Poggio (2016) simultaneously investigated batch normalization in recurrent neural networks,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4d4-6db4-11ee-8710-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-4", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "LSTM baseline across tasks, in terms of both time to convergence and performance.\nLiao & Poggio (2016) simultaneously investigated batch normalization in recurrent neural networks,\nalbeit only for very short sequences (10 steps). Ba et al. (2016) independently developed a variant\nof batch normalization that is also applicable to recurrent neural networks and delivers similar improvements as our method.\n2 P REREQUISITES\n2.1 LSTM\nLong Short-Term Memory (LSTM) networks are an instance of a more general class of recurrent\nneural networks (RNNs), which we review brie\ufb02y in this paper. Given an input sequence X=\n(x1;x2;:::;xT), an RNN de\ufb01nes a sequence of hidden states htaccording to\nht=\u001e(Whht\u00001+Wxxt+b); (1)\nwhere Wh2Rdh\u0002dh;Wx2Rdx\u0002dh;b2Rdhand the initial state h02Rdhare model parameters. A popular choice for the activation function \u001e(\u0001)istanh .\nRNNs are popular in sequence modeling thanks to their natural ability to process variable-length"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4d5-6db4-11ee-9706-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-5", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "RNNs are popular in sequence modeling thanks to their natural ability to process variable-length\nsequences. However, training RNNs using \ufb01rst-order stochastic gradient descent (SGD) is notoriously dif\ufb01cult due to the well-known problem of exploding/vanishing gradients (Bengio et al., 1994;\nHochreiter, 1991; Pascanu et al., 2012). Gradient vanishing occurs when states htare not in\ufb02uenced\nby small changes in much earlier states h\u001c,t\u001c\u001c, preventing learning of long-term dependencies in\nthe input data. Although learning long-term dependencies is fundamentally dif\ufb01cult (Bengio et al.,\n1994), its effects can be mitigated through architectural variations such as LSTM (Hochreiter &\nSchmidhuber, 1997), GRU (Cho et al., 2014) and iRNN/uRNN (Le et al., 2015; Arjovsky et al.,\n2015).\nIn what follows, we focus on the LSTM architecture (Hochreiter & Schmidhuber, 1997) with recurrent transition given by\n0\nBB@~ft\n~it\n~ot\n~gt1\nCCA=Whht\u00001+Wxxt+b (2)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4d6-6db4-11ee-b89b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-6", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "0\nBB@~ft\n~it\n~ot\n~gt1\nCCA=Whht\u00001+Wxxt+b (2)\nct=\u001b(~ft)\fct\u00001+\u001b(~it)\ftanh( ~gt) (3)\nht=\u001b(~ot)\ftanh(ct); (4)\nwhere Wh2Rdh\u00024dh;WxRdx\u00024dh;b2R4dhand the initial states h02Rdh;c02Rdhare\nmodel parameters. \u001bis the logistic sigmoid function, and the \foperator denotes the Hadamard\nproduct.\nThe LSTM differs from simple RNNs in that it has an additional memory cellctwhose update is\nnearly linear which allows the gradient to \ufb02ow back through time more easily. In addition, unlike\nthe RNN which overwrites its content at each timestep, the update of the LSTM cell is regulated by\na set of gates. The forget gate ftdetermines the extent to which information is carried over from the\nprevious timestep, and the input gate itcontrols the \ufb02ow of information from the current input xt.\nThe output gate otallows the model to read from the cell. This carefully controlled interaction with"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4d7-6db4-11ee-903e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-7", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "The output gate otallows the model to read from the cell. This carefully controlled interaction with\nthe cell is what allows the LSTM to robustly retain information for long periods of time.\n2.2 B ATCH NORMALIZATION\nCovariate shift (Shimodaira, 2000) is a phenomenon in machine learning where the features presented to a model change in distribution. In order for learning to succeed in the presence of covariate shift, the model\u2019s parameters must be adjusted not just to learn the concept at hand but also to\nadapt to the changing distribution of the inputs. In deep neural networks, this problem manifests as\n2\nPublished as a conference paper at ICLR 2017\ninternal covariate shift (Ioffe & Szegedy, 2015), where changing the parameters of a layer affects\nthe distribution of the inputs to all layers above it.\nBatch Normalization (Ioffe & Szegedy, 2015) is a recently proposed network reparameterization\nwhich aims to reduce internal covariate shift. It does so by standardizing the activations using\nempirical estimates of their means and standard deviations. However, it does not decorrelate the\nactivations due to the computationally costly matrix inversion. The batch normalizing transform is\nas follows:\nBN(h;\r;\f) =\f+\r\fh\u0000bE[h]q"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4d8-6db4-11ee-86bc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-8", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "as follows:\nBN(h;\r;\f) =\f+\r\fh\u0000bE[h]q\ndVar[h] +\u000f(5)\nwhere h2Rdis the vector of (pre)activations to be normalized, \r2Rd;\f2Rdare model\nparameters that determine the mean and standard deviation of the normalized activation, and \u000f2R\nis a regularization hyperparameter. The division should be understood to proceed elementwise.\nAt training time, the statistics E[h]andVar[h]are estimated by the sample mean and sample variance of the current minibatch. This allows for backpropagation through the statistics, preserving the\nconvergence properties of stochastic gradient descent. During inference, the statistics are typically\nestimated based on the entire training set, so as to produce a deterministic prediction.\n3 B ATCH -NORMALIZED LSTM\nThis section introduces a reparameterization of LSTM that takes advantage of batch normalization.\nContrary to Laurent et al. (2016); Amodei et al. (2015), we leverage batch normalization in both\nthe input-to-hidden andthe hidden-to-hidden transformations. We introduce the batch-normalizing\ntransform BN(\u0001;\r;\f)into the LSTM as follows:\n0\nBB@~ft"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4d9-6db4-11ee-9e55-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-9", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "transform BN(\u0001;\r;\f)into the LSTM as follows:\n0\nBB@~ft\n~it\n~ot\n~gt1\nCCA= BN( Whht\u00001;\rh;\fh) + BN( Wxxt;\rx;\fx) +b (6)\nct=\u001b(~ft)\fct\u00001+\u001b(~it)\ftanh( ~gt) (7)\nht=\u001b(~ot)\ftanh(BN( ct;\rc;\fc)) (8)\nIn our formulation, we normalize the recurrent term Whht\u00001and the input term Wxxtseparately.\nNormalizing these terms individually gives the model better control over the relative contribution\nof the terms using the \rhand\rxparameters. We set \fh=\fx=0to avoid unnecessary redundancy, instead relying on the pre-existing parameter vector bto account for both biases. In order to\nleave the LSTM dynamics intact and preserve the gradient \ufb02ow through ct, we do not apply batch\nnormalization in the cell update.\nThe batch normalization transform relies on batch statistics to standardize the LSTM activations. It\nwould seem natural to share the statistics that are used for normalization across time, just as recurrent"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4da-6db4-11ee-9ce3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-10", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "The batch normalization transform relies on batch statistics to standardize the LSTM activations. It\nwould seem natural to share the statistics that are used for normalization across time, just as recurrent\nneural networks share their parameters over time. However, we \ufb01nd that simply averaging statistics\nover time severely degrades performance. Although LSTM activations do converge to a stationary\ndistribution, we observe that their statistics during the initial transient differ signi\ufb01cantly (see Figure 5 in Appendix A). Consequently, we recommend using separate statistics for each timestep to\npreserve information of the initial transient phase in the activations.1\nGeneralizing the model to sequences longer than those seen during training is straightforward thanks\nto the rapid convergence of the activations to their steady-state distributions (cf. Figure 5). For our\nexperiments we estimate the population statistics separately for each timestep 1;:::;Tmax where\n1Note that we separate only the statistics over time and not the \rand\fparameters.\n3\nPublished as a conference paper at ICLR 2017\nTmax is the length of the longest training sequence. When at test time we need to generalize beyond\nTmax, we use the population statistic of time Tmax for all time steps beyond it.\nDuring training we estimate the statistics across the minibatch, independently for each timestep. At"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4db-6db4-11ee-9493-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-11", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "Tmax, we use the population statistic of time Tmax for all time steps beyond it.\nDuring training we estimate the statistics across the minibatch, independently for each timestep. At\ntest time we use estimates obtained by averaging the minibatch estimates over the training set.\n4 I NITIALIZING \rFOR GRADIENT FLOW\nAlthough batch normalization allows for easy control of the pre-activation variance through the \r\nparameters, common practice is to normalize to unit variance. We suspect that the previous dif\ufb01culties with recurrent batch normalization reported in Laurent et al. (2016); Amodei et al. (2015) are\nlargely due to improper initialization of the batch normalization parameters, and \rin particular. In\nthis section we demonstrate the impact of \ron gradient \ufb02ow.\n0 100 200 300 400 500 600 700 800\nt10-2610-2410-2210-2010-1810-1610-1410-1210-1010-810-610-410-2100||dloss/dh_t||_2RNN gradient propagation\ngamma=0.10\ngamma=0.20\ngamma=0.30\ngamma=0.40\ngamma=0.50\ngamma=0.60\ngamma=0.70\ngamma=0.80"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4dc-6db4-11ee-afd0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-12", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "gamma=0.40\ngamma=0.50\ngamma=0.60\ngamma=0.70\ngamma=0.80\ngamma=0.90\ngamma=1.00\n(a) We visualize the gradient \ufb02ow through a batchnormalized tanh RNN as a function of \r. High\nvariance causes vanishing gradient.\n0.0 0.2 0.4 0.6 0.8 1.0\ninput standard deviation0.00.20.40.60.81.0expected derivative (and IQR range)derivative through tanh(b) We show the empirical expected derivative and\ninterquartile range of tanh nonlinearity as a function of input variance. High variance causes saturation, which decreases the expected derivative.\nFigure 1: In\ufb02uence of pre-activation variance on gradient propagation.\nIn Figure 1(a), we show how the pre-activation variance impacts gradient propagation in a simple\nRNN on the sequential MNIST task described in Section 5.1. Since backpropagation operates in\nreverse, the plot is best read from right to left. The quantity plotted is the norm of the gradient\nof the loss with respect to the hidden state at different time steps. For large values of \r, the norm"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4dd-6db4-11ee-9d59-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-13", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "of the loss with respect to the hidden state at different time steps. For large values of \r, the norm\nquickly goes to zero as gradient is propagated back in time. For small values of \rthe norm is nearly\nconstant.\nTo demonstrate what we think is the cause of this vanishing, we drew samples xfrom a set of\ncentered Gaussian distributions with standard deviation ranging from 0 to 1, and computed the\nderivative tanh0(x) = 1\u0000tanh2(x)2[0;1]for each. Figure 1(b) shows the empirical distribution\nof the derivative as a function of standard deviation. When the input standard deviation is low, the\ninput tends to be close to the origin where the derivative is close to 1. As the standard deviation\nincreases, the expected derivative decreases as the input is more likely to be in the saturation regime.\nAt unit standard deviation, the expected derivative is much smaller than 1.\nWe conjecture that this is what causes the gradient to vanish, and recommend initializing \rto a\nsmall value. In our trials we found that values of 0.01 or lower caused instabilities during training.\nOur choice of 0.1 seems to work well across different tasks.\n5 E XPERIMENTS"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4de-6db4-11ee-93c7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-14", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "Our choice of 0.1 seems to work well across different tasks.\n5 E XPERIMENTS\nThis section presents an empirical evaluation of the proposed batch-normalized LSTM on four different tasks. Note that for all the experiments, we initialize the batch normalization scale and shift\nparameters\rand\fto0:1and0respectively.\n4\nPublished as a conference paper at ICLR 2017\n0 20000 40000 60000 80000 100000\nTraining Iteration0.00.20.40.60.81.0AccuracyPixel-by-Pixel MNIST (Validation Set)\nlstm\nbn_lstm\n0 20000 40000 60000 80000 100000\nTraining Iteration0.10.20.30.40.50.60.70.80.91.0AccuracyPixel-by-Pixel Permuted-MNIST (Validation Set)\nlstm\nbn_lstm\nFigure 2: Accuracy on the validation set for the pixel by pixel MNIST classi\ufb01cation tasks. The\nbatch-normalized LSTM is able to converge faster relatively to a baseline LSTM. Batch-normalized\nLSTM also shows some improve generalization on the permuted sequential MNIST that require to\npreserve long-term memory information.\n5.1 S EQUENTIAL MNIST"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4df-6db4-11ee-9dfd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-15", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "LSTM also shows some improve generalization on the permuted sequential MNIST that require to\npreserve long-term memory information.\n5.1 S EQUENTIAL MNIST\nWe evaluate our batch-normalized LSTM on a sequential version of the MNIST classi\ufb01cation\ntask (Le et al., 2015). The model processes each image one pixel at a time and \ufb01nally predicts\nthe label. We consider both sequential MNIST tasks, MNIST and permuted MNIST ( pMNIST). In\nMNIST, the pixels are processed in scanline order. In pMNIST the pixels are processed in a \ufb01xed\nrandom order.\nOur baseline consists of an LSTM with 100 hidden units, with a softmax classi\ufb01er to produce a\nprediction from the \ufb01nal hidden state. We use orthogonal initialization for all weight matrices,\nexcept for the hidden-to-hidden weight matrix which we initialize to be the identity matrix, as this\nyields better generalization performance on this task for both models. The model is trained using\nRMSProp (Tieleman & Hinton, 2012) with learning rate of 10\u00003and0:9momentum. We apply\ngradient clipping at 1 to avoid exploding gradients."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4e0-6db4-11ee-a81b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-16", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "gradient clipping at 1 to avoid exploding gradients.\nThe in-order MNIST task poses a unique problem for our model: the input for the \ufb01rst hundred or so\ntimesteps is constant across examples since the upper pixels are almost always black. This causes the\nvariance of the hidden states to be exactly zero for a long period of time. Normalizing these zerovariance activations involves dividing zero by a small number at many timesteps, which does not\naffect the forward-propagated activations but causes the back-propagated gradient to explode. We\nwork around this by adding Gaussian noise to the initial hidden states. Although the normalization\nampli\ufb01es the noise to signal level, we \ufb01nd that it does not hurt performance compared to datadependent ways of initializing the hidden states.\nModel MNIST pMNIST\nTANH-RNN (Le et al., 2015) 35.0 35.0\niRNN (Le et al., 2015) 97.0 82.0\nuRNN (Arjovsky et al., 2015) 95.1 91.4\nsTANH-RNN (Zhang et al., 2016) 98.1 94.0\nLSTM (ours) 98.9 90.2\nBN-LSTM (ours) 99.0 95.4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4e1-6db4-11ee-b8da-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-17", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "LSTM (ours) 98.9 90.2\nBN-LSTM (ours) 99.0 95.4\nTable 1: Accuracy obtained on the test set for the pixel by pixel MNIST classi\ufb01cation tasks\nIn Figure 2 we show the validation accuracy while training for both LSTM and batch-normalized\nLSTM (BN-LSTM). BN-LSTM converges faster than LSTM on both tasks. Additionally, we observe that BN-LSTM generalizes signi\ufb01cantly better on pMNIST. It has been highlighted in Arjovsky et al. (2015) that pMNIST contains many longer term dependencies across pixels than in\nthe original pixel ordering, where a lot of structure is local. A recurrent network therefore needs to\n5\nPublished as a conference paper at ICLR 2017\nModel Penn Treebank\nLSTM (Graves, 2013) 1.262\nHF-MRNN (Mikolov et al., 2012) 1.41\nNorm-stabilized LSTM (Krueger & Memisevic, 2016) 1.39\nME n-gram (Mikolov et al., 2012) 1.37\nLSTM (ours) 1.38\nBN-LSTM (ours) 1.32\nZoneout (Krueger et al., 2016) 1.27"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4e2-6db4-11ee-9002-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-18", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "LSTM (ours) 1.38\nBN-LSTM (ours) 1.32\nZoneout (Krueger et al., 2016) 1.27\nHM-LSTM (Chung et al., 2016) 1.24\nHyperNetworks (Ha et al., 2016) 1.22\nTable 2: Bits-per-character on the Penn Treebank test sequence.\ncharacterize dependencies across varying time scales in order to solve this task. Our results suggest\nthat BN-LSTM is better able to capture these long-term dependencies.\nTable 1 reports the test set accuracy of the early stop model for LSTM and BN-LSTM using the population statistics. Recurrent batch normalization leads to a better test score, especially for pMNIST\nwhere models have to leverage long-term temporal depencies. In addition, Table 1 shows that our\nbatch-normalized LSTM achieves state of the art on both MNIST and pMNIST.\n5.2 C HARACTER -LEVEL PENN TREEBANK\nWe evaluate our model on the task of character-level language modeling on the Penn Treebank\ncorpus (Marcus et al., 1993) according to the train/valid/test partition of Mikolov et al. (2012). For\ntraining, we segment the training sequence into examples of length 100. The training sequence does"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4e3-6db4-11ee-b2b9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-19", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "training, we segment the training sequence into examples of length 100. The training sequence does\nnot cleanly divide by 100, so for each epoch we randomly crop a subsequence that does and segment\nthat instead.\nOur baseline is an LSTM with 1000 units, trained to predict the next character using a softmax\nclassi\ufb01er on the hidden state ht. We use stochastic gradient descent on minibatches of size 64, with\ngradient clipping at 1.0 and step rule determined by Adam (Kingma & Ba, 2014) with learning rate\n0.002. We use orthogonal initialization for all weight matrices. The setup for the batch-normalized\nLSTM is the same in all respects except for the introduction of batch normalization as detailed in 3.\nWe show the learning curves in Figure 3(a). BN-LSTM converges faster and generalizes better than\nthe LSTM baseline. Figure 3(b) shows the generalization of our model to longer sequences. We\nobserve that using the population statistics improves generalization performance, which con\ufb01rms\nthat repeating the last population statistic (cf. Section 3) is a viable strategy. In table 2 we report\nthe performance of our best models (early-stopped on validation performance) on the Penn Treebank"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4e4-6db4-11ee-9334-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-20", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "the performance of our best models (early-stopped on validation performance) on the Penn Treebank\ntest sequence. Follow up works havd since improved the state of the art (Krueger et al., 2016; Chung\net al., 2016; Ha et al., 2016).\n5.3 T EXT8\nWe evaluate our model on a second character-level language modeling task on the much larger\ntext8 dataset (Mahoney, 2009). This dataset is derived from Wikipedia and consists of a sequence\nof 100M characters including only alphabetical characters and spaces. We follow Mikolov et al.\n(2012); Zhang et al. (2016) and use the \ufb01rst 90M characters for training, the next 5M for validation\nand the \ufb01nal 5M characters for testing. We train on nonoverlapping sequences of length 180.\nBoth our baseline and batch-normalized models are LSTMs with 2000 units, trained to predict the\nnext character using a softmax classi\ufb01er on the hidden state ht. We use stochastic gradient descent\non minibatches of size 128, with gradient clipping at 1.0 and step rule determined by Adam (Kingma\n& Ba, 2014) with learning rate 0.001. All weight matrices were initialized to be orthogonal.\n6\nPublished as a conference paper at ICLR 2017"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4e5-6db4-11ee-92b6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-21", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "& Ba, 2014) with learning rate 0.001. All weight matrices were initialized to be orthogonal.\n6\nPublished as a conference paper at ICLR 2017\nWe early-stop on validation performance and report the test performance of the resulting model in\ntable 3. We observe that BN-LSTM obtains a signi\ufb01cant performance improvement over the LSTM\nbaseline. Chung et al. (2016) has since improved on our performance.\nModel text8\ntd-LSTM (Zhang et al., 2016) 1.63\nHF-MRNN (Mikolov et al., 2012) 1.54\nskipping RNN (Pachitariu & Sahani, 2013) 1.48\nLSTM (ours) 1.43\nBN-LSTM (ours) 1.36\nHM-LSTM (Chung et al., 2016) 1.29\nTable 3: Bits-per-character on the text8 test sequence.\n5.4 T EACHING MACHINES TO READ AND COMPREHEND\nRecently, Hermann et al. (2015) introduced a set of challenging benchmarks for natural language\nprocessing, along with neural network architectures to address them. The tasks involve reading"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4e6-6db4-11ee-b512-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-22", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "Recently, Hermann et al. (2015) introduced a set of challenging benchmarks for natural language\nprocessing, along with neural network architectures to address them. The tasks involve reading\nreal news articles and answering questions about their content. Their principal model, the Attentive Reader, is a recurrent neural network that invokes an attention mechanism to locate relevant\ninformation in the document. Such models are notoriously hard to optimize and yet increasingly\npopular.\nTo demonstrate the generality and practical applicability of our proposal, we apply batch normalization in the Attentive Reader model and show that this drastically improves training.\nWe evaluate several variants. The \ufb01rst variant, referred to as BN-LSTM, consists of the vanilla Attentive Reader model with the LSTM simply replaced by our BN-LSTM reparameterization. The\nsecond variant, termed BN-everywhere, is exactly like the \ufb01rst, except that we also introduce batch\nnormalization into the attention computations, normalizing each term going into the tanh nonlinearities.\nOur third variant, BN-e*, is like BN-everywhere, but improved to more carefully handle variablelength sequences. Throughout this experiment we followed the common practice of padding each\nbatch of variable-length data with zeros. However, this biases the batch mean and variance of xt"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4e7-6db4-11ee-b72a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-23", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "batch of variable-length data with zeros. However, this biases the batch mean and variance of xt\ntoward zero. We address this effect using sequencewise normalization of the inputs as proposed\nby Laurent et al. (2016); Amodei et al. (2015). That is, we share statistics over time for normalization\n0 2000 4000 6000 8000 10000 12000 14000 16000\ntraining steps1.41.61.82.02.22.4bits per characterLSTM\nBN-LSTM\n(a) Performance in bits-per-character on length100 subsequences of the Penn Treebank validation\nsequence during training.\n100 200 300 400 500 600 700 800 900 1000\nsequence length1.321.341.361.381.401.421.441.46mean bits per characterLSTM\nBN-LSTM, population statistics\nBN-LSTM, batch statistics(b) Generalization to longer subsequences of Penn\nTreebank using population statistics. The subsequences are taken from the test sequence.\nFigure 3: Penn Treebank evaluation\n7\nPublished as a conference paper at ICLR 2017\n0 100 200 300 400 500 600 700 800\ntraining steps (thousands)0.00.20.40.60.81.0error rateLSTM train\nBN-LSTM train\nBN-everywhere train\nBN-e* train"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4e8-6db4-11ee-9c10-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-24", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "BN-LSTM train\nBN-everywhere train\nBN-e* train\nBN-e** train\nLSTM valid\nBN-LSTM valid\nBN-everywhere valid\nBN-e* valid\nBN-e** valid\n(a) Error rate on the validation set for the Attentive Reader models on a variant of the CNN QA\ntask (Hermann et al., 2015). As detailed in Appendix C, the theoretical lower bound on the error\nrate on this task is 43%.\n0 50 100 150 200 250 300 350 400\ntraining steps (thousands)0.10.20.30.40.50.60.70.80.91.0error rateLSTM train\nBN-e** train\nLSTM valid\nBN-e** valid(b) Error rate on the validation set on the full CNN\nQA task from Hermann et al. (2015).\nFigure 4: Training curves on the CNN question-answering tasks.\nof the input terms Wxxt, but notfor the recurrent terms Whhtor the cell output ct. Doing so avoids\nmany issues involving degenerate statistics due to input sequence padding.\nOur fourth and \ufb01nal variant BN-e** is like BN-e* but bidirectional. The main dif\ufb01culty in adapting"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4e9-6db4-11ee-8458-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-25", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "Our fourth and \ufb01nal variant BN-e** is like BN-e* but bidirectional. The main dif\ufb01culty in adapting\nto bidirectional models also involves padding. Padding poses no problem as long as it is properly\nignored (by not updating the hidden states based on padded regions of the input). However to\nperform the reverse application of a bidirectional model, it is common to simply reverse the padded\nsequences, thus moving the padding to the front. This causes similar problems as were observed\non the sequential MNIST task (Section 5.1): the hidden states will not diverge during the initial\ntimesteps and hence their variance will be severely underestimated. To get around this, we reverse\nonly the unpadded portion of the input sequences and leave the padding in place.\nSee Appendix C for hyperparameters and task details.\nFigure 4(a) shows the learning curves for the different variants of the attentive reader. BN-LSTM\ntrains dramatically faster than the LSTM baseline. BN-everywhere in turn shows a signi\ufb01cant improvement over BN-LSTM. In addition, both BN-LSTM and BN-everywhere show a generalization"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ea-6db4-11ee-b29d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-26", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "bene\ufb01t over the baseline. The validation curves have minima of 50.3%, 49.5% and 50.0% for the\nbaseline, BN-LSTM and BN-everywhere respectively. We emphasize that these results were obtained without any tweaking \u2013 all we did was to introduce batch normalization.\nBN-e* and BN-e** converge faster yet, and reach lower minima: 47.1% and 43.9% respectively.\nModel CNN valid CNN test\nAttentive Reader (Hermann et al., 2015) 38.4 37.0\nLSTM (ours) 45.5 45.0\nBN-e** (ours) 37.9 36.3\nTable 4: Error rates on the CNN question-answering task Hermann et al. (2015).\nWe train and evaluate our best model, BN-e**, on the full task from (Hermann et al., 2015). On\nthis dataset we had to reduce the number of hidden units to 120 to avoid severe over\ufb01tting. Training\ncurves for BN-e** and a vanilla LSTM are shown in Figure 4(b). Table 4 reports performances of\nthe early-stopped models.\n8\nPublished as a conference paper at ICLR 2017\n6 C ONCLUSION"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4eb-6db4-11ee-8d96-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-27", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "the early-stopped models.\n8\nPublished as a conference paper at ICLR 2017\n6 C ONCLUSION\nContrary to previous \ufb01ndings by Laurent et al. (2016); Amodei et al. (2015), we have demonstrated\nthat batch-normalizing the hidden states of recurrent neural networks greatly improves optimization. Indeed, doing so yields bene\ufb01ts similar to those of batch normalization in feed-forward neural\nnetworks: our proposed BN-LSTM trains faster and generalizes better on a variety of tasks including language modeling and question-answering. We have argued that proper initialization of\nthe batch normalization parameters is crucial, and suggest that previous dif\ufb01culties (Laurent et al.,\n2016; Amodei et al., 2015) were due in large part to improper initialization. Finally, we have shown\nour model to apply to complex settings involving variable-length data, bidirectionality and highly\nnonlinear attention mechanisms.\nACKNOWLEDGEMENTS\nThe authors would like to acknowledge the following agencies for research funding and computing\nsupport: the Nuance Foundation, Samsung, NSERC, Calcul Qu\u00e9bec, Compute Canada, the Canada\nResearch Chairs and CIFAR. Experiments were carried out using the Theano (Team et al., 2016) and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ec-6db4-11ee-a42c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-28", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "Research Chairs and CIFAR. Experiments were carried out using the Theano (Team et al., 2016) and\nthe Blocks and Fuel (van Merri\u00ebnboer et al., 2015) libraries for scienti\ufb01c computing. We thank David\nKrueger, Saizheng Zhang, Ishmael Belghazi and Yoshua Bengio for discussions and suggestions.\nREFERENCES\nD. Amodei et al. Deep speech 2: End-to-end speech recognition in english and mandarin.\narXiv:1512.02595 , 2015.\nM. Arjovsky, A. Shah, and Y . Bengio. Unitary evolution recurrent neural networks.\narXiv:1511.06464 , 2015.\nJimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450 ,\n2016.\nD. Bahdanau, K. Cho, and Y . Bengio. Neural machine translation by jointly learning to align and\ntranslate. ICLR , 2015.\nY . Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is\ndif\ufb01cult. Neural Networks, IEEE Transactions on , 1994."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ed-6db4-11ee-824e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-29", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "dif\ufb01cult. Neural Networks, IEEE Transactions on , 1994.\nK. Cho, B. Van Merri\u00ebnboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio.\nLearning phrase representations using rnn encoder-decoder for statistical machine translation.\narXiv:1406.1078 , 2014.\nJunyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. arXiv:1609.01704 , 2016.\nA. Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850 , 2013.\nDavid Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv:1609.09106 , 2016.\nK. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom.\nTeaching machines to read and comprehend. In NIPS , 2015.\nS. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Master\u2019s thesis , 1991."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ee-6db4-11ee-9ce4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-30", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Master\u2019s thesis , 1991.\nS. Hochreiter and J Schmidhuber. Long short-term memory. Neural computation , 1997.\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by\nreducing internal covariate shift. abs/1502.03167, 2015.\nD. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 , 2014.\nD Krueger and R. Memisevic. Regularizing rnns by stabilizing activations. ICLR , 2016.\n9\nPublished as a conference paper at ICLR 2017\nDavid Krueger, Tegan Maharaj, J\u00e1nos Kram\u00e1r, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Aaron Courville. Zoneout:\nRegularizing rnns by randomly preserving hidden activations. arXiv:1606.01305 , 2016.\nC. Laurent, G. Pereyra, P. Brakel, Y . Zhang, and Y . Bengio. Batch normalized recurrent neural\nnetworks. ICASSP , 2016."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ef-6db4-11ee-966a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-31", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "C. Laurent, G. Pereyra, P. Brakel, Y . Zhang, and Y . Bengio. Batch normalized recurrent neural\nnetworks. ICASSP , 2016.\nQuoc V Le, N. Jaitly, and G. Hinton. A simple way to initialize recurrent networks of recti\ufb01ed linear\nunits. arXiv:1504.00941 , 2015.\nQianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural\nnetworks and visual cortex. arXiv:1604.03640 , 2016.\nM. Mahoney. Large text compression benchmark. 2009.\nM. P. Marcus, M. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of english:\nThe penn treebank. Comput. Linguist. , 1993.\nJ. Martens and I. Sutskever. Learning recurrent neural networks with hessian-free optimization. In\nICML , 2011.\nT. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky. Subword language\nmodeling with neural networks. preprint , 2012.\nYann Ollivier. Persistent contextual neural networks for learning symbolic data sequences. CoRR ,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4f0-6db4-11ee-861e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-32", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "modeling with neural networks. preprint , 2012.\nYann Ollivier. Persistent contextual neural networks for learning symbolic data sequences. CoRR ,\nabs/1306.0514, 2013.\nMarius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are they needed? arXiv:1301.5650 , 2013.\nRazvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the dif\ufb01culty of training recurrent neural\nnetworks. arXiv:1211.5063 , 2012.\nH. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood\nfunction. Journal of statistical planning and inference , 2000.\nThe Theano Development Team et al. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints , abs/1605.02688, May 2016.\nT. Tieleman and G. Hinton. Lecture 6.5\u2014RmsProp: Divide the gradient by a running average of its\nrecent magnitude. COURSERA: Neural Networks for Machine Learning, 2012."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4f1-6db4-11ee-a586-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-33", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.\nBart van Merri\u00ebnboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David WardeFarley, Jan Chorowski, and Yoshua Bengio. Blocks and fuel: Frameworks for deep learning.\nCoRR , abs/1506.00619, 2015. URL http://arxiv.org/abs/1506.00619 .\nK. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y . Bengio. Show, attend and\ntell: Neural image caption generation with visual attention. arXiv:1502.03044 , 2015.\nL. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville. Describing videos by\nexploiting temporal structure. In ICCV , 2015.\nS. Zhang, Y . Wu, T. Che, Z. Lin, R. Memisevic, R. Salakhutdinov, and Y . Bengio. Architectural\ncomplexity measures of recurrent neural networks. arXiv:1602.08210 , 2016.\n10\nPublished as a conference paper at ICLR 2017"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4f2-6db4-11ee-84a0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-34", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "complexity measures of recurrent neural networks. arXiv:1602.08210 , 2016.\n10\nPublished as a conference paper at ICLR 2017\nA C ONVERGENCE OF POPULATION STATISTICS\n0.20\n0.15\n0.10\n0.05\n0.000.050.100.150.20mean of recurrent term\n0.20\n0.15\n0.10\n0.05\n0.000.050.100.150.20mean of cell state\n0 10 20 30 40 50\ntime steps0.00.51.01.52.0variance of recurrent term\n0 10 20 30 40 50\ntime steps0.0010.0020.0030.0040.0050.0060.0070.0080.009variance of cell state\nFigure 5: Convergence of population statistics to stationary distributions on the Penn Treebank task.\nThe horizontal axis denotes RNN time. Each curve corresponds to a single hidden unit. Only a\nrandom subset of units is shown. See Section 3 for discussion.\nB S ENSITIVITY TO INITIALIZATION OF \r\nIn Section 4 we investigated the effect of initial \ron gradient \ufb02ow. To show the practical implications of this, we performed several experiments on the pMNIST and Penn Treebank benchmarks.\nThe resulting performances are shown in Figure 6."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4f3-6db4-11ee-8821-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-35", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "The resulting performances are shown in Figure 6.\nThepMNIST training curves con\ufb01rm that higher initial values of \rare detrimental to the optimization of the model. For the Penn Treebank task however, the effect is gone.\nWe believe this is explained by the difference in the nature of the two tasks. For pMNIST, the model\nabsorbs the input sequence and only at the end of the sequence does it make a prediction on which it\nreceives feedback. Learning from this feedback requires propagating the gradient all the way back\nthrough the sequence.\nIn the Penn Treebank task on the other hand, the model makes a prediction at each timestep. At\neach step of the backward pass, a fresh learning signal is added to the backpropagated gradient.\nEssentially, the model is able to get off the ground by picking up short-term dependencies. This fails\nonpMNIST wich is dominated by long-term dependencies (Arjovsky et al., 2015).\nC T EACHING MACHINES TO READ AND COMPREHEND : TASK SETUP\nWe evaluate the models on the question answering task using the CNN corpus (Hermann et al.,\n2015), with placeholders for the named entities. We follow a similar preprocessing pipeline as Hermann et al. (2015). During training, we randomly sample the examples with replacement and shuf\ufb02e"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4f4-6db4-11ee-9c0c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-36", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "the order of the placeholders in each text inside the minibatch. We use a vocabulary of 65829 words.\nWe deviate from Hermann et al. (2015) in order to save computation: we use only the 4 most relevant\nsentences from the description, as identi\ufb01ed by a string matching procedure. Both the training and\nvalidation sets are preprocessed in this way. Due to imprecision this heuristic sometimes strips the\n11\nPublished as a conference paper at ICLR 2017\n0 10000 20000 30000 40000 50000\ntraining steps0.00.51.01.52.02.5cross entropyPermuted MNIST train\ngamma 0.10\ngamma 0.30\ngamma 0.50\ngamma 0.70\ngamma 1.00\n0 10000 20000 30000 40000 50000\ntraining steps0.00.51.01.52.02.5cross entropyPermuted MNIST valid\ngamma 0.10\ngamma 0.30\ngamma 0.50\ngamma 0.70\ngamma 1.00\n0 5000 10000 15000\ntraining steps0.800.850.900.951.001.051.10bits per characterPTB train\ngamma 0.10\ngamma 0.30\ngamma 0.50\ngamma 0.70"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4f5-6db4-11ee-bacf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-37", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "gamma 0.10\ngamma 0.30\ngamma 0.50\ngamma 0.70\ngamma 1.00\n0 5000 10000 15000\ntraining steps1.001.021.041.061.081.10bits per characterPTB valid\ngamma 0.10\ngamma 0.30\ngamma 0.50\ngamma 0.70\ngamma 1.00\nFigure 6: Training curves on pMNIST and Penn Treebank for various initializations of \r.\nanswers from the passage, putting an upper bound of 57% on the validation accuracy that can be\nachieved.\nFor the reported performances, the \ufb01rst three models (LSTM, BN-LSTM and BN-everywhere) are\ntrained using the exact same hyperparameters, which were chosen because they work well for the\nbaseline. The hidden state is composed of 240 units. We use stochastic gradient descent on minibatches of size 64, with gradient clipping at 10 and step rule determined by Adam (Kingma & Ba,\n2014) with learning rate 8\u000210\u00005.\nFor BN-e* and BN-e**, we use the same hyperparameters except that we reduce the learning rate to\n8\u000210\u00004and the minibatch size to 40."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4f6-6db4-11ee-938d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-38", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "8\u000210\u00004and the minibatch size to 40.\nD H YPERPARAMETER SEARCHES\nTable 5 reports hyperparameter values that were tried in the experiments.\n(a) MNIST and pMNIST\nLearning rate: 1e-2, 1e-3, 1e-4\nRMSProp momentum: 0.5, 0.9\nHidden state size: 100, 200, 400\nInitial\r: 1e-1, 3e-1, 5e-1, 7e-1, 1.0(b) Penn Treebank\nLearning rate: 1e-1, 1e-2, 2e-2, 1e-3\nHidden state size: 800, 1000, 1200, 1500, 2000\nBatch size: 32, 64, 100, 128\nInitial\r: 1e-1, 3e-1, 5e-1, 7e-1, 1.0\n(c) Text8\nLearning rate: 1e-1, 1e-2, 1e-3\nHidden state size: 500, 1000, 2000, 4000(d) Attentive Reader\nLearning rate: 8e-3, 8e-4, 8e-5, 8e-6\nHidden state size: 60, 120, 240, 280\nTable 5: Hyperparameter values that have been explored in the experiments."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4f7-6db4-11ee-9920-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1603.09025-39", "title": "Recurrent Batch Normalization", "source": "http://arxiv.org/pdf/1603.09025", "start_index": 0, "chroma:document": "Hidden state size: 60, 120, 240, 280\nTable 5: Hyperparameter values that have been explored in the experiments.\nFor MNIST and pMNIST, the hyperparameters were varied independently. For Penn Treebank, we\nperformed a full grid search on learning rate and hidden state size, and later performed a sensitivity\n12\nPublished as a conference paper at ICLR 2017\nanalysis on the batch size and initial \r. For the text8 task and the experiments with the Attentive\nReader, we carried out a grid search on the learning rate and hidden state size.\nThe same values were tried for both the baseline and our BN-LSTM. In each case, our reported\nresults are those of the model with the best validation performance.\n13"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4f8-6db4-11ee-8ba5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-0", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "Convolutional Random Walk Networks for Semantic Image Segmentation\nGedas Bertasius1, Lorenzo Torresani2, Stella X. Yu3, Jianbo Shi1\n1University of Pennsylvania,2Dartmouth College,3UC Berkeley ICSI\n{gberta,jshi}@seas.upenn.edu lt@dartmouth.edu stella.yu@berkeley.edu\nAbstract\nMost current semantic segmentation methods rely on\nfully convolutional networks (FCNs). However, their use of\nlarge receptive \ufb01elds and many pooling layers cause low\nspatial resolution inside the deep layers. This leads to\npredictions with poor localization around the boundaries.\nPrior work has attempted to address this issue by postprocessing predictions with CRFs or MRFs. But such models often fail to capture semantic relationships between objects, which causes spatially disjoint predictions. To overcome these problems, recent methods integrated CRFs or\nMRFs into an FCN framework. The downside of these new\nmodels is that they have much higher complexity than traditional FCNs, which renders training and testing more challenging.\nIn this work we introduce a simple, yet effective Convolutional Random Walk Network (RWN) that addresses the\nissues of poor boundary localization and spatially fragmented predictions with very little increase in model complexity. Our proposed RWN jointly optimizes the objectives"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4f9-6db4-11ee-af7c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-1", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "issues of poor boundary localization and spatially fragmented predictions with very little increase in model complexity. Our proposed RWN jointly optimizes the objectives\nof pixelwise af\ufb01nity and semantic segmentation. It combines these two objectives via a novel random walk layer\nthat enforces consistent spatial grouping in the deep layers\nof the network. Our RWN is implemented using standard\nconvolution and matrix multiplication. This allows an easy\nintegration into existing FCN frameworks and it enables\nend-to-end training of the whole network via standard backpropagation. Our implementation of RWN requires just 131\nadditional parameters compared to the traditional FCNs,\nand yet it consistently produces an improvement over the\nFCNs on semantic segmentation and scene labeling.\n1. Introduction\nFully convolutional networks (FCNs) were \ufb01rst introduced in [20] where they were shown to yield signi\ufb01cant\nimprovements in semantic image segmentation. Adopting the FCN approach, many subsequent methods have\nachieved even better performance [6, 27, 8, 12, 17, 6, 27, 8,\n19, 23]. However, traditional FCN-based methods tend to\nsuffer from several limitations. Large receptive \ufb01elds in the\nInput\n DeepLab [6]\n DeepLab-CRF [6]"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4fa-6db4-11ee-8eeb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-2", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "suffer from several limitations. Large receptive \ufb01elds in the\nInput\n DeepLab [6]\n DeepLab-CRF [6]\nFigure 1: Examples illustrating shortcomings of prior semantic segmentation methods. Segments produced by\nFCNs are poorly localized around object boundaries, while\nDense-CRF produce spatially-disjoint object segments.\nconvolutional layers and the presence of pooling layers lead\nto low spatial resolution in the deepest FCN layers. As a result, their predicted segments tend to be blobby and lack \ufb01ne\nobject boundary details. We report in Fig. 1 some examples\nillustrating typical poor localization of objects in the outputs\nof FCNs. Recently, Chen at al. [6] addressed this issue by\napplying a Dense-CRF post-processing step [15] on top of\ncoarse FCN segmentations. However, such approaches often fail to accurately capture semantic relationships between\nobjects and lead to spatially fragmented segmentations (see\nan example in the last column of Fig. 1).\nTo address these problems several recent methods integrated CRFs or MRFs directly into the FCN framework [27, 19, 17, 5]. However, these new models typically involve (1) a large number of parameters, (2) complex loss functions requiring specialized model training or\n(3) recurrent layers, which make training and testing more"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4fb-6db4-11ee-9fb7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-3", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "(3) recurrent layers, which make training and testing more\ncomplicated. We summarize the most prominent of these\napproaches and their model complexities in Table 1.\nWe note that we do not claim that using complex loss\nfunctions always makes the model overly complex and too\ndif\ufb01cult to use. If a complex loss is integrated into an FCN\nframework such that the FCN can still be trained in a standard fashion, and produce better results than using standard losses, then such a model is bene\ufb01cial. However, in\nthe context of prior segmentation methods [27, 19, 17, 5],\nsuch complex losses often require: 1) modifying the network structure (casting CNN into an RNN) [27, 5], or 2)\nusing a complicated multi-stage learning scheme, where\ndifferent layers are optimized during a different training\nstage [19, 17]. Due to such complex training procedures,\n1arXiv:1605.07681v3  [cs.CV]  8 May 2017\n[6] [5] [23] [27] [19] RWN\nrequires post-processing? 3777 7 7\nuses complex loss? 7773 3 7\nrequires recurrent layers? 7373 7 7\nmodel size (in MB) 79 79 961 514 >1000 79"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4fc-6db4-11ee-9357-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-4", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "uses complex loss? 7773 3 7\nrequires recurrent layers? 7373 7 7\nmodel size (in MB) 79 79 961 514 >1000 79\nTable 1: Summary of recent semantic segmentation models. For each model, we report whether it requires: (1) CRF\npost-processing, (2) complex loss functions, or (3) recurrent layers. We also list the size of the model (using the\nsize of Caffe [13] models in MB). We note that unlike prior\nmethods, our RWN does not need post-processing, it is implemented using standard layers and loss functions, and it\nalso has a compact model.\nwhich are adapted for speci\ufb01c tasks and datasets, these\nmodels can be quite dif\ufb01cult to adapt for new tasks and\ndatasets, which is disadvantageous.\nInspired by random walk methods [21, 3, 24], in this\nwork we introduce a simple, yet effective alternative to traditional FCNs: a Convolutional Random Walk Network\n(RWN) that combines the strengths of FCNs and random walk methods. Our model addresses the issues of\n(1) poor localization around the boundaries suffered by\nFCNs and (2) spatially disjoint segments produced by dense\nCRFs. Additionally, unlike recent semantic segmentation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4fd-6db4-11ee-b804-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-5", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "(1) poor localization around the boundaries suffered by\nFCNs and (2) spatially disjoint segments produced by dense\nCRFs. Additionally, unlike recent semantic segmentation\napproaches [23, 27, 19, 17], our RWN does so without signi\ufb01cantly increasing the complexity of the model.\nOur proposed RWN jointly optimizes (1) pixelwise af\ufb01nity and (2) semantic segmentation learning objectives that\nare linked via a novel random walk layer, which enforces\nspatial consistency in the deepest layers of the network. The\nrandom walk layer is implemented via matrix multiplication. As a result, RWN seamlessly integrates both af\ufb01nity\nand segmentation branches, and can be jointly trained endto-end via standard back-propagation with minimal modi\ufb01cations to the existing FCN framework. Additionally, our\nimplementation of RWN requires only 131additional parameters. Thus, the effective complexity of our model is\nthe same as the complexity of traditional FCNs (see Table 1). We compare our approach to several variants of the\nDeepLab semantic segmentation system [6, 7], and show\nthat our proposed RWN consistently produces better performance over these baselines for the tasks of semantic segmentation and scene labeling.\n2. Related Work\nThe recent introduction of fully convolutional networks"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4fe-6db4-11ee-8d96-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-6", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "that our proposed RWN consistently produces better performance over these baselines for the tasks of semantic segmentation and scene labeling.\n2. Related Work\nThe recent introduction of fully convolutional networks\n(FCNs) [20] has led to remarkable advances in semantic\nsegmentation. However, due to the large receptive \ufb01elds\nand many pooling layers, segments predicted by FCNs tend\nto be blobby and lack \ufb01ne object boundary details. Recently\nthere have been several attempts to address these problems.\nThese approaches can be divided into several groups.The work in [4, 6, 25, 14, 4] used FCN predictions as\nunary potentials in a separate globalization model that re\ufb01nes the segments using similarity cues based on regions\nor boundaries. One disadvantage of these methods is that\nthe learning of the unary potentials and the training of the\nglobalization model are completely disjoint. As a result,\nthese methods often fail to capture semantic relationships\nbetween objects, which produces segmentation results that\nare spatially disjoint (see the right side of Fig. 1).\nTo address these issues, several recent methods [27, 19,\n17] have proposed to integrate a CRF or a MRF into the network, thus enabling end-to-end training of the joint model."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ff-6db4-11ee-bcc9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-7", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "17] have proposed to integrate a CRF or a MRF into the network, thus enabling end-to-end training of the joint model.\nHowever, the merging of these two models leads to a dramatic increase in complexity and number of parameters.\nFor instance, the method in [27], requires to cast the original\nFCN into a Recurrent Neural Network (RNN), which renders the model much bigger in size (see Table 1). A recent\nmethod [5] jointly predicts boundaries and segmentations,\nand then combines them using a recurrent layer, which also\nrequires complex modi\ufb01cations to the existing FCN framework.\nThe work in [19] proposes to use local convolutional layers, which leads to a signi\ufb01cantly larger number of parameters. Similarly, the method in [17] proposes to model unary\nand pairwise potentials by separate multi-scale branches.\nThis leads to a network with at least twice as many parameters as the traditional FCN and a much more complex multistage training procedure.\nIn addition to the above methods, it is worth mentioning\ndeconvolutional networks [23, 12], which use deconvolution and unpooling layers to recover \ufb01ne object details from\nthe coarse FCN predictions. However, in order to effectively recover \ufb01ne details one must employ nearly as many"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab500-6db4-11ee-8aa6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-8", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "the coarse FCN predictions. However, in order to effectively recover \ufb01ne details one must employ nearly as many\ndeconvolutional layers as the number of convolutional layers, which yields a large growth in number of parameters\n(see Table 1).\nUnlike these prior methods, our implementation of RWN\nneeds only 131additional parameters over the base FCN.\nThese additional parameters represent only 0:0008% of the\ntotal number of parameters in the network. In addition, our\nRWN uses standard convolution and matrix multiplication.\nThus, it does not need to incorporate complicated loss functions or new complex layers [27, 5, 19, 17]. Finally, unlike the methods in [4, 6, 25, 14] that predict and re\ufb01ne the\nsegmentations disjointly, our RWN model jointly optimizes\npixel af\ufb01nity and semantic segmentation in an end-to-end\nfashion. Our experiments show that this leads to spatially\nsmoother segmentations.\n3. Background\nRandom Graph Walks. Random walks are one of\nthe most widely known and used methods in graph theory [21]. Most notably, the concept of random walks led to\nFC8 ActivationsDeepLab FCN Architecture\nRGB ImageSparse Pairwise L1 Distances    Sparse Pixel    Similarity Matrix1 x 1 x k conv\n   Ground Truth   Pixel Similarities"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab501-6db4-11ee-b14a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-9", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "RGB ImageSparse Pairwise L1 Distances    Sparse Pixel    Similarity Matrix1 x 1 x k conv\n   Ground Truth   Pixel Similarities\n Exp\n   Ground Truth     Segmentation\n      Segmentation            Random Walk Layer\u02c6y=AfAffinity Learning BranchPairwise L1  Distances\nn\u21e5n\u21e5mRow-Normalization:n\u21e5n\u21e53\nf2Rn2\u21e5mA2Rn2\u21e5n2W2Rn2\u21e5n2F2Rn2\u21e5n2\u21e5kFigure 2: The architecture of our proposed Random Walk Network (RWN) (best viewed in color). Our RWN consists\nof two branches: (1) one branch devoted to the segmentation predictions , and (2) another branch predicting pixel-level\naf\ufb01nities. These two branches are then merged via a novel random walk layer that encourages spatially smooth segmentation\npredictions. The entire RWN is jointly optimized end-to-end via a standard back-propagation algorithm.\nthe development of PageRank [24] and Personalized PageRank [3], which are widely used for many applications. Let\nG= (V;E)denote an undirected graph with a set of verticesVand a set of edges E. Then a random walk in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab502-6db4-11ee-8c8b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-10", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "G= (V;E)denote an undirected graph with a set of verticesVand a set of edges E. Then a random walk in\nsuch graph can be characterized by the transition probabilities between its vertices. Let Wbe a symmetric n\u0002n\naf\ufb01nity matrix, where ndenotes the number of nodes in\nthe graph and where Wij2[0;1]denotes how similar the\nnodesiandjare. In the context of a semantic segmentation problem, each pixel in the image can be viewed as\na separate node in the graph, where the similarity between\ntwo nodes can be evaluated according to some metric (e.g.\ncolor or texture similarity etc). Then let Dindicate a diagonaln\u0002nmatrix, which stores the degree values for each\nnode:Dii=Pn\nj=1Wijfor alljexcepti=j. Then, we can\nexpress our random walk transition matrix as A=D\u00001W.\nGiven this setup, we want to model how the information\nin the graph spreads if we start at a particular node, and\nperform a random walk in this graph. Let ytbe an\u00021vector denoting a node distribution at time t. In the context of\nthe PageRank algorithm, ytmay indicate the rank estimates\nassociated with each of the nWeb pages at time t. Then,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab503-6db4-11ee-9e75-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-11", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "the PageRank algorithm, ytmay indicate the rank estimates\nassociated with each of the nWeb pages at time t. Then,\naccording to the random walk theory, we can spread the\nrank information in the graph by performing a one-step random walk. This process can be expressed as yt+1=Ayt,\nwhereyt+1denotes a newly obtained rank distribution after\none random walk step, the matrix Acontains the random\nwalk transition probabilities, and ytis the rank distribution\nat time step t. Thus, we can observe that the information\namong the nodes can be diffused, by simply multiplying the\nrandom walk transition probability matrix A, with the rank\ndistribution ytat a particular time t. This process can be\nrepeated multiple times, until convergence is reached. For a\nmore detailed survey please see [21, 24].\nDifference from MRF/CRF Approaches. CRFs andMRFs have been widely used in structured prediction problems [16]. Recently, CRFs and MRFs have also been integrated into the fully convolutional network framework for\nsemantic segmentation [27, 19, 17]. We want to stress that\nwhile the goals of CRF/MRF and random walk methods are\nthe same (i.e. to globally propagate information in the graph\nstructures), the mechanism to achieve this objective is very"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab504-6db4-11ee-8396-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-12", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "the same (i.e. to globally propagate information in the graph\nstructures), the mechanism to achieve this objective is very\ndifferent in these two approaches. While MRFs and CRFs\ntypically employ graphs with a \ufb01xed grid structure (e.g., one\nwhere each node is connected to its four closest neighbors),\nrandom walk methods are much more \ufb02exible and can implement any arbitrary graph structure via the af\ufb01nity matrix\nspeci\ufb01cation. Thus, since our proposed RWN is based on\nrandom walks, it can employ any arbitrary graph structure,\nwhich can be bene\ufb01cial as different problems may require\ndifferent graph structures.\nAdditionally, to globally propagate information among\nthe nodes, MRFs and CRFs need to employ approximate\ninference techniques, because exact inference tends to be\nintractable in graphs with a grid structure. Integrating such\napproximate inference techniques into the steps of FCN\ntraining and prediction can be challenging and may require\nlots of domain-speci\ufb01c modi\ufb01cations. In comparison, random walk methods globally propagate information among\nthe nodes via a simple matrix multiplication. Not only is\nthe matrix multiplication ef\ufb01cient and exact, but it is also\neasy to integrate into the traditional FCN framework for\nboth training and prediction schemes. Additionally, due to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab505-6db4-11ee-954b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-13", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "the matrix multiplication ef\ufb01cient and exact, but it is also\neasy to integrate into the traditional FCN framework for\nboth training and prediction schemes. Additionally, due to\nthe use of standard convolution and matrix multiplication\noperations, our RWN can be trivially trained via standard\nback-propagation in an end-to-end fashion.\n4. Convolutional Random Walk Networks\nIn this work, our goal is to integrate a random walk process into the FCN architecture to encourage coherent semantic segmentation among pixels that are similar to each\nother. Such a process introduces an explicit grouping mechanism, which should be bene\ufb01cial in addressing the issues\nof (1) poor localization around the boundaries, and (2) spatially fragmented segmentations.\nA schematic illustration of our proposed RWN architecture is presented in Fig. 2. Our RWN is a network composed of two branches: (1) one branch that predicts semantic segmentation potentials, and (2) another branch devoted\nto predicting pixel-level af\ufb01nities. These two branches are\nmerged via a novel random walk layer that encourages spatially coherent semantic segmentation. The entire RWN can\nbe jointly optimized end-to-end. We now describe each of\nthe components of the RWN architecture in more detail.\n4.1. Semantic Segmentation Branch\nFor the semantic segmentation branch, we present results"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab506-6db4-11ee-9f35-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-14", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "the components of the RWN architecture in more detail.\n4.1. Semantic Segmentation Branch\nFor the semantic segmentation branch, we present results\nfor several variants of the DeepLab segmentation systems,\nincluding DeepLab-LargeFOV [6], DeepLab-attention [7],\nand DeepLab-v2, which is one of the top performing segmentation systems. DeepLab-largeFOV is a fully convolutional adaptation of the VGG [26] architecture, which contains 16convolutional layers. DeepLab-attention [7], is a\nmulti-scale VGG based network, for which each multi-scale\nbranch focuses on a speci\ufb01c part of the image. Finally,\nDeepLab-v2 is a multi-scale network based on the residual network [11] implementation. We note that even though\nwe use a DeepLab architecture in our experiments, other architectures such as [2] and many others could be integrated\ninto our framework.\n4.2. Pixel-Level Af\ufb01nity Branch\nTo learn the pairwise pixel-level af\ufb01nities, we employ a\nseparate af\ufb01nity learning branch with its own learning objective (See Fig. 2). The af\ufb01nity branch is connected with\nthe inputn\u0002n\u00023RGB image, and low-level conv1_1and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab507-6db4-11ee-8901-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-15", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "the inputn\u0002n\u00023RGB image, and low-level conv1_1and\nconv1_2layers. The feature maps corresponding to these\nlayers aren\u0002nin width and height but they have a different number of channels ( 3;64;and64respectively). Let k\nbe the total number of af\ufb01nity learning parameters (in our\ncasek= 3 + 64 + 64 = 131 ). Then, let Fbe a sparse\nn2\u0002n2\u0002kmatrix that stores L1distances between each\npixel and all of its neighbors within a radius R, according\nto each channel. Note that the distances are notsummed\nup across the kchannels, but instead they are computed and\nstored separately for each channel. The resulting matrix F\nis then used as an input to the af\ufb01nity branch, as shown in\nFigure 2.\nThe af\ufb01nity branch consists of a 1\u00021\u0002kconvolutional\nlayer and an exponential layer. The output of the exponential layer is then attached to the Euclidean loss layer and is\noptimized to predict the ground truth pixel af\ufb01nities, which\nare obtained from the original semantic segmentation anInput\n DeepLab_v2\n RWN_v2\nFigure 3: A \ufb01gure illustrating the segmentation results of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab508-6db4-11ee-9088-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-16", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "are obtained from the original semantic segmentation anInput\n DeepLab_v2\n RWN_v2\nFigure 3: A \ufb01gure illustrating the segmentation results of\nour RWN and the DeepLab-v2 network. Note that RWN\nproduced segmentations are spatially smoother and produce\nless false positive predictions than the DeepLab-v2 system.\nnotations. Speci\ufb01cally, we set the ground truth af\ufb01nity between two pixels to 1if the pixels share the same semantic\nlabel and have distance less than Rfrom each other. Note\nthatF, which is used as an input to the af\ufb01nity branch, is a\nsparse matrix, as only a small fraction of all the entries in F\nare populated with non-zero values. The rest of the entries\nare ignored during the computation.\nAlso note that we only use features from RGB, conv1_1\nandconv1_2layers, because they are not affected by pooling, and thus, preserve the original spatial resolution. We\nalso experimented with using features from deeper FCN\nlayers such as fc6, and fc7. However, we observed that\nfeatures from deeper layers are highly correlated to the\npredicted semantic segmentation unary potentials, which\ncauses redundancy and little improvement in the segmentation performance. We also experimented with using more"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab509-6db4-11ee-b609-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-17", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "features from deeper layers are highly correlated to the\npredicted semantic segmentation unary potentials, which\ncauses redundancy and little improvement in the segmentation performance. We also experimented with using more\nthan one convolutional layer in the af\ufb01nity learning branch,\nbut observed that additional layers provide negligible improvements in accuracy.\n4.3. Random Walk Layer\nTo integrate the semantic segmentation potentials and\nour learned pixel-level af\ufb01nities, we introduce a novel random walk layer, which propagates the semantic segmentation information based on the learned af\ufb01nities. The random\nwalk layer is connected to the two bottom layers: (1) the fc8\nlayer containing the semantic segmentation potentials, and\n(2) the af\ufb01nity layer that outputs a sparse n2\u0002n2random\nwalk transition matrix A. Then, let fdenote the activation values from the fc8layer, reshaped to the dimensions\nofn2\u0002m, wheren2refers to the number of pixels, and\nmis the number of object classes in the dataset. A single\nrandom walk layer implements one step of the random walk\nMethod aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean overall"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab50a-6db4-11ee-8778-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-18", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "random walk layer implements one step of the random walk\nMethod aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean overall\nDeepLab-largeFOV 79.8 71.5 78.9 70.9 72.1 87.9 81.2 85.7 46.9 80.9 56.5 82.6 77.9 79.3 80.1 64.4 77.6 52.7 80.3 70.0 73.8 76.0\nRWN-largeFOV 81.6 72.1 82.3 72.0 75.4 89.1 82.5 87.4 49.1 83.6 57.9 84.8 80.7 80.2 81.2 65.7 79.7 55.5 81.5 74.0 75.8 77.9\nDeepLab-attention 83.4 76.0 83.0 74.2 77.6 91.6 85.2 89.1 54.4 86.1 62.9 86.7 83.8 84.2 82.4 70.2 84.7 61.0 84.8 77.9 79.0 80.5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab50b-6db4-11ee-911e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-19", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "RWN-attention 84.7 76.6 85.5 74.0 79.0 92.4 85.6 90.0 55.6 87.4 63.5 88.2 85.0 84.8 83.4 70.1 85.9 62.6 85.1 79.3 79.9 81.5\nDeepLab-v2 85.5 50.6 86.9 74.4 82.7 93.1 88.4 91.9 62.1 89.7 71.5 90.3 86.2 86.3 84.6 75.1 87.6 72.2 87.8 81.3 81.4 83.4\nRWN-v2 86.0 50.0 88.4 73.5 83.9 93.4 88.6 92.5 63.9 90.9 72.6 90.9 87.3 86.9 85.7 75.0 89.0 74.0 88.1 82.3 82.1 84.3\nTable 2: Semantic segmentation results on the SBD dataset according to the per-pixel Intersection over Union evaluation\nmetric. From the results, we observe that our proposed RWN consistently outperforms DeepLab-LargeFOV , DeepLabattention, and DeepLab-v2 baselines."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab50c-6db4-11ee-ab35-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-20", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "metric. From the results, we observe that our proposed RWN consistently outperforms DeepLab-LargeFOV , DeepLabattention, and DeepLab-v2 baselines.\nprocess, which can be performed as ^y=Af, where ^yindicates the diffused segmentation predictions, and Adenotes\nthe random walk transition matrix.\nThe random walk layer is then attached to the softmax\nloss layer, and is optimized to predict ground truth semantic\nsegmentations. One of the advantages of our proposed random walk layer is that it is implemented as a matrix multiplication, which makes it possible to back-propagate the\ngradients to both (1) the af\ufb01nity branch and (2) the segmentation branch. Let the softmax-loss gradient be an n2\u0002m\nmatrix@L\n@^y, wheren2is the number of pixels in the fc8layer,\nandmis the number of predicted object classes. Then the\ngradients, which are back-propagated to the semantic segmentation branch are computed as@L\n@f=AT@L\n@^y, whereAT\nis the transposed random walk transition matrix. Also, the\ngradients, that are back-propagated to the af\ufb01nity branch\nare computed as@L\n@A=@L"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab50d-6db4-11ee-ac1c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-21", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "gradients, that are back-propagated to the af\ufb01nity branch\nare computed as@L\n@A=@L\n@^yfT, wherefTis am\u0002n2matrix\nthat contains transposed activation values from the fc8layer\nof the segmentation branch. We note that@L\n@Ais asparse\nn2\u0002n2matrix, which means that the above matrix multiplication only considers the pixel pairs that correspond to\nthe non-zero entries in the random walk transition matrix A.\n4.4. Random Walk Prediction at Testing\nIn the previous subsection, we mentioned that the prediction in the random walk layer can be done via a simple\nmatrix multiplication operation ^y=Af, whereAdenotes\nthe random walk transition matrix, and fdepicts the activation values from the fc8layer. Typically, we want to apply\nmultiple random walk steps until convergence is reached.\nHowever, we also do not want to deviate too much from our\ninitial segmentation predictions, in case the random walk\ntransition matrix is not perfectly accurate, which is a reasonable expectation. Thus, our prediction scheme needs to\nbalance two effects: (1) propagating the segmentation information across the nodes using a random walk transition\nmatrix, and (2) not deviating too much from the initial segmentation."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab50e-6db4-11ee-991c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-22", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "balance two effects: (1) propagating the segmentation information across the nodes using a random walk transition\nmatrix, and (2) not deviating too much from the initial segmentation.\nThis tradeoff between the two quantities is very similar to the idea behind MRF and CRF models, which try to\nminimize an energy formed by unary and pairwise terms.\nHowever, as discussed earlier, MRF and CRF methods tend\nto use 1) grid structure graphs and 2) various approximateinference techniques to propagate segmentation information globally. In comparison, our random walk approach\nis advantageous because it can use 1) any arbitrary graph\nstructure and 2) an exact matrix multiplication operation to\nachieve the same goal.\nLet us \ufb01rst denote our segmentation prediction after t+1\nrandom walk steps as ^yt+1. Then our general prediction\nscheme can be written as:\n^yt+1=\u000bA^yt+ (1\u0000\u000b)f (1)\nwhere\u000bdenotes a parameter [0;1]that controls the\ntradeoff between (1) diffusing segmentation information\nalong the connections of a random walk transition matrix\nand (2) not deviating too much from initial segmentation\nvalues (i.e. the outputs of the last layer of the FCN). Let us\nnow initialize ^y0to contain the output values from the fc8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab50f-6db4-11ee-9b7d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-23", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "values (i.e. the outputs of the last layer of the FCN). Let us\nnow initialize ^y0to contain the output values from the fc8\nlayer, which we denoted with f. Then we can write our prediction equation by substituting the recurrent expressions:\n^yt+1= (\u000bA)t+1f+ (1\u0000\u000b)tX\ni=0(\u000bA)if (2)\nNow, because we want to apply our random walk procedure until convergence we set t=1. Then, because\nour random walk transition matrix is stochastic we know\nthatlimt!1(\u000bA)t+1= 0 . Furthermore, we can write\nSt=Pt\ni=0Ai=I+A+A2+:::+At, whereIis\nan identity matrix, and where Stdenotes a partial sum of\nrandom walk transitions until iteration t. We can then write\nSt\u0000ASt=I\u0000At+1, which implies:\n(I\u0000A)St=I\u0000At+1(3)\nFrom our previous derivation, we already know that\nlimt!1(A)t+1= 0, which implies that\nS1= (I\u0000A)\u00001(4)\nThus, our \ufb01nal prediction equation, which corresponds\nto applying repeated random walk steps until convergence,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab510-6db4-11ee-8341-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-24", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "S1= (I\u0000A)\u00001(4)\nThus, our \ufb01nal prediction equation, which corresponds\nto applying repeated random walk steps until convergence,\ncan be written as\n^y1= (I\u0000\u000bA)\u00001f (5)\nInput\n DeepLab_v2-CRF\n RWN_v2\nFigure 4: Comparison of segmentation results produced by\nour RWN versus the DeepLab-v2-CRF system. It can be noticed that, despite not using any post-processing steps, our\nRWN predicts \ufb01ne object details (e.g., bike wheels or plane\nwings) more accurately than DeepLab-v2-CRF, which fails\nto capture some these object parts.\nIn practice, the random walk transition matrix Ais pretty\nlarge, and inverting it is impractical. To deal with this problem, we shrink the matrix (I\u0000\u000bA)using a simple and ef\ufb01cient technique presented in [1], and then invert it to compute the \ufb01nal segmentation. In the experimental section, we\nshow that such a prediction scheme produces solid results\nand is still pretty ef\ufb01cient ( \u00191 second per image). We also\nnote that we use this prediction scheme only during testing. During training we employ a scheme that uses a single"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab511-6db4-11ee-ade6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-25", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "note that we use this prediction scheme only during testing. During training we employ a scheme that uses a single\nrandom walk step (but with a much larger radius), which is\nfaster. We explain this procedure in the next subsection.\n4.5. Implementation Details\nWe jointly train our RWN in an end-to-end fashion for\n2000 iterations, with a learning rate of 10\u00005,0:9momentum, the weight decay of 5\u000110\u00005, and 15samples per batch.\nFor the RWN model, we set the tradeoff parameter \u000bto\n0:01. During testing we set the random walk connectivity\nradiusR= 5 and apply the random walk procedure until convergence. However, during training we set R= 40 ,\nand apply a single random walk step. This training strategy\nworks well because increasing the radius size eliminates the\nneed for multiple random walk steps, which speeds up the\ntraining. However, using R= 5 and applying an in\ufb01nite\nnumber of random walk steps until convergence still yields\nslightly better results (see study in 5.4), so we use it during testing. For all of our experiments, we use a Caffe library [13]. During training, we also employ data augmentation techniques such as cropping, and mirroring.\n5. Experimental Results"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab512-6db4-11ee-8140-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-26", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "5. Experimental Results\nIn this section, we present our results for semantic segmentation on the SBD [10] dataset, which contains objects\nand their per-pixel labels for 20Pascal VOC classes (excluding the background class). We also include scene label-Method mean IOU overall IOU\nDeepLab-largeFOV-CRF 75.7 77.7\nRWN-largeFOV 75.8 77.9\nDeepLab-attention-CRF 79.9 81.6\nRWN-attention 79.9 81.5\nDeepLab-v2-CRF 81.9 84.2\nRWN-v2 82.1 84.3\nDeepLab-DT 76.6 78.7\nRWN 76.7 78.8\nTable 3: Quantitative comparison between our RWN model\nand several variants of the DeepLab system that use a dense\nCRF or a domain-transfer (DT) \ufb01lter for post-processing.\nThese results suggest that our RWN acts as an effective\nglobalization scheme, since it produces results that are\nsimilar or even better than the results achieved by postprocessing the DeepLab outputs with a CRF or DT.\ning results on the commonly used Stanford background [9]\nand Sift Flow [18] datasets. We evaluate our segmentation results on these tasks using the standard metric of the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab513-6db4-11ee-bfc8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-27", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "ing results on the commonly used Stanford background [9]\nand Sift Flow [18] datasets. We evaluate our segmentation results on these tasks using the standard metric of the\nintersection-over-union (IOU) averaged per pixels across all\nthe classes from each dataset. We also include the classagnostic overall pixel intersection-over-union score, which\nmeasures the per-pixel IOU across all classes.\nWe experiment with several variants of the DeepLab\nsystem [6, 7] as our main baselines throughout our experiments: DeepLab-LargeFOV [6], DeepLab-attention [7],\nand DeepLab-v2.\nOur evaluations provide evidence for four conclusions:\n\u000fIn subsections 5.1, 5.2, we demonstrate that our proposed RWN outperforms DeepLab baselines for both\nsemantic segmentation, and scene labeling tasks.\n\u000fIn subsection 5.1, we demonstrate that, compared to\nthe dense CRF approaches, RWN predicts segmentations that are spatially smoother.\n\u000fIn Subsection 5.3, we show that our approach is more\nef\ufb01cient than the denseCRF inference.\n\u000fFinally, in Subsection, 5.4, we demonstrate that our\nrandom walk layer is bene\ufb01cial and that our model is\n\ufb02exible to use different graph structures."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab514-6db4-11ee-b4a5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-28", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "random walk layer is bene\ufb01cial and that our model is\n\ufb02exible to use different graph structures.\n5.1. Semantic Segmentation Task\nStandard Evaluation. In Table 2, we present semantic\nsegmentation results on the Pascal SBD dataset [10], which\ncontains 8055 training and 2857 testing images. These results indicate that RWN consistently outperforms all three\nof the DeepLab baselines. In Figure 3, we also compare\nqualitative segmentation results of a DeepLab-v2 network\nand our RWN model. We note that the RWN segmentations contain fewer false positive predictions and are also\nspatially smoother across the object regions.\n0 2 4 6 8 10 12 14 16 18 200.20.250.30.350.40.450.50.55\nTrimap Width (in Pixels)Pixel Classification Error (%)Localization Around the Boundaries Error\n  \nDeepLab\nRWNFigure 5: Localization error around the object boundaries\nwithin a trimap. Compared to the DeepLab system (blue),\nour RWN (red) achieves lower segmentation error around\nobject boundaries for all trimap widths.\nFurthermore, in Table 3, we present experiments where\nwe compare RWN with models using dense CRFs [15] to\npost-process the predictions of DeepLab systems. We also"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab515-6db4-11ee-a344-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-29", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "Furthermore, in Table 3, we present experiments where\nwe compare RWN with models using dense CRFs [15] to\npost-process the predictions of DeepLab systems. We also\ninclude DeepLab-DT [5], which uses domain-transfer \ufb01ltering to re\ufb01ne the segmentations inside an FCN. Based on\nthese results, we observe that, despite not using any postprocessing, our RWN produces results similar to or even\nbetter than the DeepLab models employing post-processing.\nThese results indicate that RWN can be used as a globalization mechanism to ensure spatial coherence in semantic\nsegmentation predictions. In Figure 4 we present qualitative results where we compare the \ufb01nal segmentation predictions of RWN and the DeepLab-v2-CRF system. Based\non these qualitative results, we observe that RWN captures\nmore accurately the \ufb01ne details of the objects, such as the\nbike wheels, or plane wings. The DeepLab-v2-CRF system\nmisses some of these object parts.\nLocalization Around the Boundaries. Earlier we\nclaimed that due to the use of large receptive \ufb01elds and\nmany pooling layers, FCNs tend to produce blobby segmentations that lack \ufb01ne object boundary details. We want"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab516-6db4-11ee-a05d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-30", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "many pooling layers, FCNs tend to produce blobby segmentations that lack \ufb01ne object boundary details. We want\nto show that our RWN produces more accurate segmentations around object boundaries the traditional FCNs. Thus,\nadopting the practice from [15], we evaluate segmentation\naccuracy around object boundaries. We do so by counting\nthe relative number of misclassi\ufb01ed pixels within a narrow\nband (\u201ctrimap\u201d) surrounding the ground truth object boundaries. We present these results in Figure 5. The results show\nthat RWN achieves higher segmentation accuracy than the\nDeepLab (DL) system for all trimap widths considered in\nthis test.\nSpatial Smoothness. We also argued that applying the\ndense CRF [15] as a post-processing technique often leads\nto spatially fragmented segmentations (see the right side of\nFig. 1). How can we evaluate whether a given method produces spatially smooth or spatially fragmented segmenta-Method MF AP\nDeepLab-largeFOV-CRF 0.676 0.457\nRWN-largeFOV 0.703 0.494\nDeepLab-attention-CRF 0.722 0.521\nRWN-attention 0.747 0.556\nDeepLab-v2-CRF 0.763 0.584"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab517-6db4-11ee-b5ad-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-31", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "RWN-attention 0.747 0.556\nDeepLab-v2-CRF 0.763 0.584\nRWN-v2 0.773 0.595\nTable 4: Quantitative comparison of spatial segmentation\nsmoothness. We extract the boundaries from the predicted\nsegmentations and evaluate them against ground truth object boundaries using max F-score (MF) and average precision (AP) metrics. These results suggest that RWN segmentations are spatially smoother than the DeepLab-CRF\nsegmentations across all baselines.\ntions? Intuitively, spatially fragmented segmentations will\nproduce many false boundaries that do not correspond to\nactual object boundaries. Thus, to test the spatial smoothness of a given segmentation, we extract the boundaries\nfrom the segmentation and then compare these boundaries\nagainst ground truth object boundaries using the standard\nmaximum F-score (MF) and average precision (AP) metrics, as done in the popular BSDS benchmark [22]. We perform this experiment on the Pascal SBD dataset and present\nthese results in Table 4. We can see that the boundaries extracted from the RWN segmentations yield better MF and\nAP results compared to the boundaries extracted from the\ndifferent variants of the DeepLab-CRF system. Thus, these\nresults suggest that RWN produces spatially smoother segmentations than DeepLab-CRF.\n5.2. Scene Labeling"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab518-6db4-11ee-bd6c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-32", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "results suggest that RWN produces spatially smoother segmentations than DeepLab-CRF.\n5.2. Scene Labeling\nWe also tested our RWN on the task of scene labeling\nusing two popular datasets: Stanford Background [9] and\nSift Flow [18]. Stanford Background is a relatively small\ndataset for scene labeling. It contains 715images, which\nwe randomly split into 600training images and 115testing images. In contrast, the Sift Flow dataset contains 2489\ntraining examples and 201testing images. For all of our\nexperiments, we use the DeepLab-largeFOV [6] architecture since it is smaller and more ef\ufb01cient to train and test.\nTo evaluate scene labeling results, we use the overall IOU\nevaluation metric which is a commonly used metric for this\ntask. In Table 5, we present our scene labeling results on\nboth of these datasets. Our results indicate that our RWN\nmethod outperforms the DeepLab baseline by 2:57%, and\n2:54% on these two datasets, respectively.\n5.3. Runtime Comparisons\nWe also include the runtime comparison of our RWN approach versus the denseCRF inference. We note that using a single core of a 2:7GHz Intel Core i7 processor, the\ndenseCRF inference requires 3.301 seconds per image on\nInput Image\nIteration 0Iteration 50"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab519-6db4-11ee-8e2c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-33", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "denseCRF inference requires 3.301 seconds per image on\nInput Image\nIteration 0Iteration 50\nFigure 6: A \ufb01gure illustrating how the probability predictions change as we apply more random walk steps. Note\nthat the RWN predictions become more re\ufb01ned and better localized around the object boundaries as more random\nwalk steps are applied.\naverage on a Pascal SBD dataset. In comparison, a single\niteration of a random walk, which is simply a sparse matrix\nmultiplication, takes 0.032 seconds on average on the same\nPascal SBD dataset. A DeepLab_v2 post-processed with\ndenseCRF achieves 81:9%IOU score on this same Pascal\nSBD dataset. In comparison, RWN_v2 with a single random walk iteration and with R=40 (radius) achieves 82:2%\nIOU, which is both more accurate and more than 100times\nmore ef\ufb01cient than the denseCRF inference.\n5.4. Ablation Experiments\nOptimal Number of Random Walk Steps. In Figure 7,\nwe illustrate how the IOU accuracy changes when we use a\ndifferent number of random walk steps. We observe that the\nsegmentation accuracy keeps increasing as we apply more\nrandom walk steps, and that it reaches its peak performance\nwhen the random walk process converges, which indicates"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab51a-6db4-11ee-b03d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-34", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "segmentation accuracy keeps increasing as we apply more\nrandom walk steps, and that it reaches its peak performance\nwhen the random walk process converges, which indicates\nthe effectiveness of our random walk step procedure. In\nFigure 6, we also illustrate how the predicted object segmentation probabilities change as we apply more random\nwalk steps. We observe that the object boundaries become\nmuch better localized as more iterations of random walk are\napplied.\nRadius Size. To analyze the effect of a radius size in\nthe RWN architecture, we test alternative versions of our\nmodel with different radii sizes. Our results indicate, that\nthe RWN model produces similar results with different radii\nin the interval of R > 3andR < 20if the random walk\nstep process is applied until convergence. We also note\nthat, if we select R= 40 , and apply a random walk step\nonly once , we can achieve the segmentation accuracy of\n75:5%and77:6%according to the two evaluation metrics,\n0 5 10 15 20 25 30 40 50 60 inf0.7350.740.7450.750.7550.760.765\nNumber of Random Walk StepsIOU AccuracyAccuracy versus the Number of Random Walk Steps"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab51b-6db4-11ee-a910-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-35", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "Number of Random Walk StepsIOU AccuracyAccuracy versus the Number of Random Walk Steps\n  \nMean\u2212class IOUFigure 7: IOU accuracy as a function of the number of random walk steps. From this plot we observe that the segmentation accuracy keeps improving as we apply more random walk steps and that it reaches its peak when the random\nwalk process converges.\nDeepLab-largeFOV RWN-largeFOV\nStanford-BG 65.74 68.31\nSift-Flow 67.31 69.85\nTable 5: Scene labeling results on the Stanford Background\nand Sift-Flow datasets measured according to the overall\nIOU evaluation metric. We use a DeepLab-largeFOV network as base model, and show that our RWN yields better\nresults on both of these scene labeling datasets.\nrespectively. In comparison, choosing R= 5 and applying random walk until convergence yields the accuracies of\n75:8%and77:9%, which is slightly better. However, note\nthat selecting R= 40 , and applying multiple random walk\nsteps does not yield any improvement in segmentation accuracy. These experiments show the \ufb02exibility of our model\ncompared to the MRF or CRF models, which typically use\ngraphs with a \ufb01xed grid structure. Our model has the ability\nto use different graph structures depending on the problem.\n6. Conclusion"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab51c-6db4-11ee-991a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-36", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "graphs with a \ufb01xed grid structure. Our model has the ability\nto use different graph structures depending on the problem.\n6. Conclusion\nIn this work, we introduced Random Walk Networks\n(RWNs), and showed that, compared to traditional fully\nconvolutional networks (FCNs), they produce improved accuracy for the same model complexity. Our RWN addresses\nthe issues of 1) poor localization around the segmentation\nboundaries and 2) spatially disjoint segmentations. Additionally, our implementation of RWN uses only 131additional learnable parameters ( 0:0008% of the original number of the parameters in the network) and it can be easily\nintegrated into the standard FCN learning framework for a\njoint end-to-end training. Finally, RWN provides a more\nef\ufb01cient alternative to the denseCRF approaches.\nOur future work includes experimenting with alternative RWN architectures and applying RWN to new domains\nsuch as language processing or speech recognition.\nReferences\n[1] Pablo Arbelaez, J. Pont-Tuset, Jon Barron, F. Marqu\u00e9s, and\nJitendra Malik. Multiscale combinatorial grouping. In Computer Vision and Pattern Recognition (CVPR) , 2014. 6\n[2] Vijay Badrinarayanan, Ankur Handa, and Roberto Cipolla."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab51d-6db4-11ee-beb7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-37", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "[2] Vijay Badrinarayanan, Ankur Handa, and Roberto Cipolla.\nSegnet: A deep convolutional encoder-decoder architecture\nfor robust semantic pixel-wise labelling. arXiv preprint\narXiv:1505.07293 , 2015. 4\n[3] Bahman Bahmani, Abdur Chowdhury, and Ashish Goel. Fast\nincremental and personalized pagerank. Proc. VLDB Endow., 4(3):173\u2013184, December 2010. 2, 3\n[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Semantic segmentation with boundary neural \ufb01elds. In The\nIEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 2\n[5] Liang-Chieh Chen, Jonathan T. Barron, George Papandreou,\nKevin Murphy, and Alan L. Yuille. Semantic image segmentation with task-speci\ufb01c edge detection using cnns and a\ndiscriminatively trained domain transform. CVPR , 2016. 1,\n2, 7\n[6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab51e-6db4-11ee-ba57-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-38", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "2, 7\n[6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,\nKevin Murphy, and Alan L. Yuille. Semantic image segmentation with deep convolutional nets and fully. In ICLR , 2015.\n1, 2, 4, 6, 7\n[7] Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and\nAlan L. Yuille. Attention to scale: Scale-aware semantic\nimage segmentation. CVPR , 2016. 2, 4, 6\n[8] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting\nbounding boxes to supervise convolutional networks for semantic segmentation. In The IEEE International Conference\non Computer Vision (ICCV) , December 2015. 1\n[9] S. Gould, R. Fulton, and D. Koller. Decomposing a scene\ninto geometric and semantically consistent regions. In Proceedings of the International Conference on Computer Vision (ICCV) , 2009. 6, 7\n[10] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev,\nSubhransu Maji, and Jitendra Malik. Semantic contours from\ninverse detectors. In International Conference on Computer\nVision (ICCV) , 2011. 6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab51f-6db4-11ee-952a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-39", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "Subhransu Maji, and Jitendra Malik. Semantic contours from\ninverse detectors. In International Conference on Computer\nVision (ICCV) , 2011. 6\n[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.\nDeep residual learning for image recognition. In The IEEE\nConference on Computer Vision and Pattern Recognition\n(CVPR) , June 2016. 4\n[12] Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled deep neural network for semi-supervised semantic\nsegmentation. In NIPS) , December 2015. 1, 2\n[13] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey\nKarayev, Jonathan Long, Ross Girshick, Sergio Guadarrama,\nand Trevor Darrell. Caffe: Convolutional architecture for fast\nfeature embedding. arXiv preprint arXiv:1408.5093 , 2014.\n2, 6\n[14] Iasonas Kokkinos. Surpassing humans in boundary detection\nusing deep learning. CoRR , abs/1511.07386, 2015. 2[15] Philipp Kr\u00e4henb\u00fchl and Vladlen Koltun. Ef\ufb01cient inference\nin fully connected crfs with gaussian edge potentials. In"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab520-6db4-11ee-b641-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-40", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "in fully connected crfs with gaussian edge potentials. In\nJ. Shawe-Taylor, R.S. Zemel, P.L. Bartlett, F. Pereira, and\nK.Q. Weinberger, editors, Advances in Neural Information\nProcessing Systems 24 , pages 109\u2013117. Curran Associates,\nInc., 2011. 1, 7\n[16] John D. Lafferty, Andrew McCallum, and Fernando C. N.\nPereira. Conditional random \ufb01elds: Probabilistic models for\nsegmenting and labeling sequence data. In Proceedings of\nthe Eighteenth International Conference on Machine Learning, ICML \u201901, pages 282\u2013289, San Francisco, CA, USA,\n2001. Morgan Kaufmann Publishers Inc. 3\n[17] Guosheng Lin, Chunhua Shen, Ian D. Reid, and Anton\nvan den Hengel. Ef\ufb01cient piecewise training of deep\nstructured models for semantic segmentation. CoRR ,\nabs/1504.01013, 2015. 1, 2, 3\n[18] Ce Liu, Jenny Yuen, and Antonio Torralba. Nonparametric\nScene Parsing via Label Transfer , pages 207\u2013236. Springer\nInternational Publishing, Cham, 2016. 6, 7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab521-6db4-11ee-a8e7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-41", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "Scene Parsing via Label Transfer , pages 207\u2013236. Springer\nInternational Publishing, Cham, 2016. 6, 7\n[19] Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen Change Loy, and\nXiaoou Tang. Semantic image segmentation via deep parsing\nnetwork. In ICCV , 2015. 1, 2, 3\n[20] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully\nconvolutional networks for semantic segmentation. In The\nIEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015. 1, 2\n[21] Laszlo Lovasz. Random walks on graphs: A survey, 1993.\n2, 3\n[22] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database\nof human segmented natural images and its application to\nevaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int\u2019l Conf. Computer Vision , volume 2, pages 416\u2013423, July 2001. 7\n[23] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han.\nLearning deconvolution network for semantic segmentation.\nInComputer Vision (ICCV), 2015 IEEE International Conference on , 2015. 1, 2\n[24] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab522-6db4-11ee-a4fa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1605.07681-42", "title": "Convolutional Random Walk Networks for Semantic Image Segmentation", "source": "http://arxiv.org/pdf/1605.07681", "start_index": 0, "chroma:document": "InComputer Vision (ICCV), 2015 IEEE International Conference on , 2015. 1, 2\n[24] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry\nWinograd. The pagerank citation ranking: Bringing order\nto the web. Technical Report 1999-66, Stanford InfoLab,\nNovember 1999. Previous number = SIDL-WP-1999-0120.\n2, 3\n[25] Xiaojuan Qi, Jianping Shi, Shu Liu, Renjie Liao, and Jiaya Jia. Semantic segmentation with object clique potential.\nInThe IEEE International Conference on Computer Vision\n(ICCV) , December 2015. 2\n[26] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR ,\nabs/1409.1556, 2014. 4\n[27] Shuai Zheng, Sadeep Jayasumana, Bernardino RomeraParedes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang\nHuang, and Philip Torr. Conditional random \ufb01elds as recurrent neural networks. In International Conference on Computer Vision (ICCV) , 2015. 1, 2, 3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab523-6db4-11ee-a497-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-0", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "Learning to learn by gradient descent\nby gradient descent\nMarcin Andrychowicz1, Misha Denil1, Sergio G\u00f3mez Colmenarejo1, Matthew W. Hoffman1,\nDavid Pfau1, Tom Schaul1, Brendan Shillingford1;2, Nando de Freitas1;2;3\n1Google DeepMind2University of Oxford3Canadian Institute for Advanced Research\nmarcin.andrychowicz@gmail.com\n{mdenil,sergomez,mwhoffman,pfau,schaul}@google.com\nbrendan.shillingford@cs.ox.ac.uk ,nandodefreitas@google.com\nAbstract\nThe move from hand-designed features to learned features in machine learning has\nbeen wildly successful. In spite of this, optimization algorithms are still designed\nby hand. In this paper we show how the design of an optimization algorithm can be\ncast as a learning problem, allowing the algorithm to learn to exploit structure in\nthe problems of interest in an automatic way. Our learned algorithms, implemented\nby LSTMs, outperform generic, hand-designed competitors on the tasks for which\nthey are trained, and also generalize well to new tasks with similar structure. We\ndemonstrate this on a number of tasks, including simple convex problems, training\nneural networks, and styling images with neural art."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab524-6db4-11ee-b368-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-1", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "demonstrate this on a number of tasks, including simple convex problems, training\nneural networks, and styling images with neural art.\n1 Introduction\nFrequently, tasks in machine learning can be expressed as the problem of optimizing an objective\nfunctionf(\u0012)de\ufb01ned over some domain \u00122\u0002. The goal in this case is to \ufb01nd the minimizer\n\u0012\u0003= arg min\u00122\u0002f(\u0012). While any method capable of minimizing this objective function can be\napplied, the standard approach for differentiable functions is some form of gradient descent, resulting\nin a sequence of updates\n\u0012t+1=\u0012t\u0000\u000btrf(\u0012t):\nThe performance of vanilla gradient descent, however, is hampered by the fact that it only makes use\nof gradients and ignores second-order information. Classical optimization techniques correct this\nbehavior by rescaling the gradient step using curvature information, typically via the Hessian matrix\nof second-order partial derivatives\u2014although other choices such as the generalized Gauss-Newton\nmatrix or Fisher information matrix are possible.\nMuch of the modern work in optimization is based around designing update rules tailored to speci\ufb01c\nclasses of problems, with the types of problems of interest differing between different research"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab525-6db4-11ee-a038-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-2", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "Much of the modern work in optimization is based around designing update rules tailored to speci\ufb01c\nclasses of problems, with the types of problems of interest differing between different research\ncommunities. For example, in the deep learning community we have seen a proliferation of optimization methods specialized for high-dimensional, non-convex optimization problems. These include\nmomentum [Nesterov, 1983, Tseng, 1998], Rprop [Riedmiller and Braun, 1993], Adagrad [Duchi\net al., 2011], RMSprop [Tieleman and Hinton, 2012], and ADAM [Kingma and Ba, 2015]. More\nfocused methods can also be applied when more structure of the optimization problem is known\n[Martens and Grosse, 2015]. In contrast, communities who focus on sparsity tend to favor very\ndifferent approaches [Donoho, 2006, Bach et al., 2012]. This is even more the case for combinatorial\noptimization for which relaxations are often the norm [Nemhauser and Wolsey, 1988].\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.arXiv:1606.04474v2  [cs.NE]  30 Nov 2016\noptimizeroptimizeeparameter updates\nerror signalFigure 1: The optimizer (left) is provided with\nperformance of the optimizee (right) and proposes"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab526-6db4-11ee-a35e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-3", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "optimizeroptimizeeparameter updates\nerror signalFigure 1: The optimizer (left) is provided with\nperformance of the optimizee (right) and proposes\nupdates to increase the optimizee\u2019s performance.\n[photos: Bobolas, 2009, Maley, 2011]This industry of optimizer design allows different communities to create optimization methods which exploit structure in their problems\nof interest at the expense of potentially poor\nperformance on problems outside of that scope.\nMoreover the No Free Lunch Theorems for Optimization [Wolpert and Macready, 1997] show\nthat in the setting of combinatorial optimization,\nno algorithm is able to do better than a random\nstrategy in expectation. This suggests that specialization to a subclass of problems is in fact\ntheonly way that improved performance can be\nachieved in general.\nIn this work we take a different tack and instead\npropose to replace hand-designed update rules\nwith a learned update rule, which we call the optimizerg, speci\ufb01ed by its own set of parameters\n\u001e. This results in updates to the optimizee fof\nthe form\n\u0012t+1=\u0012t+gt(rf(\u0012t);\u001e): (1)\nA high level view of this process is shown in Figure 1. In what follows we will explicitly model"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab527-6db4-11ee-b737-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-4", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "A high level view of this process is shown in Figure 1. In what follows we will explicitly model\nthe update rule gusing a recurrent neural network (RNN) which maintains its own state and hence\ndynamically updates as a function of its iterates.\n1.1 Transfer learning and generalization\nThe goal of this work is to develop a procedure for constructing a learning algorithm which performs\nwell on a particular class of optimization problems. Casting algorithm design as a learning problem\nallows us to specify the class of problems we are interested in through example problem instances.\nThis is in contrast to the ordinary approach of characterizing properties of interesting problems\nanalytically and using these analytical insights to design learning algorithms by hand.\nIt is informative to consider the meaning of generalization in this framework. In ordinary statistical\nlearning we have a particular function of interest, whose behavior is constrained through a data set of\nexample function evaluations. In choosing a model we specify a set of inductive biases about how\nwe think the function of interest should behave at points we have not observed, and generalization\ncorresponds to the capacity to make predictions about the behavior of the target function at novel\npoints. In our setting the examples are themselves problem instances , which means generalization\ncorresponds to the ability to transfer knowledge between different problems. This reuse of problem"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab528-6db4-11ee-880a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-4", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 1174, "chroma:document": "corresponds to the capacity to make predictions about the behavior of the target function at novel\npoints. In our setting the examples are themselves problem instances , which means generalization\ncorresponds to the ability to transfer knowledge between different problems. This reuse of problem\nstructure is commonly known as transfer learning , and is often treated as a subject in its own right."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab529-6db4-11ee-8d38-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-5", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "corresponds to the ability to transfer knowledge between different problems. This reuse of problem\nstructure is commonly known as transfer learning , and is often treated as a subject in its own right.\nHowever, by taking a meta-learning perspective, we can cast the problem of transfer learning as one\nof generalization, which is much better studied in the machine learning community.\nOne of the great success stories of deep-learning is that we can rely on the ability of deep networks to\ngeneralize to new examples by learning interesting sub-structures. In this work we aim to leverage\nthis generalization power, but also to lift it from simple supervised learning to the more general\nsetting of optimization.\n1.2 A brief history and related work\nThe idea of using learning to learn ormeta-learning to acquire knowledge or inductive biases has a\nlong history [Thrun and Pratt, 1998]. More recently, Lake et al. [2016] have argued forcefully for\nits importance as a building block in arti\ufb01cial intelligence. Similarly, Santoro et al. [2016] frame\nmulti-task learning as generalization, however unlike our approach they directly train a base learner\nrather than a training algorithm. In general these ideas involve learning which occurs at two different\ntime scales: rapid learning within tasks and more gradual, meta learning across many different tasks."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab52a-6db4-11ee-abf5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-6", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "rather than a training algorithm. In general these ideas involve learning which occurs at two different\ntime scales: rapid learning within tasks and more gradual, meta learning across many different tasks.\nPerhaps the most general approach to meta-learning is that of Schmidhuber [1992, 1993]\u2014building\non work from [Schmidhuber, 1987]\u2014which considers networks that are able to modify their own\nweights. Such a system is differentiable end-to-end, allowing both the network and the learning\n2\nalgorithm to be trained jointly by gradient descent with few restrictions. However this generality\ncomes at the expense of making the learning rules very dif\ufb01cult to train. Alternatively, the work\nof Schmidhuber et al. [1997] uses the Success Story Algorithm to modify its search strategy rather\nthan gradient descent; a similar approach has been recently taken in Daniel et al. [2016] which uses\nreinforcement learning to train a controller for selecting step-sizes.\nBengio et al. [1990, 1995] propose to learn updates which avoid back-propagation by using simple\nparametric rules. In relation to the focus of this paper the work of Bengio et al. could be characterized\naslearning to learn without gradient descent by gradient descent . The work of Runarsson and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab52b-6db4-11ee-b8c2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-7", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "aslearning to learn without gradient descent by gradient descent . The work of Runarsson and\nJonsson [2000] builds upon this work by replacing the simple rule with a neural network.\nCotter and Conwell [1990], and later Younger et al. [1999], also show \ufb01xed-weight recurrent neural\nnetworks can exhibit dynamic behavior without need to modify their network weights. Similarly this\nhas been shown in a \ufb01ltering context [e.g. Feldkamp and Puskorius, 1998], which is directly related\nto simple multi-timescale optimizers [Sutton, 1992, Schraudolph, 1999].\nFinally, the work of Younger et al. [2001] and Hochreiter et al. [2001] connects these different threads\nof research by allowing for the output of backpropagation from one network to feed into an additional\nlearning network, with both networks trained jointly. Our approach to meta-learning builds on this\nwork by modifying the network architecture of the optimizer in order to scale this approach to larger\nneural-network optimization problems.\n2 Learning to learn with recurrent neural networks\nIn this work we consider directly parameterizing the optimizer. As a result, in a slight abuse of notation\nwe will write the \ufb01nal optimizee parameters \u0012\u0003(f;\u001e)as a function of the optimizer parameters \u001eand"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab52c-6db4-11ee-849c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-8", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "we will write the \ufb01nal optimizee parameters \u0012\u0003(f;\u001e)as a function of the optimizer parameters \u001eand\nthe function in question. We can then ask the question: What does it mean for an optimizer to be\ngood? Given a distribution of functions fwe will write the expected loss as\nL(\u001e) =Efh\nf\u0000\n\u0012\u0003(f;\u001e)\u0001i\n: (2)\nAs noted earlier, we will take the update steps gtto be the output of a recurrent neural network m,\nparameterized by \u001e, whose state we will denote explicitly with ht. Next, while the objective function\nin(2)depends only on the \ufb01nal parameter value, for training the optimizer it will be convenient to\nhave an objective that depends on the entire trajectory of optimization, for some horizon T,\nL(\u001e) =Ef\"TX\nt=1wtf(\u0012t)#\nwhere \u0012t+1=\u0012t+gt;\u0014\ngt\nht+1\u0015\n=m(rt;ht;\u001e):(3)\nHerewt2R\u00150are arbitrary weights associated with each time-step and we will also use the notation\nrt=r\u0012f(\u0012t). This formulation is equivalent to (2)whenwt=1[t=T], but later we will describe"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab52d-6db4-11ee-8cd3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-9", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "rt=r\u0012f(\u0012t). This formulation is equivalent to (2)whenwt=1[t=T], but later we will describe\nwhy using different weights can prove useful.\nWe can minimize the value of L(\u001e)using gradient descent on \u001e. The gradient estimate @L(\u001e)=@\u001e can\nbe computed by sampling a random function fand applying backpropagation to the computational\ngraph in Figure 2. We allow gradients to \ufb02ow along the solid edges in the graph, but gradients\nalong the dashed edges are dropped. Ignoring gradients along the dashed edges amounts to making\nthe assumption that the gradients of the optimizee do not depend on the optimizer parameters, i.e.\n@rt\u000e\n@\u001e= 0. This assumption allows us to avoid computing second derivatives of f.\nExamining the objective in (3)we see that the gradient is non-zero only for terms where wt6= 0. If\nwe usewt=1[t=T]to match the original problem, then gradients of trajectory pre\ufb01xes are zero\nand only the \ufb01nal optimization step provides information for training the optimizer. This renders\nBackpropagation Through Time (BPTT) inef\ufb01cient. We solve this problem by relaxing the objective"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab52e-6db4-11ee-991a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-10", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "Backpropagation Through Time (BPTT) inef\ufb01cient. We solve this problem by relaxing the objective\nsuch thatwt>0at intermediate points along the trajectory. This changes the objective function, but\nallows us to train the optimizer on partial trajectories. For simplicity, in all our experiments we use\nwt= 1for everyt.\n3\nOptimizeeOptimizert-2t-1t\nmmm+++ft-1ftft-2\u2207t-2\u2207t-1\u2207tht-2ht-1htht+1gt-1gt\u03b8t-2\u03b8t-1\u03b8t\u03b8t+1gt-2Figure 2: Computational graph used for computing the gradient of the optimizer.\n2.1 Coordinatewise LSTM optimizer\nOne challenge in applying RNNs in our setting is that we want to be able to optimize at least tens of\nthousands of parameters. Optimizing at this scale with a fully connected RNN is not feasible as it\nwould require a huge hidden state and an enormous number of parameters. To avoid this dif\ufb01culty we\nwill use an optimizer mwhich operates coordinatewise on the parameters of the objective function,\nsimilar to other common update rules like RMSprop and ADAM. This coordinatewise network"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab52f-6db4-11ee-a910-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-11", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "will use an optimizer mwhich operates coordinatewise on the parameters of the objective function,\nsimilar to other common update rules like RMSprop and ADAM. This coordinatewise network\narchitecture allows us to use a very small network that only looks at a single coordinate to de\ufb01ne the\noptimizer and share optimizer parameters across different parameters of the optimizee.\nDifferent behavior on each coordinate is achieved by using separate activations for each objective\nfunction parameter. In addition to allowing us to use a small network for this optimizer, this setup has\nthe nice effect of making the optimizer invariant to the order of parameters in the network, since the\nsame update rule is used independently on each coordinate.\nLSTM1fLSTMn\u22071\u03b81\u03b8n++\n\u2026\u2207n\u2026\u2026\u2026\n\u2026\nFigure 3: One step of an LSTM optimizer. All\nLSTMs have shared parameters, but separate hidden states.We implement the update rule for each coordinate using a two-layer Long Short Term Memory\n(LSTM) network [Hochreiter and Schmidhuber,\n1997], using the now-standard forget gate architecture. The network takes as input the optimizee gradient for a single coordinate as well\nas the previous hidden state and outputs the update for the corresponding optimizee parameter.\nWe will refer to this architecture, illustrated in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab530-6db4-11ee-9e28-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-12", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "as the previous hidden state and outputs the update for the corresponding optimizee parameter.\nWe will refer to this architecture, illustrated in\nFigure 3, as an LSTM optimizer.\nThe use of recurrence allows the LSTM to learn\ndynamic update rules which integrate information from the history of gradients, similar to\nmomentum. This is known to have many desirable properties in convex optimization [see e.g.\nNesterov, 1983] and in fact many recent learning procedures\u2014such as ADAM\u2014use momentum in\ntheir updates.\nPreprocessing and postprocessing Optimizer inputs and outputs can have very different magnitudes depending on the class of function being optimized, but neural networks usually work robustly\nonly for inputs and outputs which are neither very small nor very large. In practice rescaling inputs\nand outputs of an LSTM optimizer using suitable constants (shared across all timesteps and functions\nf) is suf\ufb01cient to avoid this problem. In Appendix A we propose a different method of preprocessing\ninputs to the optimizer inputs which is more robust and gives slightly better performance.\n4\n20 40 60 80 10010-1100101LossQuadratics\nADAM\nRMSprop\nSGD\nNAG\nLSTM\n20 40 60 80 100\nStep100MNIST"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab531-6db4-11ee-b5d3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-13", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "ADAM\nRMSprop\nSGD\nNAG\nLSTM\n20 40 60 80 100\nStep100MNIST\n120 140 160 180 200MNIST, 200 stepsFigure 4: Comparisons between learned and hand-crafted optimizers performance. Learned optimizers are shown with solid lines and hand-crafted optimizers are shown with dashed lines. Units for the\nyaxis in the MNIST plots are logits. Left: Performance of different optimizers on randomly sampled\n10-dimensional quadratic functions. Center: the LSTM optimizer outperforms standard methods\ntraining the base network on MNIST. Right: Learning curves for steps 100-200 by an optimizer\ntrained to optimize for 100 steps (continuation of center plot).\n3 Experiments\nIn all experiments the trained optimizers use two-layer LSTMs with 20 hidden units in each layer.\nEach optimizer is trained by minimizing Equation 3 using truncated BPTT as described in Section 2.\nThe minimization is performed using ADAM with a learning rate chosen by random search.\nWe use early stopping when training the optimizer in order to avoid over\ufb01tting the optimizer. After\neach epoch (some \ufb01xed number of learning steps) we freeze the optimizer parameters and evaluate its\nperformance. We pick the best optimizer (according to the \ufb01nal validation loss) and report its average"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab532-6db4-11ee-8d2e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-14", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "performance. We pick the best optimizer (according to the \ufb01nal validation loss) and report its average\nperformance on a number of freshly sampled test problems.\nWe compare our trained optimizers with standard optimizers used in Deep Learning: SGD, RMSprop,\nADAM, and Nesterov\u2019s accelerated gradient (NAG). For each of these optimizer and each problem\nwe tuned the learning rate, and report results with the rate that gives the best \ufb01nal error for each\nproblem. When an optimizer has more parameters than just a learning rate (e.g. decay coef\ufb01cients for\nADAM) we use the default values from the optim package in Torch7. Initial values of all optimizee\nparameters were sampled from an IID Gaussian distribution.\n3.1 Quadratic functions\nIn this experiment we consider training an optimizer on a simple class of synthetic 10-dimensional\nquadratic functions. In particular we consider minimizing functions of the form\nf(\u0012) =kW\u0012\u0000yk2\n2\nfor different 10x10 matrices Wand 10-dimensional vectors ywhose elements are drawn from an IID\nGaussian distribution. Optimizers were trained by optimizing random functions from this family and\ntested on newly sampled functions from the same distribution. Each function was optimized for 100"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab533-6db4-11ee-a190-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-15", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "Gaussian distribution. Optimizers were trained by optimizing random functions from this family and\ntested on newly sampled functions from the same distribution. Each function was optimized for 100\nsteps and the trained optimizers were unrolled for 20 steps. We have not used any preprocessing, nor\npostprocessing.\nLearning curves for different optimizers, averaged over many functions, are shown in the left plot of\nFigure 4. Each curve corresponds to the average performance of one optimization algorithm on many\ntest functions; the solid curve shows the learned optimizer performance and dashed curves show\nthe performance of the standard baseline optimizers. It is clear the learned optimizers substantially\noutperform the baselines in this setting.\n3.2 Training a small neural network on MNIST\nIn this experiment we test whether trainable optimizers can learn to optimize a small neural network\non MNIST, and also explore how the trained optimizers generalize to functions beyond those they\nwere trained on. To this end, we train the optimizer to optimize a base network and explore a series\nof modi\ufb01cations to the network architecture and training procedure at test time.\n5\n20 40 60 80 100100LossMNIST, 40 units\nADAM\nRMSprop\nSGD\nNAG\nLSTM\n20 40 60 80 100\nStepsMNIST, 2 layers"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab534-6db4-11ee-abdb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-16", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "ADAM\nRMSprop\nSGD\nNAG\nLSTM\n20 40 60 80 100\nStepsMNIST, 2 layers\n20 40 60 80 100MNIST, ReLUFigure 5: Comparisons between learned and hand-crafted optimizers performance. Units for the\nyaxis are logits. Left: Generalization to the different number of hidden units (40 instead of 20).\nCenter: Generalization to the different number of hidden layers (2 instead of 1). This optimization\nproblem is very hard, because the hidden layers are very narrow. Right: Training curves for an MLP\nwith 20 hidden units using ReLU activations. The LSTM optimizer was trained on an MLP with\nsigmoid activations.\n5 20 35 5010-1100Final Loss\nLSTM\n5 20 35 50\nHidden units\nADAM\n5 20 35 50\nNAG\nLayers\n1\n2\nFigure 6: Systematic study of \ufb01nal MNIST performance as the optimizee architecture is varied,\nusing sigmoid non-linearities. The vertical dashed line in the left-most plot denotes the architecture\nat which the LSTM is trained and the horizontal line shows the \ufb01nal performance of the trained\noptimizer in this setting.\nIn this setting the objective function f(\u0012)is the cross entropy of a small MLP with parameters \u0012."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab535-6db4-11ee-8e20-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-17", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "optimizer in this setting.\nIn this setting the objective function f(\u0012)is the cross entropy of a small MLP with parameters \u0012.\nThe values of fas well as the gradients @f(\u0012)=@\u0012are estimated using random minibatches of 128\nexamples. The base network is an MLP with one hidden layer of 20 units using a sigmoid activation\nfunction. The only source of variability between different runs is the initial value \u00120and randomness\nin minibatch selection. Each optimization was run for 100 steps and the trained optimizers were\nunrolled for 20 steps. We used input preprocessing described in Appendix A and rescaled the outputs\nof the LSTM by the factor 0:1.\nLearning curves for the base network using different optimizers are displayed in the center plot of\nFigure 4. In this experiment NAG, ADAM, and RMSprop exhibit roughly equivalent performance the\nLSTM optimizer outperforms them by a signi\ufb01cant margin. The right plot in Figure 4 compares the\nperformance of the LSTM optimizer if it is allowed to run for 200 steps, despite having been trained\nto optimize for 100 steps. In this comparison we re-used the LSTM optimizer from the previous\nexperiment, and here we see that the LSTM optimizer continues to outperform the baseline optimizers\non this task."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab536-6db4-11ee-a096-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-18", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "experiment, and here we see that the LSTM optimizer continues to outperform the baseline optimizers\non this task.\nGeneralization to different architectures Figure 5 shows three examples of applying the LSTM\noptimizer to train networks with different architectures than the base network on which it was trained.\nThe modi\ufb01cations are (from left to right) (1) an MLP with 40 hidden units instead of 20, (2) a\nnetwork with two hidden layers instead of one, and (3) a network using ReLU activations instead of\nsigmoid. In the \ufb01rst two cases the LSTM optimizer generalizes well, and continues to outperform\nthe hand-designed baselines despite operating outside of its training regime. However, changing\nthe activation function to ReLU makes the dynamics of the learning procedure suf\ufb01ciently different\nthat the learned optimizer is no longer able to generalize. Finally, in Figure 6 we show the results\nof systematically varying the tested architecture; for the LSTM results we again used the optimizer\ntrained using 1 layer of 20 units and sigmoid non-linearities. Note that in this setting where the\n6\n200 400 600 800 1000100LossCIFAR-10\n200 400 600 800 1000\nStep100CIFAR-5\n200 400 600 800 100010-1CIFAR-2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab537-6db4-11ee-9dfc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-19", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "200 400 600 800 1000\nStep100CIFAR-5\n200 400 600 800 100010-1CIFAR-2\nADAM\nRMSprop\nSGD\nNAG\nLSTM\nLSTM-subFigure 7: Optimization performance on the CIFAR-10 dataset and subsets. Shown on the left is the\nLSTM optimizer versus various baselines trained on CIFAR-10 and tested on a held-out test set. The\ntwo plots on the right are the performance of these optimizers on subsets of the CIFAR labels. The\nadditional optimizer LSTM-sub has been trained only on the heldout labels and is hence transferring\nto a completely novel dataset.\ntest-set problems are similar enough to those in the training set we see even better generalization than\nthe baseline optimizers.\n3.3 Training a convolutional network on CIFAR-10\nNext we test the performance of the trained neural optimizers on optimizing classi\ufb01cation performance\nfor the CIFAR-10 dataset [Krizhevsky, 2009]. In these experiments we used a model with both\nconvolutional and feed-forward layers. In particular, the model used for these experiments includes\nthree convolutional layers with max pooling followed by a fully-connected layer with 32 hidden units;"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab538-6db4-11ee-b387-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-20", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "three convolutional layers with max pooling followed by a fully-connected layer with 32 hidden units;\nall non-linearities were ReLU activations with batch normalization.\nThe coordinatewise network decomposition introduced in Section 2.1\u2014and used in the previous\nexperiment\u2014utilizes a single LSTM architecture with shared weights, but separate hidden states,\nfor each optimizee parameter. We found that this decomposition was not suf\ufb01cient for the model\narchitecture introduced in this section due to the differences between the fully connected and convolutional layers. Instead we modify the optimizer by introducing two LSTMs: one proposes parameter\nupdates for the fully connected layers and the other updates the convolutional layer parameters. Like\nthe previous LSTM optimizer we still utilize a coordinatewise decomposition with shared weights\nand individual hidden states, however LSTM weights are now shared only between parameters of the\nsame type (i.e. fully-connected vs. convolutional).\nThe performance of this trained optimizer compared against the baseline techniques is shown in\nFigure 7. The left-most plot displays the results of using the optimizer to \ufb01t a classi\ufb01er on a held-out\ntest set. The additional two plots on the right display the performance of the trained optimizer on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab539-6db4-11ee-bf18-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-21", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "test set. The additional two plots on the right display the performance of the trained optimizer on\nmodi\ufb01ed datasets which only contain a subset of the labels, i.e. the CIFAR-2 dataset only contains\ndata corresponding to 2 of the 10 labels. Additionally we include an optimizer LSTM-sub which was\nonly trained on the held-out labels.\nIn all these examples we can see that the LSTM optimizer learns much more quickly than the baseline\noptimizers, with signi\ufb01cant boosts in performance for the CIFAR-5 and especially CIFAR-2 datsets.\nWe also see that the optimizers trained only on a disjoint subset of the data is hardly effected by this\ndifference and transfers well to the additional dataset.\n3.4 Neural Art\nThe recent work on artistic style transfer using convolutional networks, or Neural Art [Gatys et al.,\n2015], gives a natural testbed for our method, since each content and style image pair gives rise to a\ndifferent optimization problem. Each Neural Art problem starts from a content image ,c, and a style\nimage ,s, and is given by\nf(\u0012) =\u000bLcontent (c;\u0012) +\fLstyle(s;\u0012) +\rLreg(\u0012)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab53a-6db4-11ee-ae56-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-22", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "image ,s, and is given by\nf(\u0012) =\u000bLcontent (c;\u0012) +\fLstyle(s;\u0012) +\rLreg(\u0012)\nThe minimizer of fis the styled image . The \ufb01rst two terms try to match the content and style of\nthe styled image to that of their \ufb01rst argument, and the third term is a regularizer that encourages\nsmoothness in the styled image. Details can be found in [Gatys et al., 2015].\n7\n20 40 60 80 100 120\nStepLossNeural art, training resolution\n20 40 60 80 100 120\nStepDouble resolution\nADAM\nRMSprop\nSGD\nNAG\nLSTMFigure 8: Optimization curves for Neural Art. Content images come from the test set, which was not\nused during the LSTM optimizer training. Note: the y-axis is in log scale and we zoom in on the\ninteresting portion of this plot. Left: Applying the training style at the training resolution. Right:\nApplying the test style at double the training resolution.\nFigure 9: Examples of images styled using the LSTM optimizer. Each triple consists of the content\nimage (left), style (right) and image generated by the LSTM optimizer (center). Left: The result of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab53b-6db4-11ee-b256-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-23", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "image (left), style (right) and image generated by the LSTM optimizer (center). Left: The result of\napplying the training style at the training resolution to a test image. Right: The result of applying a\nnew style to a test image at double the resolution on which the optimizer was trained.\nWe train optimizers using only 1 style and 1800 content images taken from ImageNet [Deng et al.,\n2009]. We randomly select 100 content images for testing and 20 content images for validation of\ntrained optimizers. We train the optimizer on 64x64 content images from ImageNet and one \ufb01xed\nstyle image. We then test how well it generalizes to a different style image and higher resolution\n(128x128). Each image was optimized for 128 steps and trained optimizers were unrolled for 32\nsteps. Figure 9 shows the result of styling two different images using the LSTM optimizer. The\nLSTM optimizer uses inputs preprocessing described in Appendix A and no postprocessing. See\nAppendix C for additional images.\nFigure 8 compares the performance of the LSTM optimizer to standard optimization algorithms. The\nLSTM optimizer outperforms all standard optimizers if the resolution and style image are the same\nas the ones on which it was trained. Moreover, it continues to perform very well when both the\nresolution and style are changed at test time."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab53c-6db4-11ee-a6e0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-24", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "as the ones on which it was trained. Moreover, it continues to perform very well when both the\nresolution and style are changed at test time.\nFinally, in Appendix B we qualitatively examine the behavior of the step directions generated by the\nlearned optimizer.\n4 Conclusion\nWe have shown how to cast the design of optimization algorithms as a learning problem, which\nenables us to train optimizers that are specialized to particular classes of functions. Our experiments\nhave con\ufb01rmed that learned neural optimizers compare favorably against state-of-the-art optimization\nmethods used in deep learning. We witnessed a remarkable degree of transfer, with for example the\nLSTM optimizer trained on 12,288 parameter neural art tasks being able to generalize to tasks with\n49,152 parameters, different styles, and different content images all at the same time. We observed\nsimilar impressive results when transferring to different architectures in the MNIST task.\nThe results on the CIFAR image labeling task show that the LSTM optimizers outperform handengineered optimizers when transferring to datasets drawn from the same data distribution.\nReferences\nF. Bach, R. Jenatton, J. Mairal, and G. Obozinski. Optimization with sparsity-inducing penalties. Foundations\nand Trends in Machine Learning , 4(1):1\u2013106, 2012.\n8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab53d-6db4-11ee-a7ba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-25", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "and Trends in Machine Learning , 4(1):1\u2013106, 2012.\n8\nS. Bengio, Y . Bengio, and J. Cloutier. On the search for new learning rules for ANNs. Neural Processing Letters ,\n2(4):26\u201330, 1995.\nY . Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule . Universit\u00e9 de Montr\u00e9al, D\u00e9partement\nd\u2019informatique et de recherche op\u00e9rationnelle, 1990.\nY . Bengio, N. Boulanger-Lewandowski, and R. Pascanu. Advances in optimizing recurrent networks. In\nInternational Conference on Acoustics, Speech and Signal Processing , pages 8624\u20138628. IEEE, 2013.\nF. Bobolas. brain-neurons, 2009. URL https://www.flickr.com/photos/fbobolas/3822222947 . Creative Commons Attribution-ShareAlike 2.0 Generic.\nN. E. Cotter and P. R. Conwell. Fixed-weight networks can learn. In International Joint Conference on Neural\nNetworks , pages 553\u2013559, 1990.\nC. Daniel, J. Taylor, and S. Nowozin. Learning step size controllers for robust neural network training. In"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab53e-6db4-11ee-a153-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-26", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "Networks , pages 553\u2013559, 1990.\nC. Daniel, J. Taylor, and S. Nowozin. Learning step size controllers for robust neural network training. In\nAssociation for the Advancement of Arti\ufb01cial Intelligence , 2016.\nJ. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database.\nInComputer Vision and Pattern Recognition , pages 248\u2013255. IEEE, 2009.\nD. L. Donoho. Compressed sensing. Transactions on Information Theory , 52(4):1289\u20131306, 2006.\nJ. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient methods for online learning and stochastic optimization.\nJournal of Machine Learning Research , 12:2121\u20132159, 2011.\nL. A. Feldkamp and G. V . Puskorius. A signal processing framework based on dynamic neural networks\nwith application to problems in adaptation, \ufb01ltering, and classi\ufb01cation. Proceedings of the IEEE , 86(11):\n2259\u20132277, 1998.\nL. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv Report 1508.06576, 2015."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab53f-6db4-11ee-892a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-27", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv Report 1508.06576, 2015.\nA. Graves, G. Wayne, and I. Danihkela. Neural Turing machines. arXiv Report 1410.5401, 2014.\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):1735\u20131780, 1997.\nS. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In International\nConference on Arti\ufb01cial Neural Networks , pages 87\u201394. Springer, 2001.\nD. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning\nRepresentations , 2015.\nA. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.\nB. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like\npeople. arXiv Report 1604.00289, 2016."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab540-6db4-11ee-b31b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-28", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "people. arXiv Report 1604.00289, 2016.\nT. Maley. neuron, 2011. URL https://www.flickr.com/photos/taylortotz101/6280077898 . Creative\nCommons Attribution 2.0 Generic.\nJ. Martens and R. Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In\nInternational Conference on Machine Learning , pages 2408\u20132417, 2015.\nG. L. Nemhauser and L. A. Wolsey. Integer and combinatorial optimization . John Wiley & Sons, 1988.\nY . Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet\nMathematics Doklady , volume 27, pages 372\u2013376, 1983.\nJ. Nocedal and S. Wright. Numerical optimization . Springer Science & Business Media, 2006.\nM. Riedmiller and H. Braun. A direct adaptive method for faster backpropagation learning: The RPROP\nalgorithm. In International Conference on Neural Networks , pages 586\u2013591, 1993.\nT. P. Runarsson and M. T. Jonsson. Evolution and design of distributed learning rules. In IEEE Symposium on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab541-6db4-11ee-a373-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-29", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "T. P. Runarsson and M. T. Jonsson. Evolution and design of distributed learning rules. In IEEE Symposium on\nCombinations of Evolutionary Computation and Neural Networks , pages 59\u201363. IEEE, 2000.\nA. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented\nneural networks. In International Conference on Machine Learning , 2016.\nJ. Schmidhuber. Evolutionary principles in self-referential learning; On learning how to learn: The meta-meta-...\nhook. PhD thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.\nJ. Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks.\nNeural Computation , 4(1):131\u2013139, 1992.\nJ. Schmidhuber. A neural network that embeds its own meta-levels. In International Conference on Neural\nNetworks , pages 407\u2013412. IEEE, 1993.\nJ. Schmidhuber, J. Zhao, and M. Wiering. Shifting inductive bias with success-story algorithm, adaptive levin\nsearch, and incremental self-improvement. Machine Learning , 28(1):105\u2013130, 1997."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab542-6db4-11ee-863b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-30", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "search, and incremental self-improvement. Machine Learning , 28(1):105\u2013130, 1997.\nN. N. Schraudolph. Local gain adaptation in stochastic gradient descent. In International Conference on\nArti\ufb01cial Neural Networks , volume 2, pages 569\u2013574, 1999.\nR. S. Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In Association for\nthe Advancement of Arti\ufb01cial Intelligence , pages 171\u2013176, 1992.\nS. Thrun and L. Pratt. Learning to learn . Springer Science & Business Media, 1998.\nT. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent\nmagnitude. COURSERA: Neural Networks for Machine Learning , 4:2, 2012.\nP. Tseng. An incremental gradient (-projection) method with momentum term and adaptive stepsize rule. Journal\non Optimization , 8(2):506\u2013531, 1998.\nD. H. Wolpert and W. G. Macready. No free lunch theorems for optimization. Transactions on Evolutionary\nComputation , 1(1):67\u201382, 1997.\n9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab543-6db4-11ee-ac01-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-31", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "Computation , 1(1):67\u201382, 1997.\n9\nA. S. Younger, P. R. Conwell, and N. E. Cotter. Fixed-weight on-line learning. Transactions on Neural Networks ,\n10(2):272\u2013283, 1999.\nA. S. Younger, S. Hochreiter, and P. R. Conwell. Meta-learning with backpropagation. In International Joint\nConference on Neural Networks , 2001.\n10\n0 64 128\u22128\u221224\n0 64 128\u22122616\nLSTM\nADAM\nSGDFigure 10: Updates proposed by different optimizers at different optimization steps for two different\ncoordinates.\nA Gradient preprocessing\nOne potential challenge in training optimizers is that different input coordinates (i.e. the gradients\nw.r.t. different optimizee parameters) can have very different magnitudes. This is indeed the case e.g.\nwhen the optimizee is a neural network and different parameters correspond to weights in different\nlayers. This can make training an optimizer dif\ufb01cult, because neural networks naturally disregard\nsmall variations in input signals and concentrate on bigger input values.\nTo this aim we propose to preprocess the optimizer\u2019s inputs. One solution would be to give the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab544-6db4-11ee-b86b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-32", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "small variations in input signals and concentrate on bigger input values.\nTo this aim we propose to preprocess the optimizer\u2019s inputs. One solution would be to give the\noptimizer (log(jrj);sgn(r))as an input, whereris the gradient in the current timestep. This has a\nproblem that log(jrj)diverges forr! 0. Therefore, we use the following preprocessing formula\nrk!(\u0010\nlog(jrj)\np;sgn(r)\u0011\nifjrj\u0015e\u0000p\n(\u00001;epr) otherwise\nwherep>0is a parameter controlling how small gradients are disregarded (we use p= 10 in all our\nexperiments).\nWe noticed that just rescaling all inputs by an appropriate constant instead also works \ufb01ne, but the\nproposed preprocessing seems to be more robust and gives slightly better results on some problems.\nB Visualizations\nVisualizing optimizers is inherently dif\ufb01cult because their proposed updates are functions of the full\noptimization trajectory. In this section we try to peek into the decisions made by the LSTM optimizer,\ntrained on the neural art task.\nHistories of updates We select a single optimizee parameter (one color channel of one pixel in the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab545-6db4-11ee-b239-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-33", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "trained on the neural art task.\nHistories of updates We select a single optimizee parameter (one color channel of one pixel in the\nstyled image) and trace the updates proposed to this coordinate by the LSTM optimizer over a single\ntrajectory of optimization. We also record the updates that would have been proposed by both SGD\nand ADAM if they followed the same trajectory of iterates. Figure 10 shows the trajectory of updates\nfor two different optimizee parameters. From the plots it is clear that the trained optimizer makes\nbigger updates than SGD and ADAM. It is also visible that it uses some kind of momentum, but its\nupdates are more noisy than those proposed by ADAM which may be interpreted as having a shorter\ntime-scale momentum.\nProposed update as a function of current gradient Another way to visualize the optimizer\nbehavior is to look at the proposed update gtfor a single coordinate as a function of the current\ngradient evaluation rt. We follow the same procedure as in the previous experiment, and visualize\nthe proposed updates for a few selected time steps.\nThese results are shown in Figures 11\u201313. In these plots, the x-axis is the current value of the gradient\nfor the chosen coordinate, and the y-axis shows the update that each optimizer would propose should\nthe corresponding gradient value be observed. The history of gradient observations is the same for all"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab546-6db4-11ee-9d13-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-34", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "for the chosen coordinate, and the y-axis shows the update that each optimizer would propose should\nthe corresponding gradient value be observed. The history of gradient observations is the same for all\nmethods and follows the trajectory of the LSTM optimizer.\n11\nThe shape of this function for the LSTM optimizer is often step-like, which is also the case for\nADAM. Surprisingly the step is sometimes in the opposite direction as for ADAM, i.e. the bigger the\ngradient, the bigger the update.\nC Neural Art\nShown below are additional examples of images styled using the LSTM optimizer. Each triple\nconsists of the content image (left), style (right) and image generated by the LSTM optimizer (center).\nD Information sharing between coordinates\nIn previous sections we considered a coordinatewise architecture, which corresponds by analogy\nto a learned version of RMSprop or ADAM. Although diagonal methods are quite effective in\npractice, we can also consider learning more sophisticated optimizers that take the correlations\nbetween coordinates into effect. To this end, we introduce a mechanism allowing different LSTMs to\ncommunicate with each other.\nGlobal averaging cells\nThe simplest solution is to designate a subset of the cells in each LSTM layer for communication.\nThese cells operate like normal LSTM cells, but their outgoing activations are averaged at each step"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab547-6db4-11ee-8b7c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-35", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "These cells operate like normal LSTM cells, but their outgoing activations are averaged at each step\nacross all coordinates. These global averaging cells (GACs) are suf\ufb01cient to allow the networks to\nimplement L2 gradient clipping [Bengio et al., 2013] assuming that each LSTM can compute the\nsquare of the gradient. This architecture is denoted as an LSTM+GAC optimizer.\nNTM-BFGS optimizer\nWe also consider augmenting the LSTM+GAC architecture with an external memory that is shared\nbetween coordinates. Such a memory, if appropriately designed could allow the optimizer to learn\nalgorithms similar to (low-memory) approximations to Newton\u2019s method, e.g. (L-)BFGS [see Nocedal\nand Wright, 2006]. The reason for this interpretation is that such methods can be seen as a set of\nindependent processes working coordinatewise, but communicating through the inverse Hessian\napproximation stored in the memory. We designed a memory architecture that, in theory, allows the\n12\nStep 1\u221210010\nStep 2 Step 3 Step 4\nStep 5\u221210010\nStep 6 Step 7 Step 8\nStep 9\u221210010\nStep 10 Step 11 Step 12\nStep 13\u221210010\nStep 14 Step 15 Step 16\nStep 17\u221210010"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab548-6db4-11ee-8105-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-36", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "Step 6 Step 7 Step 8\nStep 9\u221210010\nStep 10 Step 11 Step 12\nStep 13\u221210010\nStep 14 Step 15 Step 16\nStep 17\u221210010\nStep 18 Step 19 Step 20\nStep 21\u221210010\nStep 22 Step 23 Step 24\nStep 25\u221210010\nStep 26 Step 27 Step 28\n\u2212400 0 400\nStep 29\u221210010\n\u2212400 0 400\nStep 30\u2212400 0 400\nStep 31\u2212400 0 400\nStep 32Figure 11: The proposed update direction for a single coordinate over 32 steps.\n13\nStep 1\u221210010\nStep 2 Step 3 Step 4\nStep 5\u221210010\nStep 6 Step 7 Step 8\nStep 9\u221210010\nStep 10 Step 11 Step 12\nStep 13\u221210010\nStep 14 Step 15 Step 16\nStep 17\u221210010\nStep 18 Step 19 Step 20\nStep 21\u221210010\nStep 22 Step 23 Step 24\nStep 25\u221210010\nStep 26 Step 27 Step 28\n\u2212400 0 400\nStep 29\u221210010\n\u2212400 0 400\nStep 30\u2212400 0 400\nStep 31\u2212400 0 400\nStep 32Figure 12: The proposed update direction for a single coordinate over 32 steps.\n14\nStep 1\u221210010\nStep 2 Step 3 Step 4\nStep 5\u221210010\nStep 6 Step 7 Step 8\nStep 9\u221210010"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab549-6db4-11ee-a410-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-37", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "14\nStep 1\u221210010\nStep 2 Step 3 Step 4\nStep 5\u221210010\nStep 6 Step 7 Step 8\nStep 9\u221210010\nStep 10 Step 11 Step 12\nStep 13\u221210010\nStep 14 Step 15 Step 16\nStep 17\u221210010\nStep 18 Step 19 Step 20\nStep 21\u221210010\nStep 22 Step 23 Step 24\nStep 25\u221210010\nStep 26 Step 27 Step 28\n\u2212400 0 400\nStep 29\u221210010\n\u2212400 0 400\nStep 30\u2212400 0 400\nStep 31\u2212400 0 400\nStep 32Figure 13: The proposed update direction for a single coordinate over 32 steps.\n15\nLSTMLSTMLSTMLSTMLSTMLSTMLSTMr1MLSTMr2r3\u2026\u2026\u2026\u2026rni1i2i3\u2026\u2026\u2026\u2026inx=\nLSTMLSTMLSTMLSTMLSTMLSTMLSTMLSTMa1a2a3\u2026\u2026\u2026\u2026anb1b2b3\u2026\u2026\u2026\u2026bnOuterProductMt\u0394MtMt+1+=Figure 14: Left: NTM-BFGS read operation. Right: NTM-BFGS write operation.\nnetwork to simulate (L-)BFGS, motivated by the approximate Newton method BFGS, named for\nBroyden, Fletcher, Goldfarb, and Shanno. We call this architecture an NTM-BFGS optimizer , because"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab54a-6db4-11ee-bca3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-38", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "Broyden, Fletcher, Goldfarb, and Shanno. We call this architecture an NTM-BFGS optimizer , because\nits use of external memory is similar to the Neural Turing Machine [Graves et al., 2014]. The pivotal\ndifferences between our construction and the NTM are (1) our memory allows only low-rank updates;\n(2) the controller (including read/write heads) operates coordinatewise.\nIn BFGS an explicit estimate of the full (inverse) Hessian is built up from the sequence of observed\ngradients. We can write a skeletonized version of the BFGS algorithm, using Mtto represent the\ninverse Hessian approximation at iteration t, as follows\ngt= read(Mt;\u0012t)\n\u0012t+1=\u0012t+gt\nMt+1= write(Mt;\u0012t;gt):\nHere we have packed up all of the details of the BFGS algorithm into the suggestively named read\nandwrite operations, which operate on the inverse Hessian approximation Mt. In BFGS these\noperations have speci\ufb01c forms, for example read(Mt;\u0012t) =\u0000Mtrf(\u0012t)is a speci\ufb01c matrix-vector\nmultiplication and the BFGS write operation corresponds to a particular low-rank update of Mt."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab54b-6db4-11ee-8317-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-39", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "multiplication and the BFGS write operation corresponds to a particular low-rank update of Mt.\nIn this work we preserve the structure of the BFGS updates, but discard their particular form. More\nspeci\ufb01cally the read operation remains a matrix-vector multiplication but the form of the vector\nused is learned. Similarly, the write operation remains a low-rank update, but the vectors involved\nare also learned. Conveniently, this structure of interaction with a large dynamically updated state\ncorresponds in a fairly direct way to the architecture of a Neural Turing Machine (NTM), where Mt\ncorresponds to the NTM memory [Graves et al., 2014].\nOur NTM-BFGS optimizer uses an LSTM+GAC as a controller; however, instead of producing the\nupdate directly we attach one or more read and write heads to the controller. Each read head produces\na read vector rtwhich is combined with the memory to produce a read result itwhich is fed back\ninto the controller at the following time step. Each write head produces two outputs, a left write\nvectoratand a right write vector bt. The two write vectors are used to update the memory state by\naccumulating their outer product. The read and write operation for a single head is diagrammed in\nFigure 14 and the way read and write heads are attached to the controller is depicted in Figure 15."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab54c-6db4-11ee-832c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-40", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "Figure 14 and the way read and write heads are attached to the controller is depicted in Figure 15.\nIn can be shown that NTM-BFGS with one read head and 3 write heads can simulate inverse Hessian\nBFGS assuming that the controller can compute arbitrary (coordinatewise) functions and have access\nto 2 GACs.\nNTM-L-BFGS optimizer\nIn cases where memory is constrained we can follow the example of L-BFGS and maintain a low\nrank approximation of the full memory ( vis. inverse Hessian). The simplest way to do this is to store a\nsliding history of the left and right write vectors, allowing us to form the matrix vector multiplication\nrequired by the read operation ef\ufb01ciently.\n16\nController         ReadWriteMtrtht+1Mt+1rt+1itatbt\u2207thtLSTM1LSTM2\u2026gt\nLSTMk\u2207kikakbkrkhkhkgkFigure 15: Left: Interaction between the controller and the external memory in NTM-BFGS. The\ncontroller is composed of replicated coordinatewise LSTMs (possibly with GACs), but the read and\nwrite operations are global across all coordinates. Right: A single LSTM for the kth coordinate in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab54d-6db4-11ee-aa83-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1606.04474-41", "title": "Learning to learn by gradient descent by gradient descent", "source": "http://arxiv.org/pdf/1606.04474", "start_index": 0, "chroma:document": "write operations are global across all coordinates. Right: A single LSTM for the kth coordinate in\nthe NTM-BFGS controller. Note that here we have dropped the time index tto simplify notation.\n17"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab54e-6db4-11ee-9687-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-0", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "Layer Normalization\nJimmy Lei Ba\nUniversity of Toronto\njimmy@psi.toronto.eduJamie Ryan Kiros\nUniversity of Toronto\nrkiros@cs.toronto.eduGeoffrey E. Hinton\nUniversity of Toronto\nand Google Inc.\nhinton@cs.toronto.edu\nAbstract\nTraining state-of-the-art, deep neural networks is computationally expensive. One\nway to reduce the training time is to normalize the activities of the neurons. A\nrecently introduced technique called batch normalization uses the distribution of\nthe summed input to a neuron over a mini-batch of training cases to compute a\nmean and variance which are then used to normalize the summed input to that\nneuron on each training case. This signi\ufb01cantly reduces the training time in feedforward neural networks. However, the effect of batch normalization is dependent\non the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by\ncomputing the mean and variance used for normalization from all of the summed\ninputs to the neurons in a layer on a single training case. Like batch normalization,\nwe also give each neuron its own adaptive bias and gain which are applied after\nthe normalization but before the non-linearity. Unlike batch normalization, layer"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab54f-6db4-11ee-b05a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-1", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "we also give each neuron its own adaptive bias and gain which are applied after\nthe normalization but before the non-linearity. Unlike batch normalization, layer\nnormalization performs exactly the same computation at training and test times.\nIt is also straightforward to apply to recurrent neural networks by computing the\nnormalization statistics separately at each time step. Layer normalization is very\neffective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time\ncompared with previously published techniques.\n1 Introduction\nDeep neural networks trained with some version of Stochastic Gradient Descent have been shown\nto substantially outperform previous approaches on various supervised learning tasks in computer\nvision [Krizhevsky et al., 2012] and speech processing [Hinton et al., 2012]. But state-of-the-art\ndeep neural networks often require many days of training. It is possible to speed-up the learning\nby computing gradients for different subsets of the training cases on different machines or splitting\nthe neural network itself over many machines [Dean et al., 2012], but this can require a lot of communication and complex software. It also tends to lead to rapidly diminishing returns as the degree\nof parallelization increases. An orthogonal approach is to modify the computations performed in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab550-6db4-11ee-bbd9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-1", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 1122, "chroma:document": "the neural network itself over many machines [Dean et al., 2012], but this can require a lot of communication and complex software. It also tends to lead to rapidly diminishing returns as the degree\nof parallelization increases. An orthogonal approach is to modify the computations performed in\nthe forward pass of the neural net to make learning easier. Recently, batch normalization [Ioffe and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab551-6db4-11ee-a799-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-2", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "the forward pass of the neural net to make learning easier. Recently, batch normalization [Ioffe and\nSzegedy, 2015] has been proposed to reduce training time by including additional normalization\nstages in deep neural networks. The normalization standardizes each summed input using its mean\nand its standard deviation across the training data. Feedforward neural networks trained using batch\nnormalization converge faster even with simple SGD. In addition to training time improvement, the\nstochasticity from the batch statistics serves as a regularizer during training.\nDespite its simplicity, batch normalization requires running averages of the summed input statistics. In feed-forward networks with \ufb01xed depth, it is straightforward to store the statistics separately\nfor each hidden layer. However, the summed inputs to the recurrent neurons in a recurrent neural network (RNN) often vary with the length of the sequence so applying batch normalization to\nRNNs appears to require different statistics for different time-steps. Furthermore, batch normaliza-arXiv:1607.06450v1  [stat.ML]  21 Jul 2016\ntion cannot be applied to online learning tasks or to extremely large distributed models where the\nminibatches have to be small.\nThis paper introduces layer normalization, a simple normalization method to improve the training\nspeed for various neural network models. Unlike batch normalization, the proposed method directly"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab552-6db4-11ee-8722-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-3", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "This paper introduces layer normalization, a simple normalization method to improve the training\nspeed for various neural network models. Unlike batch normalization, the proposed method directly\nestimates the normalization statistics from the summed inputs to the neurons within a hidden layer\nso the normalization does not introduce any new dependencies between training cases. We show that\nlayer normalization works well for RNNs and improves both the training time and the generalization\nperformance of several existing RNN models.\n2 Background\nA feed-forward neural network is a non-linear mapping from a input pattern xto an output vector\ny. Consider the lthhidden layer in a deep feed-forward, neural network, and let albe the vector\nrepresentation of the summed inputs to the neurons in that layer. The summed inputs are computed\nthrough a linear projection with the weight matrix Wland the bottom-up inputs hlgiven as follows:\nal\ni=wl\ni>hlhl+1\ni=f(al\ni+bl\ni) (1)\nwheref(\u0001)is an element-wise non-linear function and wl\niis the incoming weights to the ithhidden\nunits andbl\niis the scalar bias parameter. The parameters in the neural network are learnt using\ngradient-based optimization algorithms with the gradients being computed by back-propagation."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab553-6db4-11ee-91a5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-4", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "units andbl\niis the scalar bias parameter. The parameters in the neural network are learnt using\ngradient-based optimization algorithms with the gradients being computed by back-propagation.\nOne of the challenges of deep learning is that the gradients with respect to the weights in one layer\nare highly dependent on the outputs of the neurons in the previous layer especially if these outputs\nchange in a highly correlated way. Batch normalization [Ioffe and Szegedy, 2015] was proposed\nto reduce such undesirable \u201ccovariate shift\u201d. The method normalizes the summed inputs to each\nhidden unit over the training cases. Speci\ufb01cally, for the ithsummed input in the lthlayer, the batch\nnormalization method rescales the summed inputs according to their variances under the distribution\nof the data\n\u0016al\ni=gl\ni\n\u001bl\ni\u0000\nal\ni\u0000\u0016l\ni\u0001\n\u0016l\ni=E\nx\u0018P(x)\u0002\nal\ni\u0003\n\u001bl\ni=r\nE\nx\u0018P(x)h\u0000\nal\ni\u0000\u0016l\ni\u00012i\n(2)\nwhere \u0016al"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab554-6db4-11ee-b81d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-5", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "\u001bl\ni=r\nE\nx\u0018P(x)h\u0000\nal\ni\u0000\u0016l\ni\u00012i\n(2)\nwhere \u0016al\niis normalized summed inputs to the ithhidden unit in the lthlayer andgiis a gain parameter scaling the normalized activation before the non-linear activation function. Note the expectation\nis under the whole training data distribution. It is typically impractical to compute the expectations\nin Eq. (2) exactly, since it would require forward passes through the whole training dataset with the\ncurrent set of weights. Instead, \u0016and\u001bare estimated using the empirical samples from the current\nmini-batch. This puts constraints on the size of a mini-batch and it is hard to apply to recurrent\nneural networks.\n3 Layer normalization\nWe now consider the layer normalization method which is designed to overcome the drawbacks of\nbatch normalization.\nNotice that changes in the output of one layer will tend to cause highly correlated changes in the\nsummed inputs to the next layer, especially with ReLU units whose outputs can change by a lot.\nThis suggests the \u201ccovariate shift\u201d problem can be reduced by \ufb01xing the mean and the variance of\nthe summed inputs within each layer. We, thus, compute the layer normalization statistics over all\nthe hidden units in the same layer as follows:\n\u0016l=1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab555-6db4-11ee-9f50-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-6", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "the summed inputs within each layer. We, thus, compute the layer normalization statistics over all\nthe hidden units in the same layer as follows:\n\u0016l=1\nHHX\ni=1al\ni\u001bl=vuut1\nHHX\ni=1\u0000\nal\ni\u0000\u0016l\u00012(3)\nwhereHdenotes the number of hidden units in a layer. The difference between Eq. (2) and Eq. (3)\nis that under layer normalization, all the hidden units in a layer share the same normalization terms\n\u0016and\u001b, but different training cases have different normalization terms. Unlike batch normalization,\nlayer normaliztion does not impose any constraint on the size of a mini-batch and it can be used in\nthe pure online regime with batch size 1.\n2\n3.1 Layer normalized recurrent neural networks\nThe recent sequence to sequence models [Sutskever et al., 2014] utilize compact recurrent neural\nnetworks to solve sequential prediction problems in natural language processing. It is common\namong the NLP tasks to have different sentence lengths for different training cases. This is easy to\ndeal with in an RNN because the same weights are used at every time-step. But when we apply batch\nnormalization to an RNN in the obvious way, we need to to compute and store separate statistics for"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab556-6db4-11ee-b12b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-7", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "normalization to an RNN in the obvious way, we need to to compute and store separate statistics for\neach time step in a sequence. This is problematic if a test sequence is longer than any of the training\nsequences. Layer normalization does not have such problem because its normalization terms depend\nonly on the summed inputs to a layer at the current time-step. It also has only one set of gain and\nbias parameters shared over all time-steps.\nIn a standard RNN, the summed inputs in the recurrent layer are computed from the current input\nxtand previous vector of hidden states ht\u00001which are computed as at=Whhht\u00001+Wxhxt. The\nlayer normalized recurrent layer re-centers and re-scales its activations using the extra normalization\nterms similar to Eq. (3):\nht=fhg\n\u001bt\f\u0000\nat\u0000\u0016t\u0001\n+bi\n\u0016t=1\nHHX\ni=1at\ni\u001bt=vuut1\nHHX\ni=1(at\ni\u0000\u0016t)2(4)\nwhereWhhis the recurrent hidden to hidden weights and Wxhare the bottom up input to hidden\nweights.\fis the element-wise multiplication between two vectors. bandgare de\ufb01ned as the bias\nand gain parameters of the same dimension as ht."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab557-6db4-11ee-a294-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-8", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "weights.\fis the element-wise multiplication between two vectors. bandgare de\ufb01ned as the bias\nand gain parameters of the same dimension as ht.\nIn a standard RNN, there is a tendency for the average magnitude of the summed inputs to the recurrent units to either grow or shrink at every time-step, leading to exploding or vanishing gradients. In\na layer normalized RNN, the normalization terms make it invariant to re-scaling all of the summed\ninputs to a layer, which results in much more stable hidden-to-hidden dynamics.\n4 Related work\nBatch normalization has been previously extended to recurrent neural networks [Laurent et al., 2015,\nAmodei et al., 2015, Cooijmans et al., 2016]. The previous work [Cooijmans et al., 2016] suggests\nthe best performance of recurrent batch normalization is obtained by keeping independent normalization statistics for each time-step. The authors show that initializing the gain parameter in the\nrecurrent batch normalization layer to 0.1 makes signi\ufb01cant difference in the \ufb01nal performance of\nthe model. Our work is also related to weight normalization [Salimans and Kingma, 2016]. In\nweight normalization, instead of the variance, the L2 norm of the incoming weights is used to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab558-6db4-11ee-9ef0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-9", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "weight normalization, instead of the variance, the L2 norm of the incoming weights is used to\nnormalize the summed inputs to a neuron. Applying either weight normalization or batch normalization using expected statistics is equivalent to have a different parameterization of the original\nfeed-forward neural network. Re-parameterization in the ReLU network was studied in the Pathnormalized SGD [Neyshabur et al., 2015]. Our proposed layer normalization method, however, is\nnot a re-parameterization of the original neural network. The layer normalized model, thus, has\ndifferent invariance properties than the other methods, that we will study in the following section.\n5 Analysis\nIn this section, we investigate the invariance properties of different normalization schemes.\n5.1 Invariance under weights and data transformations\nThe proposed layer normalization is related to batch normalization and weight normalization. Although, their normalization scalars are computed differently, these methods can be summarized as\nnormalizing the summed inputs aito a neuron through the two scalars \u0016and\u001b. They also learn an\nadaptive bias band gaingfor each neuron after the normalization.\nhi=f(gi\n\u001bi(ai\u0000\u0016i) +bi) (5)\nNote that for layer normalization and batch normalization, \u0016and\u001bis computed according to Eq. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab559-6db4-11ee-abb5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-10", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "\u001bi(ai\u0000\u0016i) +bi) (5)\nNote that for layer normalization and batch normalization, \u0016and\u001bis computed according to Eq. 2\nand 3. In weight normalization, \u0016is 0, and\u001b=kwk2.\n3\nWeight matrix Weight matrix Weight vector Dataset Dataset Single training case\nre-scaling re-centering re-scaling re-scaling re-centering re-scaling\nBatch norm Invariant No Invariant Invariant Invariant No\nWeight norm Invariant No Invariant No No No\nLayer norm Invariant Invariant No Invariant No Invariant\nTable 1: Invariance properties under the normalization methods.\nTable 1 highlights the following invariance results for three normalization methods.\nWeight re-scaling and re-centering: First, observe that under batch normalization and weight\nnormalization, any re-scaling to the incoming weights wiof a single neuron has no effect on the\nnormalized summed inputs to a neuron. To be precise, under batch and weight normalization, if\nthe weight vector is scaled by \u000e, the two scalar \u0016and\u001bwill also be scaled by \u000e. The normalized\nsummed inputs stays the same before and after scaling. So the batch and weight normalization are"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab55a-6db4-11ee-a541-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-11", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "summed inputs stays the same before and after scaling. So the batch and weight normalization are\ninvariant to the re-scaling of the weights. Layer normalization, on the other hand, is not invariant\nto the individual scaling of the single weight vectors. Instead, layer normalization is invariant to\nscaling of the entire weight matrix and invariant to a shift to all of the incoming weights in the\nweight matrix. Let there be two sets of model parameters \u0012,\u00120whose weight matrices WandW0\ndiffer by a scaling factor \u000eand all of the incoming weights in W0are also shifted by a constant\nvector \r, that isW0=\u000eW+1\r>. Under layer normalization, the two models effectively compute\nthe same output:\nh0=f(g\n\u001b0(W0x\u0000\u00160) +b) =f(g\n\u001b0\u0000\n(\u000eW+1\r>)x\u0000\u00160\u0001\n+b)\n=f(g\n\u001b(Wx\u0000\u0016) +b) =h: (6)\nNotice that if normalization is only applied to the input before the weights, the model will not be\ninvariant to re-scaling and re-centering of the weights.\nData re-scaling and re-centering: We can show that all the normalization methods are invariant"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab55b-6db4-11ee-8916-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-12", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "Data re-scaling and re-centering: We can show that all the normalization methods are invariant\nto re-scaling the dataset by verifying that the summed inputs of neurons stays constant under the\nchanges. Furthermore, layer normalization is invariant to re-scaling of individual training cases,\nbecause the normalization scalars \u0016and\u001bin Eq. (3) only depend on the current input data. Let x0\nbe a new data point obtained by re-scaling xby\u000e. Then we have,\nh0\ni=f(gi\n\u001b0\u0000\nw>\nix0\u0000\u00160\u0001\n+bi) =f(gi\n\u000e\u001b\u0000\n\u000ew>\nix\u0000\u000e\u0016\u0001\n+bi) =hi: (7)\nIt is easy to see re-scaling individual data points does not change the model\u2019s prediction under layer\nnormalization. Similar to the re-centering of the weight matrix in layer normalization, we can also\nshow that batch normalization is invariant to re-centering of the dataset.\n5.2 Geometry of parameter space during learning\nWe have investigated the invariance of the model\u2019s prediction under re-centering and re-scaling of\nthe parameters. Learning, however, can behave very differently under different parameterizations,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab55c-6db4-11ee-9f63-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-13", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "the parameters. Learning, however, can behave very differently under different parameterizations,\neven though the models express the same underlying function. In this section, we analyze learning\nbehavior through the geometry and the manifold of the parameter space. We show that the normalization scalar \u001bcan implicitly reduce learning rate and makes learning more stable.\n5.2.1 Riemannian metric\nThe learnable parameters in a statistical model form a smooth manifold that consists of all possible\ninput-output relations of the model. For models whose output is a probability distribution, a natural\nway to measure the separation of two points on this manifold is the Kullback-Leibler divergence\nbetween their model output distributions. Under the KL divergence metric, the parameter space is a\nRiemannian manifold.\nThe curvature of a Riemannian manifold is entirely captured by its Riemannian metric, whose\nquadratic form is denoted as ds2. That is the in\ufb01nitesimal distance in the tangent space at a point\nin the parameter space. Intuitively, it measures the changes in the model output from the parameter\nspace along a tangent direction. The Riemannian metric under KL was previously studied [Amari,\n1998] and was shown to be well approximated under second order Taylor expansion using the Fisher\n4\ninformation matrix:\nds2= D KL\u0002"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab55d-6db4-11ee-a236-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-14", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "1998] and was shown to be well approximated under second order Taylor expansion using the Fisher\n4\ninformation matrix:\nds2= D KL\u0002\nP(yjx;\u0012)kP(yjx;\u0012+\u000e)\u0003\n\u00191\n2\u000e>F(\u0012)\u000e; (8)\nF(\u0012) = E\nx\u0018P(x);y\u0018P(yjx)\"\n@logP(yjx;\u0012)\n@\u0012@logP(yjx;\u0012)\n@\u0012>#\n; (9)\nwhere,\u000eis a small change to the parameters. The Riemannian metric above presents a geometric\nview of parameter spaces. The following analysis of the Riemannian metric provides some insight\ninto how normalization methods could help in training neural networks.\n5.2.2 The geometry of normalized generalized linear models\nWe focus our geometric analysis on the generalized linear model. The results from the following\nanalysis can be easily applied to understand deep neural networks with block-diagonal approximation to the Fisher information matrix, where each block corresponds to the parameters for a single\nneuron.\nA generalized linear model (GLM) can be regarded as parameterizing an output distribution from\nthe exponential family using a weight vector wand bias scalar b. To be consistent with the previous"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab55e-6db4-11ee-94f0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-15", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "A generalized linear model (GLM) can be regarded as parameterizing an output distribution from\nthe exponential family using a weight vector wand bias scalar b. To be consistent with the previous\nsections, the log likelihood of the GLM can be written using the summed inputs aas the following:\nlogP(yjx;w;b) =(a+b)y\u0000\u0011(a+b)\n\u001e+c(y;\u001e); (10)\nE[yjx] =f(a+b) =f(w>x+b);Var[yjx] =\u001ef0(a+b); (11)\nwhere,f(\u0001)is the transfer function that is the analog of the non-linearity in neural networks, f0(\u0001)\nis the derivative of the transfer function, \u0011(\u0001)is a real valued function and c(\u0001)is the log partition function. \u001eis a constant that scales the output variance. Assume a H-dimensional output\nvector y= [y1;y2;\u0001\u0001\u0001;yH]is modeled using Hindependent GLMs and logP(yjx;W;b) =PH\ni=1logP(yijx;wi;bi). LetWbe the weight matrix whose rows are the weight vectors of the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab55f-6db4-11ee-9b82-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-16", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "i=1logP(yijx;wi;bi). LetWbe the weight matrix whose rows are the weight vectors of the\nindividual GLMs, bdenote the bias vector of length Handvec(\u0001)denote the Kronecker vector operator. The Fisher information matrix for the multi-dimensional GLM with respect to its parameters\n\u0012= [w>\n1;b1;\u0001\u0001\u0001;w>\nH;bH]>= vec([W;b]>)is simply the expected Kronecker product of the data\nfeatures and the output covariance matrix:\nF(\u0012) = E\nx\u0018P(x)\u0014Cov[yjx]\n\u001e2\n\u0014\nxx>x\nx>1\u0015\u0015\n: (12)\nWe obtain normalized GLMs by applying the normalization methods to the summed inputs ain\nthe original model through \u0016and\u001b. Without loss of generality, we denote \u0016Fas the Fisher information matrix under the normalized multi-dimensional GLM with the additional gain parameters\n\u0012= vec([W;b;g]>):\n\u0016F(\u0012) =2\n664\u0016F11\u0001\u0001\u0001 \u0016F1H\n.........\n\u0016FH1\u0001\u0001\u0001 \u0016FHH3\n775;\u0016Fij=E\nx\u0018P(x)2\n664Cov[yi; yjjx]"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab560-6db4-11ee-910f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-17", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "\u0016FH1\u0001\u0001\u0001 \u0016FHH3\n775;\u0016Fij=E\nx\u0018P(x)2\n664Cov[yi; yjjx]\n\u001e22\n664gigj\n\u001bi\u001bj\u001fi\u001f>\nj\u001figi\n\u001bi\u001figi(aj\u0000\u0016j)\n\u001bi\u001bj\n\u001f>\njgj\n\u001bj1aj\u0000\u0016j\n\u001bj\n\u001f>\njgj(ai\u0000\u0016i)\n\u001bi\u001bjai\u0000\u0016i\n\u001bi(ai\u0000\u0016i)(aj\u0000\u0016j)\n\u001bi\u001bj3\n7753\n775\n(13)\n\u001fi=x\u0000@\u0016i\n@wi\u0000ai\u0000\u0016i\n\u001bi@\u001bi\n@wi: (14)\nImplicit learning rate reduction through the growth of the weight vector: Notice that, comparing to standard GLM, the block \u0016Fijalong the weight vector widirection is scaled by the gain\nparameters and the normalization scalar \u001bi. If the norm of the weight vector wigrows twice as large,\neven though the model\u2019s output remains the same, the Fisher information matrix will be different.\nThe curvature along the widirection will change by a factor of1\n2because the \u001biwill also be twice"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab561-6db4-11ee-b80c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-18", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "The curvature along the widirection will change by a factor of1\n2because the \u001biwill also be twice\nas large. As a result, for the same parameter update in the normalized model, the norm of the weight\nvector effectively controls the learning rate for the weight vector. During learning, it is harder to\nchange the orientation of the weight vector with large norm. The normalization methods, therefore,\n5\n050100150200250300\niteration x 30034353637383940414243mean Recall@1Image Retrieval (Validation)\nOrder-Embedding + LN\nOrder-Embedding(a) Recall@1\n050100150200250300\niteration x 3007172737475767778mean Recall@5Image Retrieval (Validation)\nOrder-Embedding + LN\nOrder-Embedding (b) Recall@5\n050100150200250300\niteration x 30084858687888990mean Recall@10Image Retrieval (Validation)\nOrder-Embedding + LN\nOrder-Embedding (c) Recall@10\nFigure 1: Recall@K curves using order-embeddings with and without layer normalization.\nMSCOCO\nCaption Retrieval Image Retrieval"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab562-6db4-11ee-87ba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-19", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "Figure 1: Recall@K curves using order-embeddings with and without layer normalization.\nMSCOCO\nCaption Retrieval Image Retrieval\nModel R@1 R@5 R@10 Mean rR@1 R@5 R@10 Mean r\nSym [Vendrov et al., 2016] 45.4 88.7 5.8 36.3 85.8 9.0\nOE [Vendrov et al., 2016] 46.7 88.9 5.7 37.9 85.9 8.1\nOE (ours) 46.6 79.3 89.1 5.2 37.8 73.6 85.7 7.9\nOE + LN 48.5 80.6 89.8 5.1 38.9 74.3 86.3 7.6\nTable 2: Average results across 5 test splits for caption and image retrieval. R@K is Recall@K\n(high is good). Mean ris the mean rank (low is good). Sym corresponds to the symmetric baseline\nwhile OE indicates order-embeddings.\nhave an implicit \u201cearly stopping\u201d effect on the weight vectors and help to stabilize learning towards\nconvergence.\nLearning the magnitude of incoming weights: In normalized models, the magnitude of the incoming weights is explicitly parameterized by the gain parameters. We compare how the model output"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab563-6db4-11ee-bc09-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-20", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "convergence.\nLearning the magnitude of incoming weights: In normalized models, the magnitude of the incoming weights is explicitly parameterized by the gain parameters. We compare how the model output\nchanges between updating the gain parameters in the normalized GLM and updating the magnitude\nof the equivalent weights under original parameterization during learning. The direction along the\ngain parameters in \u0016Fcaptures the geometry for the magnitude of the incoming weights. We show\nthat Riemannian metric along the magnitude of the incoming weights for the standard GLM is scaled\nby the norm of its input, whereas learning the gain parameters for the batch normalized and layer\nnormalized models depends only on the magnitude of the prediction error. Learning the magnitude\nof incoming weights in the normalized model is therefore, more robust to the scaling of the input\nand its parameters than in the standard model. See Appendix for detailed derivations.\n6 Experimental results\nWe perform experiments with layer normalization on 6 tasks, with a focus on recurrent neural networks: image-sentence ranking, question-answering, contextual language modelling, generative\nmodelling, handwriting sequence generation and MNIST classi\ufb01cation. Unless otherwise noted,\nthe default initialization of layer normalization is to set the adaptive gains to 1and the biases to 0in\nthe experiments.\n6.1 Order embeddings of images and language"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab564-6db4-11ee-9ccb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-21", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "the default initialization of layer normalization is to set the adaptive gains to 1and the biases to 0in\nthe experiments.\n6.1 Order embeddings of images and language\nIn this experiment, we apply layer normalization to the recently proposed order-embeddings model\nof Vendrov et al. [2016] for learning a joint embedding space of images and sentences. We follow\nthe same experimental protocol as Vendrov et al. [2016] and modify their publicly available code\nto incorporate layer normalization1which utilizes Theano [Team et al., 2016]. Images and sentences from the Microsoft COCO dataset [Lin et al., 2014] are embedded into a common vector\nspace, where a GRU [Cho et al., 2014] is used to encode sentences and the outputs of a pre-trained\nVGG ConvNet [Simonyan and Zisserman, 2015] (10-crop) are used to encode images. The orderembedding model represents images and sentences as a 2-level partial ordering and replaces the\ncosine similarity scoring function used in Kiros et al. [2014] with an asymmetric one.\n1https://github.com/ivendrov/order-embedding\n6\n0 100 200 300 400 500 600 700 800\ntraining steps (thousands)0.40.50.60.70.80.91.0validation error rateAttentive reader"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab565-6db4-11ee-86ac-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-22", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "0 100 200 300 400 500 600 700 800\ntraining steps (thousands)0.40.50.60.70.80.91.0validation error rateAttentive reader\nLSTM\nBN-LSTM\nBN-everywhere\nLN-LSTMFigure 2: Validation curves for the attentive reader model. BN results are taken from [Cooijmans\net al., 2016].\nWe trained two models: the baseline order-embedding model as well as the same model with layer\nnormalization applied to the GRU. After every 300 iterations, we compute Recall@K (R@K) values\non a held out validation set and save the model whenever R@K improves. The best performing\nmodels are then evaluated on 5 separate test sets, each containing 1000 images and 5000 captions,\nfor which the mean results are reported. Both models use Adam [Kingma and Ba, 2014] with the\nsame initial hyperparameters and both models are trained using the same architectural choices as\nused in Vendrov et al. [2016]. We refer the reader to the appendix for a description of how layer\nnormalization is applied to GRU.\nFigure 1illustrates the validation curves of the models, with and without layer normalization. We\nplot R@1, R@5 and R@10 for the image retrieval task. We observe that layer normalization offers"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab566-6db4-11ee-ab0a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-23", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "plot R@1, R@5 and R@10 for the image retrieval task. We observe that layer normalization offers\na per-iteration speedup across all metrics and converges to its best validation model in 60% of the\ntime it takes the baseline model to do so. In Table 2, the test set results are reported from which we\nobserve that layer normalization also results in improved generalization over the original model. The\nresults we report are state-of-the-art for RNN embedding models, with only the structure-preserving\nmodel of Wang et al. [2016] reporting better results on this task. However, they evaluate under\ndifferent conditions (1 test set instead of the mean over 5) and are thus not directly comparable.\n6.2 Teaching machines to read and comprehend\nIn order to compare layer normalization to the recently proposed recurrent batch normalization\n[Cooijmans et al., 2016], we train an unidirectional attentive reader model on the CNN corpus both\nintroduced by Hermann et al. [2015]. This is a question-answering task where a query description\nabout a passage must be answered by \ufb01lling in a blank. The data is anonymized such that entities"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab567-6db4-11ee-b6a8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-24", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "about a passage must be answered by \ufb01lling in a blank. The data is anonymized such that entities\nare given randomized tokens to prevent degenerate solutions, which are consistently permuted during training and evaluation. We follow the same experimental protocol as Cooijmans et al. [2016]\nand modify their public code to incorporate layer normalization2which uses Theano [Team et al.,\n2016]. We obtained the pre-processed dataset used by Cooijmans et al. [2016] which differs from\nthe original experiments of Hermann et al. [2015] in that each passage is limited to 4 sentences.\nIn Cooijmans et al. [2016], two variants of recurrent batch normalization are used: one where BN\nis only applied to the LSTM while the other applies BN everywhere throughout the model. In our\nexperiment, we only apply layer normalization within the LSTM.\nThe results of this experiment are shown in Figure 2. We observe that layer normalization not only\ntrains faster but converges to a better validation result over both the baseline and BN variants. In\nCooijmans et al. [2016], it is argued that the scale parameter in BN must be carefully chosen and is\nset to 0.1 in their experiments. We experimented with layer normalization for both 1.0 and 0.1 scale"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab568-6db4-11ee-9b89-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-25", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "set to 0.1 in their experiments. We experimented with layer normalization for both 1.0 and 0.1 scale\ninitialization and found that the former model performed signi\ufb01cantly better. This demonstrates that\nlayer normalization is not sensitive to the initial scale in the same way that recurrent BN is.3\n6.3 Skip-thought vectors\nSkip-thoughts [Kiros et al., 2015] is a generalization of the skip-gram model [Mikolov et al., 2013]\nfor learning unsupervised distributed sentence representations. Given contiguous text, a sentence is\n2https://github.com/cooijmanstim/Attentive_reader/tree/bn\n3We only produce results on the validation set, as in the case of Cooijmans et al. [2016]\n7\n5 10 15 20\niteration x 5000082.082.583.083.584.084.585.085.586.0Pearson x 100\nSkip-Thoughts + LN\nSkip-Thoughts\nOriginal(a) SICK(r)\n5 10 15 20\niteration x 500002728293031323334MSE x 100\nSkip-Thoughts + LN\nSkip-Thoughts\nOriginal (b) SICK(MSE)\n5 10 15 20"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab569-6db4-11ee-a7d0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-26", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "Skip-Thoughts + LN\nSkip-Thoughts\nOriginal (b) SICK(MSE)\n5 10 15 20\niteration x 5000070727476788082Accuracy\nSkip-Thoughts + LN\nSkip-Thoughts\nOriginal (c) MR\n5 10 15 20\niteration x 5000074767880828486Accuracy\nSkip-Thoughts + LN\nSkip-Thoughts\nOriginal\n(d) CR\n5 10 15 20\niteration x 5000090.090.591.091.592.092.593.093.594.094.5Accuracy\nSkip-Thoughts + LN\nSkip-Thoughts\nOriginal (e) SUBJ\n5 10 15 20\niteration x 50000838485868788899091Accuracy\nSkip-Thoughts + LN\nSkip-Thoughts\nOriginal (f) MPQA\nFigure 3: Performance of skip-thought vectors with and without layer normalization on downstream\ntasks as a function of training iterations. The original lines are the reported results in [Kiros et al.,\n2015]. Plots with error use 10-fold cross validation. Best seen in color.\nMethod SICK( r) SICK(\u001a) SICK( MSE ) MR CR SUBJ MPQA"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab56a-6db4-11ee-856e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-27", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "Method SICK( r) SICK(\u001a) SICK( MSE ) MR CR SUBJ MPQA\nOriginal [Kiros et al., 2015] 0.848 0.778 0.287 75.5 79.3 92.1 86.9\nOurs 0.842 0.767 0.298 77.3 81.8 92.6 87.9\nOurs + LN 0.854 0.785 0.277 79.5 82.6 93.4 89.0\nOurs + LN y 0.858 0.788 0.270 79.4 83.1 93.7 89.3\nTable 3: Skip-thoughts results. The \ufb01rst two evaluation columns indicate Pearson and Spearman correlation, the third is mean squared error and the remaining indicate classi\ufb01cation accuracy. Higher is\nbetter for all evaluations except MSE. Our models were trained for 1M iterations with the exception\nof (y) which was trained for 1 month (approximately 1.7M iterations)\nencoded with a encoder RNN and decoder RNNs are used to predict the surrounding sentences.\nKiros et al. [2015] showed that this model could produce generic sentence representations that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbcd-6db4-11ee-b76e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-28", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "Kiros et al. [2015] showed that this model could produce generic sentence representations that\nperform well on several tasks without being \ufb01ne-tuned. However, training this model is timeconsuming, requiring several days of training in order to produce meaningful results.\nIn this experiment we determine to what effect layer normalization can speed up training. Using the\npublicly available code of Kiros et al. [2015]4, we train two models on the BookCorpus dataset [Zhu\net al., 2015]: one with and one without layer normalization. These experiments are performed with\nTheano [Team et al., 2016]. We adhere to the experimental setup used in Kiros et al. [2015], training\na 2400-dimensional sentence encoder with the same hyperparameters. Given the size of the states\nused, it is conceivable layer normalization would produce slower per-iteration updates than without.\nHowever, we found that provided CNMeM5is used, there was no signi\ufb01cant difference between the\ntwo models. We checkpoint both models after every 50,000 iterations and evaluate their performance\non \ufb01ve tasks: semantic-relatedness (SICK) [Marelli et al., 2014], movie review sentiment (MR)\n[Pang and Lee, 2005], customer product reviews (CR) [Hu and Liu, 2004], subjectivity/objectivity"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbce-6db4-11ee-9c10-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-29", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "[Pang and Lee, 2005], customer product reviews (CR) [Hu and Liu, 2004], subjectivity/objectivity\nclassi\ufb01cation (SUBJ) [Pang and Lee, 2004] and opinion polarity (MPQA) [Wiebe et al., 2005]. We\nplot the performance of both models for each checkpoint on all tasks to determine whether the\nperformance rate can be improved with LN.\nThe experimental results are illustrated in Figure 3. We observe that applying layer normalization\nresults both in speedup over the baseline as well as better \ufb01nal results after 1M iterations are performed as shown in Table 3. We also let the model with layer normalization train for a total of a\nmonth, resulting in further performance gains across all but one task. We note that the performance\n4https://github.com/ryankiros/skip-thoughts\n5https://github.com/NVIDIA/cnmem\n8\n100101102103\nUpdates x 200\u2212900\u2212800\u2212700\u2212600\u2212500\u2212400\u2212300\u2212200\u22121000Negative Log LikelihoodBaseline test\nBaseline train\nLN test\nLN trainFigure 5: Handwriting sequence generation model negative log likelihood with and without layer\nnormalization. The models are trained with mini-batch size of 8 and sequence length of 500,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbcf-6db4-11ee-aac2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-30", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "LN trainFigure 5: Handwriting sequence generation model negative log likelihood with and without layer\nnormalization. The models are trained with mini-batch size of 8 and sequence length of 500,\ndifferences between the original reported results and ours are likely due to the fact that the publicly\navailable code does not condition at each timestep of the decoder, where the original model does.\n6.4 Modeling binarized MNIST using DRAW\n0 20 40 60 80 100\nEpoch80859095100Test Variational BoundBaseline\nWN\nLN\nFigure 4: DRAW model test negative log likelihood with and without\nlayer normalization.We also experimented with the generative modeling on the\nMNIST dataset. Deep Recurrent Attention Writer (DRAW)\n[Gregor et al., 2015] has previously achieved the state-of-theart performance on modeling the distribution of MNIST digits. The model uses a differential attention mechanism and\na recurrent neural network to sequentially generate pieces of\nan image. We evaluate the effect of layer normalization on\na DRAW model using 64 glimpses and 256 LSTM hidden\nunits. The model is trained with the default setting of Adam\n[Kingma and Ba, 2014] optimizer and the minibatch size of\n128. Previous publications on binarized MNIST have used\nvarious training protocols to generate their datasets. In this"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbd0-6db4-11ee-91d1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-31", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "128. Previous publications on binarized MNIST have used\nvarious training protocols to generate their datasets. In this\nexperiment, we used the \ufb01xed binarization from Larochelle\nand Murray [2011]. The dataset has been split into 50,000\ntraining, 10,000 validation and 10,000 test images.\nFigure 4 shows the test variational bound for the \ufb01rst 100\nepoch. It highlights the speedup bene\ufb01t of applying layer normalization that the layer normalized DRAW converges almost twice as fast than the baseline model.\nAfter 200 epoches, the baseline model converges to a variational log likelihood of 82.36 nats on the\ntest data and the layer normalization model obtains 82.09 nats.\n6.5 Handwriting sequence generation\nThe previous experiments mostly examine RNNs on NLP tasks whose lengths are in the range of 10\nto 40. To show the effectiveness of layer normalization on longer sequences, we performed handwriting generation tasks using the IAM Online Handwriting Database [Liwicki and Bunke, 2005].\nIAM-OnDB consists of handwritten lines collected from 221 different writers. When given the input\ncharacter string, the goal is to predict a sequence of x and y pen co-ordinates of the corresponding"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbd1-6db4-11ee-af71-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-32", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "character string, the goal is to predict a sequence of x and y pen co-ordinates of the corresponding\nhandwriting line on the whiteboard. There are, in total, 12179 handwriting line sequences. The input\nstring is typically more than 25 characters and the average handwriting line has a length around 700.\nWe used the same model architecture as in Section (5.2) of Graves [2013]. The model architecture\nconsists of three hidden layers of 400 LSTM cells, which produce 20 bivariate Gaussian mixture\ncomponents at the output layer, and a size 3 input layer. The character sequence was encoded with\none-hot vectors, and hence the window vectors were size 57. A mixture of 10 Gaussian functions\nwas used for the window parameters, requiring a size 30 parameter vector. The total number of\nweights was increased to approximately 3.7M. The model is trained using mini-batches of size 8\nand the Adam [Kingma and Ba, 2014] optimizer.\nThe combination of small mini-batch size and very long sequences makes it important to have very\nstable hidden dynamics. Figure 5 shows that layer normalization converges to a comparable log\nlikelihood as the baseline model but is much faster.\n9\n0 10 20 30 40 50 60\nEpoch10-710-610-510-410-310-210-1100Train NLL\nBatchNorm bz128"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbd2-6db4-11ee-bb9e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-33", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "9\n0 10 20 30 40 50 60\nEpoch10-710-610-510-410-310-210-1100Train NLL\nBatchNorm bz128\nBaseline bz128\nLayerNorm bz128\n0 10 20 30 40 50 60\nEpoch0.0050.0100.0150.0200.025Test Err.\nBatchNorm bz128\nBaseline bz128\nLayerNorm bz128\n0 10 20 30 40 50 60\nEpoch10-710-610-510-410-310-210-1100Train NLL\nLayerNorm bz4\nBaseline bz4\nBatchNorm bz4\n0 10 20 30 40 50 60\nEpoch0.0050.0100.0150.0200.025Test Err.\nLayerNorm bz4\nBaseline bz4\nBatchNorm bz4Figure 6: Permutation invariant MNIST 784-1000-1000-10 model negative log likelihood and test\nerror with layer normalization and batch normalization. (Left) The models are trained with batchsize of 128. (Right) The models are trained with batch-size of 4.\n6.6 Permutation invariant MNIST\nIn addition to RNNs, we investigated layer normalization in feed-forward networks. We show how"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbd3-6db4-11ee-a28f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-34", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "6.6 Permutation invariant MNIST\nIn addition to RNNs, we investigated layer normalization in feed-forward networks. We show how\nlayer normalization compares with batch normalization on the well-studied permutation invariant\nMNIST classi\ufb01cation problem. From the previous analysis, layer normalization is invariant to input\nre-scaling which is desirable for the internal hidden layers. But this is unnecessary for the logit\noutputs where the prediction con\ufb01dence is determined by the scale of the logits. We only apply\nlayer normalization to the fully-connected hidden layers that excludes the last softmax layer.\nAll the models were trained using 55000 training data points and the Adam [Kingma and Ba, 2014]\noptimizer. For the smaller batch-size, the variance term for batch normalization is computed using\nthe unbiased estimator. The experimental results from Figure 6 highlight that layer normalization is\nrobust to the batch-sizes and exhibits a faster training convergence comparing to batch normalization\nthat is applied to all layers.\n6.7 Convolutional Networks\nWe have also experimented with convolutional neural networks. In our preliminary experiments, we\nobserved that layer normalization offers a speedup over the baseline model without normalization,\nbut batch normalization outperforms the other methods. With fully connected layers, all the hidden"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbd4-6db4-11ee-8a45-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-35", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "observed that layer normalization offers a speedup over the baseline model without normalization,\nbut batch normalization outperforms the other methods. With fully connected layers, all the hidden\nunits in a layer tend to make similar contributions to the \ufb01nal prediction and re-centering and rescaling the summed inputs to a layer works well. However, the assumption of similar contributions\nis no longer true for convolutional neural networks. The large number of the hidden units whose\nreceptive \ufb01elds lie near the boundary of the image are rarely turned on and thus have very different\nstatistics from the rest of the hidden units within the same layer. We think further research is needed\nto make layer normalization work well in ConvNets.\n7 Conclusion\nIn this paper, we introduced layer normalization to speed-up the training of neural networks. We\nprovided a theoretical analysis that compared the invariance properties of layer normalization with\nbatch normalization and weight normalization. We showed that layer normalization is invariant to\nper training-case feature shifting and scaling.\nEmpirically, we showed that recurrent neural networks bene\ufb01t the most from the proposed method\nespecially for long sequences and small mini-batches.\nAcknowledgments\nThis research was funded by grants from NSERC, CFI, and Google.\n10\nReferences"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbd5-6db4-11ee-b341-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-36", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "especially for long sequences and small mini-batches.\nAcknowledgments\nThis research was funded by grants from NSERC, CFI, and Google.\n10\nReferences\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classi\ufb01cation with deep convolutional neural\nnetworks. In NIPS , 2012.\nGeoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior,\nVincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in\nspeech recognition: The shared views of four research groups. IEEE , 2012.\nJeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker,\nKe Yang, Quoc V Le, et al. Large scale distributed deep networks. In NIPS , 2012.\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing\ninternal covariate shift. ICML , 2015.\nIlya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In\nAdvances in neural information processing systems , pages 3104\u20133112, 2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbd6-6db4-11ee-b9dc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-37", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "Advances in neural information processing systems , pages 3104\u20133112, 2014.\nC\u00b4esar Laurent, Gabriel Pereyra, Phil \u00b4emon Brakel, Ying Zhang, and Yoshua Bengio. Batch normalized recurrent\nneural networks. arXiv preprint arXiv:1510.01378 , 2015.\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen,\nMike Chrzanowski, Adam Coates, Greg Diamos, et al. Deep speech 2: End-to-end speech recognition in\nenglish and mandarin. arXiv preprint arXiv:1512.02595 , 2015.\nTim Cooijmans, Nicolas Ballas, C \u00b4esar Laurent, and Aaron Courville. Recurrent batch normalization. arXiv\npreprint arXiv:1603.09025 , 2016.\nTim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868 , 2016.\nBehnam Neyshabur, Ruslan R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbd7-6db4-11ee-b0a9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-38", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "Behnam Neyshabur, Ruslan R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep\nneural networks. In Advances in Neural Information Processing Systems , pages 2413\u20132421, 2015.\nShun-Ichi Amari. Natural gradient works ef\ufb01ciently in learning. Neural computation , 1998.\nIvan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language.\nICLR , 2016.\nThe Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller,\nDzmitry Bahdanau, Nicolas Ballas, Fr \u00b4ed\u00b4eric Bastien, Justin Bayer, Anatoly Belikov, et al. Theano: A python\nframework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 , 2016.\nTsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \u00b4ar, and\nC Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV , 2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbd8-6db4-11ee-b187-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-39", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV , 2014.\nKyunghyun Cho, Bart Van Merri \u00a8enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger\nSchwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical\nmachine translation. EMNLP , 2014.\nKaren Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.\nICLR , 2015.\nRyan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 , 2014.\nD. Kingma and J. L. Ba. Adam: a method for stochastic optimization. ICLR , 2014. arXiv:1412.6980.\nLiwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embeddings.\nCVPR , 2016.\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbd9-6db4-11ee-979b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-40", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "CVPR , 2016.\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman,\nand Phil Blunsom. Teaching machines to read and comprehend. In NIPS , 2015.\nRyan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja\nFidler. Skip-thought vectors. In NIPS , 2015.\nTomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Ef\ufb01cient estimation of word representations in\nvector space. arXiv preprint arXiv:1301.3781 , 2013.\nYukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja\nFidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading\nbooks. In ICCV , 2015.\nMarco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli.\nSemeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbda-6db4-11ee-861c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-41", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through\nsemantic relatedness and textual entailment. SemEval-2014 , 2014.\n11\nBo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect\nto rating scales. In ACL, pages 115\u2013124, 2005.\nMinqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM\nSIGKDD international conference on Knowledge discovery and data mining , 2004.\nBo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based\non minimum cuts. In ACL, 2004.\nJanyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation , 2005.\nK. Gregor, I. Danihelka, A. Graves, and D. Wierstra. DRAW: a recurrent neural network for image generation.\narXiv:1502.04623, 2015.\nHugo Larochelle and Iain Murray. The neural autoregressive distribution estimator. In AISTATS , volume 6,\npage 622, 2011.\nMarcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbdb-6db4-11ee-8267-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-42", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "page 622, 2011.\nMarcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten\ntext on a whiteboard. In ICDAR , 2005.\nAlex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013.\n12\nSupplementary Material\nApplication of layer normalization to each experiment\nThis section describes how layer normalization is applied to each of the papers\u2019 experiments. For\nnotation convenience, we de\ufb01ne layer normalization as a function mapping LN:RD!RDwith\ntwo set of adaptive parameters, gains \u000band biases \f:\nLN(z;\u000b;\f) =(z\u0000\u0016)\n\u001b\f\u000b+\f; (15)\n\u0016=1\nDDX\ni=1zi; \u001b =vuut1\nDDX\ni=1(zi\u0000\u0016)2; (16)\nwhere,ziis theithelement of the vector z.\nTeaching machines to read and comprehend and handwriting sequence generation\nThe basic LSTM equations used for these experiment are given by:\n0\nB@ft\nit\not\ngt1\nCA=Whht\u00001+Wxxt+b (17)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbdc-6db4-11ee-a95d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-43", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "0\nB@ft\nit\not\ngt1\nCA=Whht\u00001+Wxxt+b (17)\nct=\u001b(ft)\fct\u00001+\u001b(it)\ftanh(gt) (18)\nht=\u001b(ot)\ftanh(ct) (19)\nThe version that incorporates layer normalization is modi\ufb01ed as follows:\n0\nB@ft\nit\not\ngt1\nCA=LN(Whht\u00001;\u000b1;\f1) +LN(Wxxt;\u000b2;\f2) +b (20)\nct=\u001b(ft)\fct\u00001+\u001b(it)\ftanh(gt) (21)\nht=\u001b(ot)\ftanh(LN(ct;\u000b3;\f3)) (22)\nwhere \u000bi;\fiare the additive and multiplicative parameters, respectively. Each \u000biis initialized to a\nvector of zeros and each \fiis initialized to a vector of ones.\nOrder embeddings and skip-thoughts\nThese experiments utilize a variant of gated recurrent unit which is de\ufb01ned as follows:\n\u0012\nzt\nrt\u0013\n=Whht\u00001+Wxxt (23)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbdd-6db4-11ee-8329-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-44", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "\u0012\nzt\nrt\u0013\n=Whht\u00001+Wxxt (23)\n^ht=tanh(Wxt+\u001b(rt)\f(Uht\u00001)) (24)\nht= (1\u0000\u001b(zt))ht\u00001+\u001b(zt)^ht (25)\nLayer normalization is applied as follows:\n\u0012\nzt\nrt\u0013\n=LN(Whht\u00001;\u000b1;\f1) +LN(Wxxt;\u000b2;\f2) (26)\n^ht=tanh(LN(Wxt;\u000b3;\f3) +\u001b(rt)\fLN(Uht\u00001;\u000b4;\f4)) (27)\nht= (1\u0000\u001b(zt))ht\u00001+\u001b(zt)^ht (28)\njust as before, \u000biis initialized to a vector of zeros and each \fiis initialized to a vector of ones.\n13\nModeling binarized MNIST using DRAW\nThe layer norm is only applied to the output of the LSTM hidden states in this experiment:\nThe version that incorporates layer normalization is modi\ufb01ed as follows:0\nB@ft\nit\not\ngt1\nCA=Whht\u00001+Wxxt+b (29)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbde-6db4-11ee-b87c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-45", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "B@ft\nit\not\ngt1\nCA=Whht\u00001+Wxxt+b (29)\nct=\u001b(ft)\fct\u00001+\u001b(it)\ftanh(gt) (30)\nht=\u001b(ot)\ftanh(LN(ct;\u000b;\f)) (31)\nwhere \u000b;\fare the additive and multiplicative parameters, respectively. \u000bis initialized to a vector\nof zeros and \fis initialized to a vector of ones.\nLearning the magnitude of incoming weights\nWe now compare how gradient descent updates changing magnitude of the equivalent weights between the normalized GLM and original parameterization. The magnitude of the weights are explicitly parameterized using the gain parameter in the normalized model. Assume there is a gradient\nupdate that changes norm of the weight vectors by \u000eg. We can project the gradient updates to the\nweight vector for the normal GLM. The KL metric, ie how much the gradient update changes the\nmodel prediction, for the normalized model depends only on the magnitude of the prediction error.\nSpeci\ufb01cally,\nunder batch normalization:\nds2=1\n2vec([0;0;\u000eg]>)>\u0016F(vec([W;b;g]>) vec([0;0;\u000eg]>) =1\n2\u000e>\ngE"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbdf-6db4-11ee-abf9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-46", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "2\u000e>\ngE\nx\u0018P(x)\u0014Cov[yjx]\n\u001e2\u0015\n\u000eg:\n(32)\nUnder layer normalization:\nds2=1\n2vec([0;0;\u000eg]>)>\u0016F(vec([W;b;g]>) vec([0;0;\u000eg]>)\n=1\n2\u000e>\ng1\n\u001e2E\nx\u0018P(x)2\n664Cov(y1;y1jx)(a1\u0000\u0016)2\n\u001b2\u0001\u0001\u0001 Cov(y1;yHjx)(a1\u0000\u0016)(aH\u0000\u0016)\n\u001b2\n.........\nCov(yH;y1jx)(aH\u0000\u0016)(a1\u0000\u0016)\n\u001b2\u0001\u0001\u0001 Cov(yH;yHjx)(aH\u0000\u0016)2\n\u001b23\n775\u000eg\n(33)\nUnder weight normalization:\nds2=1\n2vec([0;0;\u000eg]>)>\u0016F(vec([W;b;g]>) vec([0;0;\u000eg]>)\n=1\n2\u000e>\ng1\n\u001e2E\nx\u0018P(x)2\n6664Cov(y1;y1jx)a2\n1\nkw1k2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbe0-6db4-11ee-89de-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-47", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "g1\n\u001e2E\nx\u0018P(x)2\n6664Cov(y1;y1jx)a2\n1\nkw1k2\n2\u0001\u0001\u0001 Cov(y1;yHjx)a1aH\nkw1k2kwHk2\n.........\nCov(yH;y1jx)aHa1\nkwHk2kw1k2\u0001\u0001\u0001 Cov(yH;yHjx)a2\nH\nkwHk2\n23\n7775\u000eg:(34)\nWhereas, the KL metric in the standard GLM is related to its activities ai=w>\nix, that is depended\non both its current weights and input data. We project the gradient updates to the gain parameter \u000egi\nof theithneuron to its weight vector as \u000egiwi\nkwik2in the standard GLM model:\n1\n2vec([\u000egiwi\nkwik2;0;\u000egjwj\nkwik2;0]>)>F([w>\ni;bi;w>\nj;bj]>) vec([\u000egiwi\nkwik2;0;\u000egjwj\nkwjk2;0]>)\n=\u000egi\u000egj\n2\u001e2E\nx\u0018P(x)\u0014"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbe1-6db4-11ee-b3d7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1607.06450-48", "title": "Layer Normalization", "source": "http://arxiv.org/pdf/1607.06450", "start_index": 0, "chroma:document": "kwik2;0;\u000egjwj\nkwjk2;0]>)\n=\u000egi\u000egj\n2\u001e2E\nx\u0018P(x)\u0014\nCov(yi;yjjx)aiaj\nkwik2kwjk2\u0015\n(35)\nThe batch normalized and layer normalized models are therefore more robust to the scaling of the\ninput and its parameters than the standard model.\n14"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbe2-6db4-11ee-9c13-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-0", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Published as a conference paper at ICLR 2017\nHIERARCHICAL MULTISCALE\nRECURRENT NEURAL NETWORKS\nJunyoung Chung, Sungjin Ahn & Yoshua Bengio\u0003\nD\u00e9partement d\u2019informatique et de recherche op\u00e9rationnelle\nUniversit\u00e9 de Montr\u00e9al\n{junyoung.chung,sungjin.ahn,yoshua.bengio}@umontreal.ca\nABSTRACT\nLearning both hierarchical and temporal representation has been among the longstanding challenges of recurrent neural networks. Multiscale recurrent neural\nnetworks have been considered as a promising approach to resolve this issue, yet\nthere has been a lack of empirical evidence showing that this type of models can\nactually capture the temporal dependencies by discovering the latent hierarchical\nstructure of the sequence. In this paper, we propose a novel multiscale approach,\ncalled the hierarchical multiscale recurrent neural network, that can capture the\nlatent hierarchical structure in the sequence by encoding the temporal dependencies\nwith different timescales using a novel update mechanism. We show some evidence\nthat the proposed model can discover underlying hierarchical structure in the\nsequences without using explicit boundary information. We evaluate our proposed\nmodel on character-level language modelling and handwriting sequence generation.\n1 I NTRODUCTION\nOne of the key principles of learning in deep neural networks as well as in the human brain is to obtain"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbe3-6db4-11ee-acd3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-1", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "1 I NTRODUCTION\nOne of the key principles of learning in deep neural networks as well as in the human brain is to obtain\na hierarchical representation with increasing levels of abstraction (Bengio, 2009; LeCun et al., 2015;\nSchmidhuber, 2015). A stack of representation layers, learned from the data in a way to optimize\nthe target task, make deep neural networks entertain advantages such as generalization to unseen\nexamples (Hoffman et al., 2013), sharing learned knowledge among multiple tasks, and discovering\ndisentangling factors of variation (Kingma & Welling, 2013). The remarkable recent successes of\nthe deep convolutional neural networks are particularly based on this ability to learn hierarchical\nrepresentation for spatial data (Krizhevsky et al., 2012). For modelling temporal data, the recent\nresurgence of recurrent neural networks (RNN) has led to remarkable advances (Mikolov et al., 2010;\nGraves, 2013; Cho et al., 2014; Sutskever et al., 2014; Vinyals et al., 2015). However, unlike the\nspatial data, learning both hierarchical and temporal representation has been among the long-standing\nchallenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbe4-6db4-11ee-8650-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-2", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "spatial data, learning both hierarchical and temporal representation has been among the long-standing\nchallenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many\ntemporal data (Schmidhuber, 1991; Mozer, 1993; El Hihi & Bengio, 1995; Lin et al., 1996; Koutn\u00edk\net al., 2014).\nA promising approach to model such hierarchical and temporal representation is the multiscale\nRNNs (Schmidhuber, 1992; El Hihi & Bengio, 1995; Koutn\u00edk et al., 2014). Based on the observation\nthat high-level abstraction changes slowly with temporal coherency while low-level abstraction\nhas quickly changing features sensitive to the precise local timing (El Hihi & Bengio, 1995), the\nmultiscale RNNs group hidden units into multiple modules of different timescales. In addition to\nthe fact that the architecture \ufb01ts naturally to the latent hierarchical structures in many temporal data,\nthe multiscale approach provides the following advantages that resolve some inherent problems\nof standard RNNs: (a) computational ef\ufb01ciency obtained by updating the high-level layers less\nfrequently, (b) ef\ufb01ciently delivering long-term dependencies with fewer updates at the high-level"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbe5-6db4-11ee-9f1f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-3", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "frequently, (b) ef\ufb01ciently delivering long-term dependencies with fewer updates at the high-level\nlayers, which mitigates the vanishing gradient problem, (c) \ufb02exible resource allocation (e.g., more\nhidden units to the higher layers that focus on modelling long-term dependencies and less hidden\nunits to the lower layers which are in charge of learning short-term dependencies). In addition, the\nlearned latent hierarchical structures can provide useful information to other downstream tasks such\n\u0003Yoshua Bengio is CIFAR Senior Fellow.\n1arXiv:1609.01704v7  [cs.LG]  9 Mar 2017\nPublished as a conference paper at ICLR 2017\nas module structures in computer program learning, sub-task structures in hierarchical reinforcement\nlearning, and story segments in video understanding.\nThere have been various approaches to implementing the multiscale RNNs. The most popular\napproach is to set the timescales as hyperparameters (El Hihi & Bengio, 1995; Koutn\u00edk et al., 2014;\nBahdanau et al., 2016) instead of treating them as dynamic variables that can be learned from the\ndata (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016). However, considering the fact that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbe6-6db4-11ee-9678-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-4", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016). However, considering the fact that\nnon-stationarity is prevalent in temporal data, and that many entities of abstraction such as words\nand sentences are in variable length, we claim that it is important for an RNN to dynamically adapt\nits timescales to the particulars of the input entities of various length. While this is trivial if the\nhierarchical boundary structure is provided (Sordoni et al., 2015), it has been a challenge for an RNN\nto discover the latent hierarchical structure in temporal data without explicit boundary information.\nIn this paper, we propose a novel multiscale RNN model, which can learn the hierarchical multiscale\nstructure from temporal data without explicit boundary information. This model, called a hierarchical\nmultiscale recurrent neural network (HM-RNN), does not assign \ufb01xed update rates, but adaptively\ndetermines proper update times corresponding to different abstraction levels of the layers. We \ufb01nd\nthat this model tends to learn \ufb01ne timescales for low-level layers and coarse timescales for high-level\nlayers. To do this, we introduce a binary boundary detector at each layer. The boundary detector is\nturned on only at the time steps where a segment of the corresponding abstraction level is completely"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbe7-6db4-11ee-8a4d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-5", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "layers. To do this, we introduce a binary boundary detector at each layer. The boundary detector is\nturned on only at the time steps where a segment of the corresponding abstraction level is completely\nprocessed. Otherwise, i.e., during the within segment processing, it stays turned off. Using the\nhierarchical boundary states, we implement three operations, UPDATE, COPY and FLUSH, and\nchoose one of them at each time step. The UPDATE operation is similar to the usual update rule of\nthe long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997), except that it is executed\nsparsely according to the detected boundaries. The COPY operation simply copies the cell and hidden\nstates of the previous time step. Unlike the leaky integration of the LSTM or the Gated Recurrent\nUnit (GRU) (Cho et al., 2014), the COPY operation retains the whole states without any loss of\ninformation. The FLUSH operation is executed when a boundary is detected, where it \ufb01rst ejects the\nsummarized representation of the current segment to the upper layer and then reinitializes the states\nto start processing the next segment. Learning to select a proper operation at each time step and to\ndetect the boundaries, the HM-RNN discovers the latent hierarchical structure of the sequences. We"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbe8-6db4-11ee-bcd9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-6", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "to start processing the next segment. Learning to select a proper operation at each time step and to\ndetect the boundaries, the HM-RNN discovers the latent hierarchical structure of the sequences. We\n\ufb01nd that the straight-through estimator (Hinton, 2012; Bengio et al., 2013; Courbariaux et al., 2016)\nis ef\ufb01cient for training this model containing discrete variables.\nWe evaluate our model on two tasks: character-level language modelling and handwriting sequence\ngeneration. For the character-level language modelling, the HM-RNN achieves the state-of-the-art\nresults on the Text8 dataset, and comparable results to the state-of-the-art on the Penn Treebank\nand Hutter Prize Wikipedia datasets. The HM-RNN also outperforms the standard RNN on the\nhandwriting sequence generation using the IAM-OnDB dataset. In addition, we demonstrate that the\nhierarchical structure found by the HM-RNN is indeed very similar to the intrinsic structure observed\nin the data. The contributions of this paper are:\n\u000fWe propose for the \ufb01rst time an RNN model that can learn a latent hierarchical structure of\na sequence without using explicit boundary information.\n\u000fWe show that it is bene\ufb01cial to utilize the above structure through empirical evaluation."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbe9-6db4-11ee-a953-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-7", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "a sequence without using explicit boundary information.\n\u000fWe show that it is bene\ufb01cial to utilize the above structure through empirical evaluation.\n\u000fWe show that the straight-through estimator is an ef\ufb01cient way of training a model containing\ndiscrete variables.\n\u000fWe propose the slope annealing trick to improve the training procedure based on the\nstraight-through estimator.\n2 R ELATED WORK\nTwo notable early attempts inspiring our model are Schmidhuber (1992) and El Hihi & Bengio (1995).\nIn these works, it is advocated to stack multiple layers of RNNs in a decreasing order of update\nfrequency for computational and learning ef\ufb01ciency. In Schmidhuber (1992), the author shows a\nmodel that can self-organize a hierarchical multiscale structure. Particularly in El Hihi & Bengio\n(1995), the advantages of incorporating a priori knowledge, \u201c temporal dependencies are structured\nhierarchically \", into the RNN architecture is studied. The authors propose an RNN architecture that\nupdates each layer with a \ufb01xed but different rate, called a hierarchical RNN.\n2\nPublished as a conference paper at ICLR 2017\nLSTMs (Hochreiter & Schmidhuber, 1997) employ the multiscale update concept, where the hidden"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbea-6db4-11ee-aa17-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-8", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "2\nPublished as a conference paper at ICLR 2017\nLSTMs (Hochreiter & Schmidhuber, 1997) employ the multiscale update concept, where the hidden\nunits have different forget and update rates and thus can operate with different timescales. However,\nunlike our model, these timescales are not organized hierarchically. Although the LSTM has a selfloop for the gradients that helps to capture the long-term dependencies by mitigating the vanishing\ngradient problem, in practice, it is still limited to a few hundred time steps due to the leaky integration\nby which the contents to memorize for a long-term is gradually diluted at every time step. Also, the\nmodel remains computationally expensive because it has to perform the update at every time step\nfor each unit. However, our model is less prone to these problems because it learns a hierarchical\nstructure such that, by design, high-level layers learn to perform less frequent updates than low-level\nlayers. We hypothesize that this property mitigates the vanishing gradient problem more ef\ufb01ciently\nwhile also being computationally more ef\ufb01cient.\nA more recent model, the clockwork RNN (CW-RNN) (Koutn\u00edk et al., 2014) extends the hierarchical"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbeb-6db4-11ee-a0fe-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-9", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "A more recent model, the clockwork RNN (CW-RNN) (Koutn\u00edk et al., 2014) extends the hierarchical\nRNN (El Hihi & Bengio, 1995) and the NARX RNN (Lin et al., 1996)1. The CW-RNN tries to\nsolve the issue of using soft timescales in the LSTM, by explicitly assigning hard timescales. In the\nCW-RNN, hidden units are partitioned into several modules, and different timescales are assigned to\nthe modules such that a module iupdates its hidden units at every 2(i\u00001)-th time step. The CW-RNN\nis computationally more ef\ufb01cient than the standard RNN including the LSTM since hidden units\nare updated only at the assigned clock rates. However, \ufb01nding proper timescales in the CW-RNN\nremains as a challenge whereas our model learns the intrinsic timescales from the data. In the biscale\nRNNs (Chung et al., 2016), the authors proposed to model layer-wise timescales adaptively by having\nadditional gating units, however this approach still relies on the softgating mechanism like LSTMs.\nOther forms of Hierarchical RNN (HRNN) architectures have been proposed in the cases where"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbec-6db4-11ee-8dc3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-10", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Other forms of Hierarchical RNN (HRNN) architectures have been proposed in the cases where\nthe explicit hierarchical boundary structure is provided. In Ling et al. (2015), after obtaining the\nword boundary via tokenization, the HRNN architecture is used for neural machine translation by\nmodelling the characters and words using the \ufb01rst and second RNN layers, respectively. A similar\nHRNN architecture is also adopted in Sordoni et al. (2015) to model dialogue utterances. However,\nin many cases, hierarchical boundary information is not explicitly observed or expensive to obtain.\nAlso, it is unclear how to deploy more layers than the number of boundary levels that is explicitly\nobserved in the data.\nWhile the above models focus on online prediction problems, where a prediction needs to be made by\nusing only the past data, in some cases, predictions are made after observing the whole sequence. In\nthis setting, the input sequence can be regarded as 1-D spatial data, convolutional neural networks\nwith 1-D kernels are proposed in Kim (2014) and Kim et al. (2015) for language modelling and\nsentence classi\ufb01cation. Also, in Chan et al. (2016) and Bahdanau et al. (2016), the authors proposed\nto obtain high-level representation of the sequences of reduced length by repeatedly merging or\npooling the lower-level representation of the sequences."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbed-6db4-11ee-9ae9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-11", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "to obtain high-level representation of the sequences of reduced length by repeatedly merging or\npooling the lower-level representation of the sequences.\nHierarchical RNN architectures have also been used to discover the segmentation structure in\nsequences (Fern\u00e1ndez et al., 2007; Kong et al., 2015). It is however different to our model in the\nsense that they optimize the objective with explicit labels on the hierarchical segments while our\nmodel discovers the intrinsic structure only from the sequences without segment label information.\nThe COPY operation used in our model can be related to Zoneout (Krueger et al., 2016) which\nis a recurrent generalization of stochastic depth (Huang et al., 2016). In Zoneout, an identity\ntransformation is randomly applied to each hidden unit at each time step according to a Bernoulli\ndistribution. This results in occasional copy operations of the previous hidden states. While the focus\nof Zoneout is to propose a regularization technique similar to dropout (Srivastava et al., 2014) (where\nthe regularization strength is controlled by a hyperparameter), our model learns (a) to dynamically\ndetermine when to copy from the context inputs and (b) to discover the hierarchical multiscale\nstructure and representation. Although the main goal of our proposed model is not regularization, we\nfound that our model also shows very good generalization performance."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbee-6db4-11ee-ab54-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-12", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "structure and representation. Although the main goal of our proposed model is not regularization, we\nfound that our model also shows very good generalization performance.\n3\nPublished as a conference paper at ICLR 2017\nh20h22h25h15h14h13h12h11h10x1x2x3x4x5word1word2phrase1\nh20h21h22h23h24h25h15h14h13h12h11h10x1x2x3x4x5word1word2phrase1\n(a) (b)\nFigure 1: (a) The HRNN architecture, which requires the knowledge of the hierarchical boundaries.\n(b) The HM-RNN architecture that discovers the hierarchical multiscale structure in the data.\n3 H IERARCHICAL MULTISCALE RECURRENT NEURAL NETWORKS\n3.1 M OTIVATION\nTo begin with, we provide an example of how a stacked RNN can model temporal data in an ideal\nsetting, i.e., when the hierarchy of segments is provided (Sordoni et al., 2015; Ling et al., 2015). In\nFigure 1 (a), we depict a hierarchical RNN (HRNN) for language modelling with two layers: the \ufb01rst\nlayer receives characters as inputs and generates word-level representations (C2W-RNN), and the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbef-6db4-11ee-9456-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-13", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "layer receives characters as inputs and generates word-level representations (C2W-RNN), and the\nsecond layer takes the word-level representations as inputs and yields phrase-level representations\n(W2P-RNN).\nAs shown, by means of the provided end-of-word labels, the C2W-RNN obtains word-level representation after processing the last character of each word and passes the word-level representation to the\nW2P-RNN. Then, the W2P-RNN performs an update of the phrase-level representation. Note that the\nhidden states of the W2P-RNN remains unchanged while all the characters of a word are processed by\nthe C2W-RNN. When the C2W-RNN starts to process the next word, its hidden states are reinitialized\nusing the latest hidden states of the W2P-RNN, which contain summarized representation of all the\nwords that have been processed by that time step, in that phrase.\nFrom this simple example, we can see the advantages of having a hierarchical multiscale structure: (1)\nas the W2P-RNN is updated at a much slower update rate than the C2W-RNN, a considerable amount\nof computation can be saved, (2) gradients are backpropagated through a much smaller number of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbf0-6db4-11ee-9a9c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-14", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "of computation can be saved, (2) gradients are backpropagated through a much smaller number of\ntime steps, and (3) layer-wise capacity control becomes possible (e.g., use a smaller number of hidden\nunits in the \ufb01rst layer which models short-term dependencies but whose updates are invoked much\nmore often).\nCan an RNN discover such hierarchical multiscale structure without explicit hierarchical boundary\ninformation? Considering the fact that the boundary information is dif\ufb01cult to obtain (for example,\nconsider languages where words are not always cleanly separated by spaces or punctuation symbols,\nand imperfect rules are used to separately perform segmentation) or usually not provided at all, this is\na legitimate problem. It gets worse when we consider higher-level concepts which we would like\nthe RNN to discover autonomously. In Section 2, we discussed the limitations of the existing RNN\nmodels under this setting, which either have to update all units at every time step or use \ufb01xed update\nfrequencies (El Hihi & Bengio, 1995; Koutn\u00edk et al., 2014). Unfortunately, this kind of approach is\nnot well suited to the case where different segments in the hierarchical decomposition have different\nlengths: for example, different words have different lengths, so a \ufb01xed hierarchy would not update its"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbf1-6db4-11ee-b2d8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-15", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "lengths: for example, different words have different lengths, so a \ufb01xed hierarchy would not update its\nupper-level units in synchrony with the natural boundaries in the data.\n3.2 T HEPROPOSED MODEL\nA key element of our model is the introduction of a parametrized boundary detector, which outputs\na binary value, in each layer of a stacked RNN, and learns when a segment should end in such\na way to optimize the overall target objective. Whenever the boundary detector is turned on at a\ntime step of layer `(i.e., when the boundary state is 1), the model considers this to be the end of a\n1The acronym NARX stands for Non-linear Auto-Regressive model with eXogenous inputs.\n4\nPublished as a conference paper at ICLR 2017\nsegment corresponding to the latent abstraction level of that layer (e.g., word or phrase) and feeds the\nsummarized representation of the detected segment into the upper layer ( `+ 1). Using the boundary\nstates, at each time step, each layer selects one of the following operations: UPDATE, COPY or\nFLUSH. The selection is determined by (1) the boundary state of the current time step in the layer\nbelowz`\u00001\ntand (2) the boundary state of the previous time step in the same layer z`\nt\u00001."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbf2-6db4-11ee-a3de-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-16", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "belowz`\u00001\ntand (2) the boundary state of the previous time step in the same layer z`\nt\u00001.\nIn the following, we describe an HM-RNN based on the LSTM update rule. We call this model\na hierarchical multiscale LSTM (HM-LSTM). Consider an HM-LSTM model of Llayers (`=\n1;:::;L ) which, at each layer `, performs the following update at time step t:\nh`\nt;c`\nt;z`\nt=f`\nHM-LSTM (c`\nt\u00001;h`\nt\u00001;h`\u00001\nt;h`+1\nt\u00001;z`\nt\u00001;z`\u00001\nt): (1)\nHere, handcdenote the hidden and cell states, respectively. The function f`\nHM-LSTM is implemented\nas follows. First, using the two boundary states z`\nt\u00001andz`\u00001\nt, the cell state is updated by:\nc`\nt=8\n<\n:f`\nt\fc`\nt\u00001+i`\nt\fg`\ntifz`\nt\u00001= 0andz`\u00001\nt= 1(UPDATE)\nc`"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbf3-6db4-11ee-b074-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-17", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "t\u00001+i`\nt\fg`\ntifz`\nt\u00001= 0andz`\u00001\nt= 1(UPDATE)\nc`\nt\u00001 ifz`\nt\u00001= 0andz`\u00001\nt= 0(COPY)\ni`\nt\fg`\nt ifz`\nt\u00001= 1(FLUSH);(2)\nand then the hidden state is obtained by:\nh`\nt=\u001ah`\nt\u00001 if COPY ,\no`\nt\ftanh (c`\nt)otherwise.(3)\nHere, (f;i;o)are forget, input, output gates, and gis a cell proposal vector. Note that unlike the\nLSTM, it is not necessary to compute these gates and cell proposal values at every time step. For\nexample, in the case of the COPY operation, we do not need to compute any of these values and thus\ncan save computations.\nThe COPY operation, which simply performs (c`\nt;h`\nt) (c`\nt\u00001;h`\nt\u00001), implements the observation\nthat an upper layer should keep its state unchanged until it receives the summarized input from\nthe lower layer. The UPDATE operation is performed to update the summary representation of the\nlayer`if the boundary z`\u00001"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbf4-6db4-11ee-8af6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-18", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "the lower layer. The UPDATE operation is performed to update the summary representation of the\nlayer`if the boundary z`\u00001\ntis detected from the layer below but the boundary z`\nt\u00001was not found\nat the previous time step. Hence, the UPDATE operation is executed sparsely unlike the standard\nRNNs where it is executed at every time step, making it computationally inef\ufb01cient. If a boundary is\ndetected, the FLUSH operation is executed. The FLUSH operation consists of two sub-operations:\n(a) EJECT to pass the current state to the upper layer and then (b) RESET to reinitialize the state\nbefore starting to read a new segment. This operation implicitly forces the upper layer to absorb the\nsummary information of the lower layer segment, because otherwise it will be lost. Note that the\nFLUSH operation is a hard reset in the sense that it completely erases all the previous states of the\nsame layer, which is different from the softreset or softforget operation in the GRU or LSTM.\nWhenever needed (depending on the chosen operation), the gate values ( f`\nt;i`\nt;o`\nt), the cell proposal\ng`\nt, and the pre-activation of the boundary detector ~z`\nt2are then obtained by:\n0\nBBB@f`\nt\ni`"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbf5-6db4-11ee-b345-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-19", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "g`\nt, and the pre-activation of the boundary detector ~z`\nt2are then obtained by:\n0\nBBB@f`\nt\ni`\nt\no`\nt\ng`\nt\n~z`\nt1\nCCCA=0\nBBB@sigm\nsigm\nsigm\ntanh\nhard sigm1\nCCCAfslice\u0010\nsrecurrent (`)\nt +stop-down (`)\nt +sbottom-up (`)\nt +b(`)\u0011\n; (4)\nwhere\nsrecurrent (`)\nt =U`\n`h`\nt\u00001; (5)\nstop-down (`)\nt =z`\nt\u00001U`\n`+1h`+1\nt\u00001; (6)\nsbottom-up (`)\nt =z`\u00001\ntW`\n`\u00001h`\u00001\nt: (7)\nHere, we use Wj\ni2R(4dim (h`)+1)\u0002dim (h`\u00001);Uj\ni2R(4dim (h`)+1)\u0002dim (h`)to denote state transition"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbf6-6db4-11ee-9516-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-20", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "i2R(4dim (h`)+1)\u0002dim (h`)to denote state transition\nparameters from layer ito layerj, and b2R4dim (h`)+1is a bias term. In the last layer L, the\n2~z`\ntcan also be implemented as a function of h`\nt, e.g., ~z`\nt=hard sigm (Uh`\nt).\n5\nPublished as a conference paper at ICLR 2017\n1\u0000z`t\u00001z`t\u00001W``\u00001z`\u00001tz`t\u00001h`+1t\u00001h`t\u00001h`+1th`th`\u00001tU``+1U``W`+1`z`t\n\u02c6xt+1g1tg2tg3th1th2th3thet\nFigure 2: Left: The gating mechanism of the HM-RNN. Right: The output module when L= 3.\ntop-down connection is ignored, and we use h0\nt=xt. Since the input should not be omitted, we\nsetz0\nt= 1for allt. Also, we do not use the boundary detector for the last layer. The hard sigm is\nde\ufb01ned by hard sigm (x) = max\u0000\n0;min\u0000"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbf7-6db4-11ee-97cd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-21", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "de\ufb01ned by hard sigm (x) = max\u0000\n0;min\u0000\n1;ax+1\n2\u0001\u0001withabeing the slope variable.\nUnlike the standard LSTM, the HM-LSTM has a top-down connection from (`+ 1) to`, which is\nallowed to be activated only if a boundary is detected at the previous time step of the layer `(see\nEq. 6). This makes the layer `to be initialized with more long-term information after the boundary\nis detected and execute the FLUSH operation. In addition, the input from the lower layer ( `\u00001)\nbecomes effective only when a boundary is detected at the current time step in the layer ( `\u00001) due\nto the binary gate z`\u00001\nt. Figure 2 (left) shows the gating mechanism of the HM-LSTM at time step t.\nFinally, the binary boundary state z`\ntis obtained by:\nz`\nt=fbound(~z`\nt): (8)\nFor the binarization function fbound :R!f0;1g, we can either use a deterministic step function:\nz`\nt=\u001a1if~z`\nt>0:5\n0otherwise;(9)\nor sample from a Bernoulli distribution z`\nt\u0018Bernoulli (~z`"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbf8-6db4-11ee-a647-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-22", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "t>0:5\n0otherwise;(9)\nor sample from a Bernoulli distribution z`\nt\u0018Bernoulli (~z`\nt). Although this binary decision is a key to\nour model, it is usually dif\ufb01cult to use stochastic gradient descent to train such model with discrete\ndecisions as it is not differentiable.\n3.3 C OMPUTING GRADIENT OF BOUNDARY DETECTOR\nTraining neural networks with discrete variables requires more efforts since the standard backpropagation is no longer applicable due to the non-differentiability. Among a few methods for training a\nneural network with discrete variables such as the REINFORCE (Williams, 1992; Mnih & Gregor,\n2014) and the straight-through estimator (Hinton, 2012; Bengio et al., 2013), we use the straightthrough estimator to train our model. The straight-through estimator is a biased estimator because the\nnon-differentiable function used in the forward pass (i.e., the step function in our case) is replaced by\na differentiable function during the backward pass (i.e., the hard sigmoid function in our case). The\nstraight-through estimator, however, is much simpler and often works more ef\ufb01ciently in practice"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbf9-6db4-11ee-98fe-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-23", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "straight-through estimator, however, is much simpler and often works more ef\ufb01ciently in practice\nthan other unbiased but high-variance estimators such as the REINFORCE. The straight-through\nestimator has also been used in Courbariaux et al. (2016) and Vezhnevets et al. (2016).\nThe Slope Annealing Trick . In our experiment, we use the slope annealing trick to reduce the bias\nof the straight-through estimator. The idea is to reduce the discrepancy between the two functions\nused during the forward pass and the backward pass. That is, by gradually increasing the slope aof\nthe hard sigmoid function, we make the hard sigmoid be close to the step function. Note that starting\nwith a high slope value from the beginning can make the training dif\ufb01cult while it is more applicable\nlater when the model parameters become more stable. In our experiments, starting from slope a= 1,\nwe slowly increase the slope until it reaches a threshold with an appropriate scheduling.\n4 E XPERIMENTS\nWe evaluate the proposed model on two tasks, character-level language modelling and handwriting\nsequence generation. Character-level language modelling is a representative example of discrete\n6\nPublished as a conference paper at ICLR 2017\nPenn Treebank\nModel BPC"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbfa-6db4-11ee-9b43-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-24", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "sequence generation. Character-level language modelling is a representative example of discrete\n6\nPublished as a conference paper at ICLR 2017\nPenn Treebank\nModel BPC\nNorm-stabilized RNN (Krueger & Memisevic, 2015) 1.48\nCW-RNN (Koutn\u00edk et al., 2014) 1.46\nHF-MRNN (Mikolov et al., 2012) 1.41\nMI-RNN (Wu et al., 2016) 1.39\nMEn-gram (Mikolov et al., 2012) 1.37\nBatchNorm LSTM (Cooijmans et al., 2016) 1.32\nZoneout RNN (Krueger et al., 2016) 1.27\nHyperNetworks (Ha et al., 2016) 1.27\nLayerNorm HyperNetworks (Ha et al., 2016) 1.23\nLayerNorm CW-RNNy1.40\nLayerNorm LSTMy1.29\nLayerNorm HM-LSTM Sampling 1.27\nLayerNorm HM-LSTM Soft\u00031.27\nLayerNorm HM-LSTM Step Fn. 1.25\nLayerNorm HM-LSTM Step Fn. & Slope Annealing 1.24Hutter Prize Wikipedia\nModel BPC\nStacked LSTM (Graves, 2013) 1.67"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbfb-6db4-11ee-bda9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-25", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Model BPC\nStacked LSTM (Graves, 2013) 1.67\nMRNN (Sutskever et al., 2011) 1.60\nGF-LSTM (Chung et al., 2015) 1.58\nGrid-LSTM (Kalchbrenner et al., 2015) 1.47\nMI-LSTM (Wu et al., 2016) 1.44\nRecurrent Memory Array Structures (Rocki, 2016a) 1.40\nSF-LSTM (Rocki, 2016b)z1.37\nHyperNetworks (Ha et al., 2016) 1.35\nLayerNorm HyperNetworks (Ha et al., 2016) 1.34\nRecurrent Highway Networks (Zilly et al., 2016) 1.32\nLayerNorm LSTMy1.39\nHM-LSTM 1.34\nLayerNorm HM-LSTM 1.32\nPAQ8hp12 (Mahoney, 2005) 1.32\ndecomp8 (Mahoney, 2009) 1.28\nTable 1: BPC on the Penn Treebank test set (left) and Hutter Prize Wikipedia test set (right). ( \u0003) This\nmodel is a variant of the HM-LSTM that does not discretize the boundary detector states. ( y) These\nmodels are implemented by the authors to evaluate the performance using layer normalization (Ba"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbfc-6db4-11ee-8a6a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-26", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "models are implemented by the authors to evaluate the performance using layer normalization (Ba\net al., 2016) with the additional output module. ( z) This method uses test error signals for predicting\nthe next characters, which makes it not comparable to other methods that do not.\nsequence modelling, where the discrete symbols form a distinct hierarchical multiscale structure. The\nperformance on real-valued sequences is tested on the handwriting sequence generation in which a\nrelatively clear hierarchical multiscale structure exists compared to other data such as speech signals.\n4.1 C HARACTER -LEVEL LANGUAGE MODELLING\nA sequence modelling task aims at learning the probability distribution over sequences by minimizing\nthe negative log-likelihood of the training sequences:\nmin\n\u0012\u00001\nNNX\nn=1TnX\nt=1logp(xn\ntjxn\n<t;\u0012); (10)\nwhere\u0012is the model parameter, Nis the number of training sequences, and Tnis the length of the\nn-th sequence. A symbol at time tof sequence nis denoted by xn\nt, andxn\n<tdenotes all previous\nsymbols at time t. We evaluate our model on three benchmark text corpora: (1) Penn Treebank, (2)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbfd-6db4-11ee-b7c1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-27", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "<tdenotes all previous\nsymbols at time t. We evaluate our model on three benchmark text corpora: (1) Penn Treebank, (2)\nText8 and (3) Hutter Prize Wikipedia. We use the bits-per-character (BPC), E[\u0000log2p(xt+1jx\u0014t)],\nas the evaluation metric.\nModel We use a model consisting of an input embedding layer, an RNN module and an output\nmodule. The input embedding layer maps each input symbol into 128-dimensional continuous\nvector without using any non-linearity. The RNN module is the HM-LSTM, described in Section 3,\nwith three layers. The output module is a feedforward neural network with two layers, an output\nembedding layer and a softmax layer. Figure 2 (right) shows a diagram of the output module. At each\ntime step, the output embedding layer receives the hidden states of the three RNN layers as input. In\norder to adaptively control the importance of each layer at each time step, we also introduce three\nscalar gating units g`\nt2Rto each of the layer outputs:\ng`\nt=sigm (w`[h1\nt;\u0001\u0001\u0001;hL\nt]); (11)\nwhere w`2RPL"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbfe-6db4-11ee-ab55-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-28", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "g`\nt=sigm (w`[h1\nt;\u0001\u0001\u0001;hL\nt]); (11)\nwhere w`2RPL\n`=1dim (h`)is the weight parameter. The output embedding he\ntis computed by:\nhe\nt=ReLU LX\n`=1g`\ntWe\n`h`\nt!\n; (12)\nwhereL= 3andReLU (x) = max(0;x)(Nair & Hinton, 2010). Finally, the probability distribution\nfor the next target character is computed by the softmax function, softmax (xj) =exjPK\nk=1exk, where\neach output class is a character.\n7\nPublished as a conference paper at ICLR 2017\nText8\nModel BPC\ntd-LSTM (Zhang et al., 2016) 1.63\nHF-MRNN (Mikolov et al., 2012) 1.54\nMI-RNN (Wu et al., 2016) 1.52\nSkipping-RNN (Pachitariu & Sahani, 2013) 1.48\nMI-LSTM (Wu et al., 2016) 1.44\nBatchNorm LSTM (Cooijmans et al., 2016) 1.36\nHM-LSTM 1.32"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adbff-6db4-11ee-8fdf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-29", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "BatchNorm LSTM (Cooijmans et al., 2016) 1.36\nHM-LSTM 1.32\nLayerNorm HM-LSTM 1.29\nTable 2: BPC on the Text8 test set.\nPenn Treebank We process the Penn Treebank dataset (Marcus et al., 1993) by following the\nprocedure introduced in Mikolov et al. (2012). Each update is done by using a mini-batch of 64\nexamples of length 100to prevent the memory over\ufb02ow problem when unfolding the RNN in time\nfor backpropagation. The last hidden state of a sequence is used to initialize the hidden state of the\nnext sequence to approximate the full backpropagation. We train the model using Adam (Kingma &\nBa, 2014) with an initial learning rate of 0:002. We divide the learning rate by a factor of 50when\nthe validation negative log-likelihood stopped decreasing. The norm of the gradient is clipped with a\nthreshold of 1(Mikolov et al., 2010; Pascanu et al., 2012). We also apply layer normalization (Ba\net al., 2016) to our models. For all of the character-level language modelling experiments, we apply\nthe same procedure, but only change the number of hidden units, mini-batch size and the initial\nlearning rate."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc00-6db4-11ee-9ee5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-30", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "the same procedure, but only change the number of hidden units, mini-batch size and the initial\nlearning rate.\nFor the Penn Treebank dataset, we use 512units in each layer of the HM-LSTM and for the output\nembedding layer. In Table 1 (left), we compare the test BPCs of four variants of our model to other\nbaseline models. Note that the HM-LSTM using the step function for the hard boundary decision\noutperforms the others using either sampling orsoftboundary decision (i.e., hard sigmoid). The test\nBPC is further improved with the slope annealing trick, which reduces the bias of the straight-through\nestimator. We increased the slope awith the following schedule a= min (5 ;1 + 0:04\u0001Nepoch ), where\nNepoch is the maximum number of epochs. The HM-LSTM achieves test BPC score of 1:24. For the\nremaining tasks, we \ufb01xed the hard boundary decision using the step function without slope annealing\ndue to the dif\ufb01culty of \ufb01nding a good annealing schedule on large-scale datasets.\nText8 The Text8 dataset (Mahoney, 2009) consists of 100M characters extracted from the Wikipedia"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc01-6db4-11ee-9f66-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-31", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Text8 The Text8 dataset (Mahoney, 2009) consists of 100M characters extracted from the Wikipedia\ncorpus. Text8 contains only alphabets and spaces, and thus we have total 27 symbols. In order to\ncompare with other previous works, we follow the data splits used in Mikolov et al. (2012). We use\n1024 units for each HM-LSTM layer and 2048 units for the output embedding layer. The mini-batch\nsize and the initial learning rate are set to 128and0:001, respectively. The results are shown in\nTable 2. The HM-LSTM obtains the state-of-the-art test BPC 1:29.\nHutter Prize Wikipedia The Hutter Prize Wikipedia ( enwik8 ) dataset (Hutter, 2012) contains\n205symbols including XML markups and special characters. We follow the data splits used in Graves\n(2013) where the \ufb01rst 90M characters are used to train the model, the next 5M characters for validation,\nand the remainders for the test set. We use the same model size, mini-batch size and the initial\nlearning rate as in the Text8. In Table 1 (right), we show the HM-LSTM achieving the test BPC 1:32,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc02-6db4-11ee-9d58-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-32", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "learning rate as in the Text8. In Table 1 (right), we show the HM-LSTM achieving the test BPC 1:32,\nwhich is a tie with the state-of-the-art result among the neural models. Although the neural models,\nshow remarkable performances, their compression performance is still behind the best models such\nas PAQ8hp12 (Mahoney, 2005) and decomp8 (Mahoney, 2009).\nVisualizing Learned Hierarchical Multiscale Structure In Figure 3 and 4, we visualize the\nboundaries detected by the boundary detectors of the HM-LSTM while reading a character sequence\nof total length 270taken from the validation set of either the Penn Treebank or Hutter Prize Wikipedia\ndataset. Due to the page width limit, the \ufb01gure contains the sequence partitioned into three segments\nof length 90. The white blocks indicate boundaries z`\nt= 1 while the black blocks indicate the\nnon-boundaries z`\nt= 0.\nInterestingly in both \ufb01gures, we can observe that the boundary detector of the \ufb01rst layer, z1, tends\nto be turned on when it sees a space or after it sees a space, which is a reasonable breakpoint to\nseparate between words. This is somewhat surprising because the model self-organizes this structure\n8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc03-6db4-11ee-95c7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-33", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "separate between words. This is somewhat surprising because the model self-organizes this structure\n8\nPublished as a conference paper at ICLR 2017\nFigure 3: Hierarchical multiscale structure in the Wikipedia dataset captured by the boundary\ndetectors of the HM-LSTM.\nPenn Treebank Line 1Penn Treebank Line 2Penn Treebank Line 3\nFigure 4: The `2-norm of the hidden states shown together with the states of the boundary detectors\nof the HM-LSTM.\nwithout any explicit boundary information. In Figure 3, we observe that the z1tends to detect the\nboundaries of the words but also \ufb01res within the words, where the z2tends to \ufb01re when it sees either\nan end of a word or 2;3-grams. In Figure 4, we also see \ufb02ushing in the middle of a word, e.g.,\n\u201ctele-FLUSH-phone\u201d. Note that \u201ctele\u201d is a pre\ufb01x after which a various number of post\ufb01xes can follow.\nFrom these, it seems that the model uses to some extent the concept of surprise to learn the boundary.\nAlthough interpretation of the second layer boundaries is not as apparent as the \ufb01rst layer boundaries,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc04-6db4-11ee-8500-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-34", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Although interpretation of the second layer boundaries is not as apparent as the \ufb01rst layer boundaries,\nit seems to segment at reasonable semantic / syntactic boundaries, e.g., \u201cconsumers may\u201d - \u201cwant to\nmove their telephones a\u201d - \u201clittle closer to the tv set <unk>\u201d, and so on.\nAnother remarkable point is the fact that we do not pose any constraint on the number of boundaries\nthat the model can \ufb01re up. The model, however, learns that it is more bene\ufb01cial to delay the\ninformation ejection to some extent. This is somewhat counterintuitive because it might look more\nbene\ufb01cial to feed the fresh update to the upper layers at every time step without any delay. We\nconjecture the reason that the model works in this way is due to the FLUSH operation that poses an\nimplicit constraint on the frequency of boundary detection, because it contains both a reward (feeding\nfresh information to upper layers) and a penalty (erasing accumulated information). The model \ufb01nds\nan optimal balance between the reward and the penalty.\nTo understand the update mechanism more intuitively, in Figure 4, we also depict the heatmap of the\n`2-norm of the hidden states along with the states of the boundary detectors. As we expect, we can"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc05-6db4-11ee-bedd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-35", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "`2-norm of the hidden states along with the states of the boundary detectors. As we expect, we can\nsee that there is no change in the norm value within segments due to the COPY operation. Also, the\ncolor ofkh1kchanges quickly (at every time step) because there is no COPY operation in the \ufb01rst\nlayer. The color of kh2kchanges less frequently based on the states of z1\ntandz2\nt\u00001. The color of\nkh3kchanges even slowly, i.e., only when z2\nt= 1.\nA notable advantage of the proposed architecture is that the internal process of the RNN becomes\nmore interpretable. For example, we can substitute the states of z1\ntandz2\nt\u00001into Eq. 2 and infer\nwhich operation among the UPDATE, COPY and FLUSH was applied to the second layer at time step\nt. We can also inspect the update frequencies of the layers simply by counting how many UPDATE\nand FLUSH operations were made in each layer. For example in Figure 4, we see that the \ufb01rst layer\nupdates at every time step (which is 270UPDATE operations), the second layer updates 56times,\n9\nPublished as a conference paper at ICLR 2017\nIAM-OnDB\nModel Average Log-Likelihood\nStandard LSTM 1081"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc06-6db4-11ee-a524-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-36", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "9\nPublished as a conference paper at ICLR 2017\nIAM-OnDB\nModel Average Log-Likelihood\nStandard LSTM 1081\nHM-LSTM 1137\nHM-LSTM & Slope Annealing 1167\nTable 3: Average log-likelihood per sequence on the IAM-OnDB test set.\nthe ground truth of pen-tip locationVisualization by segments usingVisualization by segments usingthe states ofz2\nFigure 5: The visualization by segments based on either the given pen-tip location or states of the z2.\nand only 9updates has made in the third layer. Note that, by design, the \ufb01rst layer performs UPDATE\noperation at every time step and then the number of UPDATE operations decreases as the layer level\nincreases. In this example, the total number of updates is 335for the HM-LSTM which is 60% of\nreduction from the 810updates of the standard RNN architecture.\n4.2 H ANDWRITING SEQUENCE GENERATION\nWe extend the evaluation of the HM-LSTM to a real-valued sequence modelling task using IAMOnDB (Liwicki & Bunke, 2005) dataset. The IAM-OnDB dataset consists of 12;179handwriting"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc07-6db4-11ee-94c4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-37", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "examples, each of which is a sequence of (x;y)coordinate and a binary indicator pfor pen-tip\nlocation, giving us (x1:Tn;y1:Tn;p1:Tn), wherenis an index of a sequence. At each time step,\nthe model receives (xt;yt;pt), and the goal is to predict (xt+1;yt+1;pt+1). The pen-up ( pt= 1)\nindicates an end of a stroke, and the pen-down ( pt= 0) indicates that a stroke is in progress. There\nis usually a large shift in the (x;y)coordinate to start a new stroke after the pen-up happens. We\nremove all sequences whose length is shorter than 300. This leaves us 10;465sequences for training,\n581for validation, 582for test. The average length of the sequences is 648. We normalize the range\nof the (x;y)coordinates separately with the mean and standard deviation obtained from the training\nset. We use the mini-batch size of 32, and the initial learning rate is set to 0:0003 .\nWe use the same model architecture as used in the character-level language model, except that the\noutput layer is modi\ufb01ed to predict real-valued outputs. We use the mixture density network as the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc08-6db4-11ee-ab05-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-38", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "output layer is modi\ufb01ed to predict real-valued outputs. We use the mixture density network as the\noutput layer following Graves (2013), and use 400units for each HM-LSTM layer and for the output\nembedding layer. In Table 3, we compare the log-likelihood averaged over the test sequences of the\nIAM-OnDB dataset. We observe that the HM-LSTM outperforms the standard LSTM. The slope\nannealing trick further improves the test log-likelihood of the HM-LSTM into 1167 in our setting. In\nthis experiment, we increased the slope awith the following schedule a= min (3 ;1 + 0:004\u0001Nepoch ).\nIn Figure 5, we let the HM-LSTM to read a randomly picked validation sequence and present the\nvisualization of handwriting examples by segments based on either the states of z2or the states of\npen-tip location3.\n5 C ONCLUSION\nIn this paper, we proposed the HM-RNN that can capture the latent hierarchical structure of the\nsequences. We introduced three types of operations to the RNN, which are the COPY , UPDATE\nand FLUSH operations. In order to implement these operations, we introduced a set of binary\nvariables and a novel update rule that is dependent on the states of these binary variables. Each binary"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc09-6db4-11ee-b4e3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-39", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "and FLUSH operations. In order to implement these operations, we introduced a set of binary\nvariables and a novel update rule that is dependent on the states of these binary variables. Each binary\nvariable is learned to \ufb01nd segments at its level, therefore, we call this binary variable, a boundary\ndetector. On the character-level language modelling, the HM-LSTM achieved state-of-the-art result\non the Text8 dataset and comparable results to the state-of-the-art results on the Penn Treebank\nand Hutter Prize Wikipedia datasets. Also, the HM-LSTM outperformed the standard LSTM on\nthe handwriting sequence generation. Our results and analysis suggest that the proposed HM-RNN\ncan discover the latent hierarchical structure of the sequences and can learn ef\ufb01cient hierarchical\nmultiscale representation that leads to better generalization performance.\n3The plot function could be found at blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/ .\n10\nPublished as a conference paper at ICLR 2017\nACKNOWLEDGMENTS\nThe authors would like to thank Alex Graves, Tom Schaul and Hado van Hasselt for their fruitful\ncomments and discussion. We acknowledge the support of the following agencies for research"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc0a-6db4-11ee-84ac-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-40", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "The authors would like to thank Alex Graves, Tom Schaul and Hado van Hasselt for their fruitful\ncomments and discussion. We acknowledge the support of the following agencies for research\nfunding and computing support: Ubisoft, Samsung, IBM, Facebook, Google, Microsoft, NSERC,\nCalcul Qu\u00e9bec, Compute Canada, the Canada Research Chairs and CIFAR. The authors thank the\ndevelopers of Theano (Team et al., 2016). JC would like to thank Arnaud Bergenon and Fr\u00e9d\u00e9ric\nBastien for their technical support. JC would also like to thank Guillaume Alain, Kyle Kastner and\nDavid Ha for providing us useful pieces of code.\nREFERENCES\nJimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 ,\n2016.\nDzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Yoshua Bengio, et al. End-to-end attention-based large\nvocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal\nProcessing (ICASSP) , pp. 4945\u20134949. IEEE, 2016."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc0b-6db4-11ee-89a8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-41", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal\nProcessing (ICASSP) , pp. 4945\u20134949. IEEE, 2016.\nYoshua Bengio. Learning deep architectures for ai. Foundations and trends R\rin Machine Learning , 2(1):1\u2013127,\n2009.\nYoshua Bengio, Nicholas L\u00e9onard, and Aaron Courville. Estimating or propagating gradients through stochastic\nneurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.\nWilliam Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large\nvocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech\nand Signal Processing (ICASSP) , pp. 4960\u20134964. IEEE, 2016.\nKyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua\nBengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In\nProceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) , October 2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc0c-6db4-11ee-9665-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-42", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) , October 2014.\nJunyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural\nnetworks. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , 2015.\nJunyoung Chung, Kyunghyun Cho, and Yoshua Bengio. A character-level decoder without explicit segmentation\nfor neural machine translation. Association for Computational Linguistics (ACL) , 2016.\nTim Cooijmans, Nicolas Ballas, C\u00e9sar Laurent, and Aaron Courville. Recurrent batch normalization. arXiv\npreprint arXiv:1603.09025 , 2016.\nMatthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural\nnetworks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint\narXiv:1602.02830 , 2016.\nSalah El Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long-term dependencies. In\nAdvances in Neural Information Processing Systems , pp. 493\u2013499. Citeseer, 1995."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc0d-6db4-11ee-b0be-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-43", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Advances in Neural Information Processing Systems , pp. 493\u2013499. Citeseer, 1995.\nSantiago Fern\u00e1ndez, Alex Graves, and J\u00fcrgen Schmidhuber. Sequence labelling in structured domains with\nhierarchical recurrent neural networks. In Proceedings of the 20th international joint conference on Arti\ufb01cal\nintelligence , pp. 774\u2013779. Morgan Kaufmann Publishers Inc., 2007.\nAlex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013.\nDavid Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016.\nG. Hinton. Neural networks for machine learning. Coursera, video lectures, 2012.\nSepp Hochreiter and J\u00fcrgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735\u20131780,\n1997.\nJudy Hoffman, Eric Tzeng, Jeff Donahue, Yangqing Jia, Kate Saenko, and Trevor Darrell. One-shot adaptation\nof supervised deep convolutional models. arXiv preprint arXiv:1312.6204 , 2013."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc0e-6db4-11ee-a6b3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-44", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "of supervised deep convolutional models. arXiv preprint arXiv:1312.6204 , 2013.\nGao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth.\narXiv preprint arXiv:1603.09382 , 2016.\n11\nPublished as a conference paper at ICLR 2017\nMarcus Hutter. The human knowledge compression contest. 2012. URL http://prize.hutter1.net/ .\nNal Kalchbrenner, Ivo Danihelka, and Alex Graves. Grid long short-term memory. arXiv preprint\narXiv:1507.01526 , 2015.\nYoon Kim. Convolutional neural networks for sentence classi\ufb01cation. arXiv preprint arXiv:1408.5882 , 2014.\nYoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models.\narXiv preprint arXiv:1508.06615 , 2015.\nDiederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,\n2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc0f-6db4-11ee-b6f4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-45", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,\n2014.\nDiederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.\nLingpeng Kong, Chris Dyer, and Noah A Smith. Segmental recurrent neural networks. arXiv preprint\narXiv:1511.06018 , 2015.\nJan Koutn\u00edk, Klaus Greff, Faustino Gomez, and J\u00fcrgen Schmidhuber. A clockwork rnn. In Proceedings of the\n31st International Conference on Machine Learning (ICML 2014) , 2014.\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classi\ufb01cation with deep convolutional neural\nnetworks. In Advances in Neural Information Processing Systems , pp. 1097\u20131105, 2012.\nDavid Krueger and Roland Memisevic. Regularizing rnns by stabilizing activations. arXiv preprint\narXiv:1511.08400 , 2015."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc10-6db4-11ee-bef8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-46", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "David Krueger and Roland Memisevic. Regularizing rnns by stabilizing activations. arXiv preprint\narXiv:1511.08400 , 2015.\nDavid Krueger, Tegan Maharaj, J\u00e1nos Kram\u00e1r, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh\nGoyal, Yoshua Bengio, Hugo Larochelle, Aaron Courville, et al. Zoneout: Regularizing rnns by randomly\npreserving hidden activations. arXiv preprint arXiv:1606.01305 , 2016.\nYann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436\u2013444, 2015.\nTsungnan Lin, Bill G Horne, Peter Tino, and C Lee Giles. Learning long-term dependencies in narx recurrent\nneural networks. IEEE Transactions on Neural Networks , 7(6):1329\u20131338, 1996.\nWang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. Character-based neural machine translation. arXiv\npreprint arXiv:1511.04586 , 2015.\nMarcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc11-6db4-11ee-88c8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-47", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten\ntext on a whiteboard. In Eighth International Conference on Document Analysis and Recognition (ICDAR\u201905) ,\npp. 956\u2013961. IEEE, 2005.\nMatthew V Mahoney. Adaptive weighing of context models for lossless data compression. 2005.\nMatthew V Mahoney. Large text compression benchmark. URL: http://www. mattmahoney. net/text/text. html ,\n2009.\nMitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of\nenglish: The penn treebank. Computational linguistics , 19(2):313\u2013330, 1993.\nTomas Mikolov, Martin Kara\ufb01\u00e1t, Lukas Burget, Jan Cernock `y, and Sanjeev Khudanpur. Recurrent neural\nnetwork based language model. In INTERSPEECH , volume 2, pp. 3, 2010.\nTomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and J Cernocky. Subword language modeling with neural networks. Preprint , 2012. URL http://www.fit.vutbr.cz/"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc12-6db4-11ee-8c1a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-48", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "~imikolov/rnnlm/char.pdf .\nAndriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In Proceedings of\nthe 31st International Conference on Machine Learning (ICML-14) , pp. 1791\u20131799, 2014.\nMichael C Mozer. Induction of multiscale temporal structure. Advances in neural information processing\nsystems , pp. 275\u2013275, 1993.\nVinod Nair and Geoffrey E Hinton. Recti\ufb01ed linear units improve restricted boltzmann machines. In Proceedings\nof the 27th International Conference on Machine Learning (ICML-10) , pp. 807\u2013814, 2010.\nMarius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are\nthey needed? arXiv preprint arXiv:1301.5650 , 2013.\n12\nPublished as a conference paper at ICLR 2017\nRazvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the dif\ufb01culty of training recurrent neural networks.\narXiv preprint arXiv:1211.5063 , 2012.\nKamil M Rocki. Recurrent memory array structures. arXiv preprint arXiv:1607.03085 , 2016a."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc13-6db4-11ee-a544-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-49", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Kamil M Rocki. Recurrent memory array structures. arXiv preprint arXiv:1607.03085 , 2016a.\nKamil M Rocki. Surprisal-driven feedback in recurrent networks. arXiv preprint arXiv:1608.06027 , 2016b.\nJ\u00fcrgen Schmidhuber. Neural sequence chunkers. 1991.\nJ\u00fcrgen Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural\nComputation , 4(2):234\u2013242, 1992.\nJ\u00fcrgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks , 61:85\u2013117, 2015.\nAlessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie.\nA hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of\nthe 24th ACM International on Conference on Information and Knowledge Management , pp. 553\u2013562. ACM,\n2015.\nNitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout:"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc14-6db4-11ee-a1e6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-50", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout:\na simple way to prevent neural networks from over\ufb01tting. Journal of Machine Learning Research , 15(1):\n1929\u20131958, 2014.\nIlya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In\nProceedings of the 28th International Conference on Machine Learning (ICML\u201911) , pp. 1017\u20131024, 2011.\nIlya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances\nin Neural Information Processing Systems , pp. 3104\u20133112, 2014.\nThe Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller,\nDzmitry Bahdanau, Nicolas Ballas, Fr\u00e9d\u00e9ric Bastien, Justin Bayer, Anatoly Belikov, et al. Theano: A python\nframework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 , 2016."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc15-6db4-11ee-bcc0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-51", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 , 2016.\nAlexander Vezhnevets, V olodymyr Mnih, John Agapiou, Simon Osindero, Alex Graves, Oriol Vinyals, Koray\nKavukcuoglu, et al. Strategic attentive writer for learning macro-actions. arXiv preprint arXiv:1606.04695 ,\n2016.\nOriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption\ngenerator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 3156\u2013\n3164, 2015.\nRonald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.\nMachine learning , 8(3-4):229\u2013256, 1992.\nYuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan Salakhutdinov. On multiplicative\nintegration with recurrent neural networks. arXiv preprint arXiv:1606.06630 , 2016.\nSaizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdinov, and Yoshua"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc16-6db4-11ee-bf4e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.01704-52", "title": "Hierarchical Multiscale Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1609.01704", "start_index": 0, "chroma:document": "Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdinov, and Yoshua\nBengio. Architectural complexity measures of recurrent neural networks. arXiv preprint arXiv:1602.08210 ,\n2016.\nJulian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutn\u00edk, and J\u00fcrgen Schmidhuber. Recurrent highway\nnetworks. arXiv preprint arXiv:1607.03474 , 2016.\n13"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc17-6db4-11ee-a19e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-0", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 1\nBottom-up Instance Segmentation using\nDeep Higher-Order CRFs\nAnurag Arnab\nanurag.arnab@eng.ox.ac.uk\nPhilip H.S. Torr\nphilip.torr@eng.ox.ac.ukDepartment of Engineering Science\nUniversity of Oxford\nUnited Kingdom\nAbstract\nTraditional Scene Understanding problems such as Object Detection and Semantic\nSegmentation have made breakthroughs in recent years due to the adoption of deep learning. However, the former task is not able to localise objects at a pixel level, and the latter\ntask has no notion of different instances of objects of the same class. We focus on the task\nof Instance Segmentation which recognises and localises objects down to a pixel level.\nOur model is based on a deep neural network trained for semantic segmentation. This\nnetwork incorporates a Conditional Random Field with end-to-end trainable higher order potentials based on object detector outputs. This allows us to reason about instances\nfrom an initial, category-level semantic segmentation. Our simple method effectively\nleverages the great progress recently made in semantic segmentation and object detection. The accurate instance-level segmentations that our network produces is re\ufb02ected by\nthe considerable improvements obtained over previous work at high APrIoU thresholds."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc18-6db4-11ee-ad09-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-1", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "the considerable improvements obtained over previous work at high APrIoU thresholds.\n1 Introduction\nObject detection and semantic segmentation have been two of the most popular Scene Understanding problems within the Computer Vision community. Great advances have been made\nin recent years due to the adoption of deep learning and Convolutional Neural Networks\n[19, 22, 32]. In this paper, we focus on the problem of Instance Segmentation . Instance\nSegmentation lies at the intersection of Object Detection \u2013 which localises different objects\nat a bounding box level, but does not segment them \u2013 and Semantic Segmentation \u2013 which\ndetermines the object-class label of each pixel in the image, but has no notion of different\ninstances of the same class. As shown in Figure 1, the task of instance segmentation localises\nobjects to a pixel level.\nMany recent instance segmentation works have built on the \u201cSimultaneous Detection\nand Segmentation\u201d (SDS) approach of Hariharan et al. [13]. These methods [4, 5, 13, 14]\ngenerate object proposals [1], classify each proposal into an object category, and then re\ufb01ne\nthe bounding box proposal into a segmentation of the primary object the proposal contains.\nHowever, because these methods localise an object before segmenting it, they are restricted"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc19-6db4-11ee-a28a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-2", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "the bounding box proposal into a segmentation of the primary object the proposal contains.\nHowever, because these methods localise an object before segmenting it, they are restricted\nby the quality of the initial proposal. Moreover, the proposal generator [1] these works used\ntakes about 40 seconds to process an image.\nc/circlecopyrt2016. The copyright of this document resides with its authors.\nIt may be distributed unchanged freely in print or electronic forms.\narXiv:1609.02583v1  [cs.CV]  8 Sep 2016\n2 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION\n(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation\nFigure 1: Our method uses the results of an object detector and the original image (a) to produce a category-level semantic segmentation (b). The semantic segmentation result, object\ndetection bounding boxes, and recalibrated detection scores (\u00a73.2) are then used to re\ufb01ne\nthe semantic segmentation into an instance segmentation (c). The semantic segmentation,\nin which a large number of pixels have been correctly labelled as \u201cperson\u201d, does not tell us"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc1a-6db4-11ee-93f9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-3", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "the semantic segmentation into an instance segmentation (c). The semantic segmentation,\nin which a large number of pixels have been correctly labelled as \u201cperson\u201d, does not tell us\nhow many people there are and where each person is in the image. The instance segmentation answers these questions. Also note how our method is not affected by the false-positive\n\u201cbottle\u201d detections.\nWe present a different approach to instance segmentation, where we initially perform\na category-level, semantic segmentation of the image before reasoning about the different\ninstances in the scene. We can identify instances because our semantic segmentation network\nincorporates a Higher Order Conditional Random Field (CRF) which uses cues from the\noutput of an object detector. This CRF is fully differentiable and can be inserted as a layer\nof a neural network [2, 37]. As a result, it can be used as a simple extension to an existing\nsemantic segmentation network to perform the related task of instance segmentation.\nOur simple, bottom-up method is able to effectively leverage the progress made by stateof-the-art semantic segmentation and object detection networks to perform the related task\nof instance segmentation. We show this by evaluating our approach on the PASCAL VOC\n2012 [8] dataset, using the standard APrmeasure [13]. The fact that our system is able to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc1b-6db4-11ee-b875-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-4", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "2012 [8] dataset, using the standard APrmeasure [13]. The fact that our system is able to\nproduce accurate instance segmentations is re\ufb02ected by the considerable improvements that\nwe make over previous work at high overlap thresholds.\n2 Related Work\nInstance segmentation is closely related to the widely-studied problems of semantic segmentation and object detection. We thus review these two tasks \ufb01rst.\nSemantic Segmentation: This task was traditionally performed by extracting dense features from an image and classifying them on a per-pixel or per-region basis [16, 31]. The\nindividual predictions made by these classi\ufb01ers were often noisy as they lacked global context. As a result, they were post-processed with a CRF whose priors encourages nearby\npixels, and pixels of similar appearance, to share the same class label. The CRF of [31]\ninitially contained only unary terms (obtained from the classi\ufb01er) and 8-connected pairwise\nterms. Subsequent improvements included introducing densely-connected pairwise potentials [18], using higher order potentials encouraging consistency over cliques larger than two\npixels [17, 35], modelling the co-occurrence of labels [20, 35] and utilising the results of\nobject detectors [21, 33, 36]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc1c-6db4-11ee-b34a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-5", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "pixels [17, 35], modelling the co-occurrence of labels [20, 35] and utilising the results of\nobject detectors [21, 33, 36].\nUsing CNNs for semantic segmentation can greatly improve the unary, per-pixel predictions, as shown by [27]. Chen et al. showed further improvements by post-processing the\nARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 3\nCNN output with a CRF [3]. Current state-of-the-art methods [2, 25, 37] incorporate inference of a CRF as layers within a deep network which perform the differentiable mean \ufb01eld\ninference algorithm. In our work, we extend the end-to-end trainable, higher-order, object\ndetection potentials proposed by [2] to the task of instance segmentation. Additionally, unlike the aforementioned works which always assume a \ufb01xed number of labels in the CRF at\nall times, our system includes a CRF where the number of labels varies per input image.\nObject Detection: Part-based models [9] were popular before CNNs, which had been\nshown to excel in the task of image classi\ufb01cation [19, 32], were adapted to object detection. Recent detection algorithms are based on using CNNs to classify object proposals"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc1d-6db4-11ee-a74a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-6", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "shown to excel in the task of image classi\ufb01cation [19, 32], were adapted to object detection. Recent detection algorithms are based on using CNNs to classify object proposals\nand then performing bounding box regression to re\ufb01ne the initial proposals\u2019 bounding boxes\n[10, 11, 12, 28]. This pipeline can be accelerated by computing convolutional features over\nthe whole image, and then pooling features for each proposal [11, 15]. Object proposals\nwere initially hand-crafted [1, 34], but this step has now been integrated into CNNs [28].\nInstance Segmentation: This task was popularised by Hariharan et al. [13]. Their approach was similar to object detection pipelines in that region proposals were \ufb01rst generated and classi\ufb01ed into different object categories before using bounding box regression as\npost-processing. A class-speci\ufb01c segmentation was then performed to simultaneously detect\nand segment an object. However, approaches that segment instances by re\ufb01ning detections\n[4, 5, 13, 14] are inherently limited by the quality of the initial proposals, and the systems\ncannot recover from errors made by the initial detector. Furthermore, since these methods\ngenerate and classify multiple overlapping region proposals per image, they cannot actually"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc1e-6db4-11ee-afac-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-7", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "cannot recover from errors made by the initial detector. Furthermore, since these methods\ngenerate and classify multiple overlapping region proposals per image, they cannot actually\nproduce segmentation maps of the form shown in Figures 1(b) and (c) [24]. Liang et al. [23]\naddress some of these shortcomings by iteratively re\ufb01ning initial object proposals, whilst [7]\ngenerates box-proposals, creates masks from these proposals and then classi\ufb01es these masks\nin a single network trained with three loss functions summed together.\nA proposal-free method was recently presented by [24] where a semantic segmentation of the image is \ufb01rst performed using the network of [3]. Thereafter the category-level\nsegmentation output, along with CNN features, are used to predict instance-level bounding\nboxes for the objects in the image. The number of instances of each object category are also\npredicted in order to facilitate the \ufb01nal clustering step. An alternate method was proposed\nby [29] where a Recurrent Neural Network outputs an object instance at each time step. This\nmethod, however, has not been evaluated on multiple classes."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc1f-6db4-11ee-8cf6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-8", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "by [29] where a Recurrent Neural Network outputs an object instance at each time step. This\nmethod, however, has not been evaluated on multiple classes.\nOur method also \ufb01rst performs a bottom-up, semantic segmentation of the image. However, our semantic segmentation network uses the outputs of an object detector as an additional cue in its \ufb01nal CRF layer. During inference, each object hypothesis from the detector\nis evaluated in light of other energies in the CRF. Thus, the relative score of false positive\ndetections can be decreased, and correct detections increased. These rescored detections\nare then used to identify instances from the initial semantic segmentation. The fact that our\nsystem combines information from semantic segmentation and object detection networks allows us to bypass the complexity of the \u201cinstance bounding box\u201d and \u201cinstance number\u201d\nprediction steps used by [24].\n3 Proposed Approach\nOur proposed method \ufb01rst performs a semantic segmentation of the input image, classifying\neach pixel into one of K+1 categories where Kis the number of foreground classes. The\nresulting semantic segmentation is then re\ufb01ned into an instance-level segmentation, where\n4 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION\nPixelwise CNN  Higher Order CRF"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc20-6db4-11ee-8387-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-9", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "4 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION\nPixelwise CNN  Higher Order CRF  \nInstance Identification  Instance CRF  \nK+1  \nObject Detector  \nD+1  End-to-end object segmentation subnetwork  \nEnd-to-end Instance segmentation subnetwork  \nInput  \nOutput  Object  \nSegmentation  \nIdentified  \nInstances  \nFigure 2: Overview of our end-to-end method. Our system \ufb01rst computes a category-level\nsemantic segmentation of the image. A CRF with higher-order detection potentials is used\nto obtain this result in a semantic segmentation subnetwork. This results in a W\u00d7H\u00d7(K+\n1)dimensional volume where Kis the number of foreground classes. WandHare the\nimage\u2019s width and height respectively. The original detections and the detector con\ufb01dences\nrecalibrated by the Higher Order CRF are used to identify instances in the image, producing\naW\u00d7H\u00d7(D+1)dimensional volume where Dis the number of detections (variable per\nimage). Thereafter, another Instance CRF is used to compute the \ufb01nal result."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc21-6db4-11ee-9e12-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-10", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "image). Thereafter, another Instance CRF is used to compute the \ufb01nal result.\nthe object class of each instance segment is obtained from the previous semantic segmentation. Both of these stages, while conceptually different, are fully differentiable and the entire\nsystem can be implemented as a neural network.\nFigure 2 shows our category-level segmentation network, consisting of a pixelwise CNN\n[27] followed by a fully differentiable CRF with object detection potentials, as described in\nSection 3.2. The object detection results that our CRF takes in, as well as the recalibrated\ndetection scores that it outputs, allows us to identify object instances, and obtain a \ufb01nal\ninstance-level segmentation in the next stage of our pipeline. This is described in Section\n3.3. However, we \ufb01rst review Conditional Random Fields and introduce the notation used in\nthis paper.\n3.1 Conditional Random Fields\nAssume an image Iwith Npixels, indexed 1 ,2...N, and de\ufb01ne a set of random variables,\nX1,X2,...,XN, one for every pixel. In a generic labelling problem, we assign every pixel a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc22-6db4-11ee-8be2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-11", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "X1,X2,...,XN, one for every pixel. In a generic labelling problem, we assign every pixel a\nlabel from a prede\ufb01ned set of labels Lsuch that each Xi\u2208L. Let X= [X1X2...XN]T. In\nthis case, any particular assignment xtoXis a solution to our labelling problem.\nIn the case of semantic segmentation, we assign each pixel, Xi, a label corresponding\nto object categories such as \u201cperson\u201d and \u201ccar\u201d. In instance segmentation, the labels are\ndrawn from the product label space of object categories and instance numbers. Examples are\n\u201cperson_1\u201d and \u201cperson_2\u201d as shown by the different colours in Figure 1.\nGiven a graph Gwhere the vertices are from {X}and connections between these variables are de\ufb01ned by edges, the pair (I,X)is modelled as a CRF de\ufb01ned by Pr (X=x|I) =\n(1/Z(I))exp(\u2212E(x|I)). The term, E(x|I)is the energy of the labelling xandZ(I)is the\nARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc23-6db4-11ee-b0de-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-12", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 5\ndata-dependent partition function. From here onwards, we drop the conditioning on Ito\nkeep the notation uncluttered. The energy of an assignment is de\ufb01ned over the set of cliques\nCin the graph Gas\nE(x) =\u2211\nc\u2208C\u03c8c(xc), (1)\nwhere xcis the vector formed by selecting elements of xthat correspond to random variables\nbelonging to the clique c, and\u03c8c(.)is the cost function for a clique c.\nThe energy function considered by many semantic segmentation works, E(x) =\u2211i\u03c8U\ni(xi)+\n\u2211i<j\u03c8P\ni j(xi,xj), consists of only unary (cliques of size one) and densely-connected pairwise\npotentials (cliques of size two) [3, 18, 37]. When dense pairwise potentials are used in a\nCRF to obtain higher accuracy, exact inference is not tractable and approximate inference\nmethods such as mean \ufb01eld are used [18].\n3.2 Higher Order Detection Potentials\nIn addition to the standard unary and densely-connected pairwise terms, we also include a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc24-6db4-11ee-aab1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-13", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "3.2 Higher Order Detection Potentials\nIn addition to the standard unary and densely-connected pairwise terms, we also include a\nhigher order term based on object detections. Intuitively, a state-of-the-art object detector\n[11, 28] can help segmentation performance in cases where the detector correctly predicts an\nobject at a location in which the pixels have been misclassi\ufb01ed by the unary classi\ufb01er, and\nthus reduce the recognition error [6]. Arnab et al. [2] formulated object detection potentials\nin a manner that is amenable to the differentiable mean \ufb01eld inference algorithm so that it\ncould be integrated into a neural network, and showed that it does indeed improve semantic\nsegmentation performance. We review the detection potentials described in [2] as they also\nhelp us to reason about instances in an input image, a problem not considered by [2].\nAssume that we have Dobject detections for an input image (this number varies for every\nimage), and that the dthdetection is of the form (ld,sd,Fd,Bd)where ld\u2208Lis the object class\nlabel of the detected object, sdis the detector\u2019s con\ufb01dence score, Fd\u2282{1,2,...N}is the set"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc25-6db4-11ee-9b3b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-14", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "label of the detected object, sdis the detector\u2019s con\ufb01dence score, Fd\u2282{1,2,...N}is the set\nof indices of the pixels belonging to the foreground of the detection and Bd\u2282{1,2,...N}is\nthe set of indices falling in the detection\u2019s bounding box. The foreground is obtained from\na foreground/background segmentation method such as GrabCut [30] and indicates a rough\nsegmentation of the detected object. The detection potentials should encourage the set of\npixels represented by Fdto take the object class label, ld. However, this should be a soft\nconstraint, since the foreground segmentation could be inaccurate and the entire detection\nitself could be a false-positive. The detection dshould also be identi\ufb01ed as invalid if other\nenergies in the CRF strongly suggest that many pixels in Fddo not take the label ld.\nThis is formulated by introducing a latent binary random variable, Y1,Y2...YDfor every\ndetection. If the dthdetection has been found to be valid after inference, Ydwill be set to 1,\nand 0 otherwise. The \ufb01nal value of Ydis determined probabilistically by mean \ufb01eld inference."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc26-6db4-11ee-ab83-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-15", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "and 0 otherwise. The \ufb01nal value of Ydis determined probabilistically by mean \ufb01eld inference.\nAll latent Ydvariables are added to the CRF which previously only contained Xivariables.\nLet each (Xd,Yd), where{Xd}={Xi\u2208{X}|i\u2208Fd}, form a clique in the CRF. An assignment\n(xd,yd)to the clique (Xd,Yd)has the energy:\n\u03c8Det\nd(Xd=xd,Yd=yd) =\uf8f1\n\uf8f4\uf8f2\n\uf8f4\uf8f3wlsd\n|Fd|\u2211|Fd|\ni=1[x(i)\nd=ld]ifyd=0,\nwlsd\n|Fd|\u2211|Fd|\ni=1[x(i)\nd/negationslash=ld]ifyd=1,(2)\nwhere x(i)\ndis the ithelement in vector xd,[.]is the Iverson bracket and wlis a class-speci\ufb01c,\nlearnable weight parameter. This potential encourages consistency among X(i)\ndvariables and\n6 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc27-6db4-11ee-87f0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-16", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "learnable weight parameter. This potential encourages consistency among X(i)\ndvariables and\n6 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION\nYdsince it encourages X(i)\ndvariables to take the label ldwhen Ydis 1, and also encourages Yd\nto be 0 when many X(i)\ndvariables do not take the label ld.\nA unary potential for the latent Yvariables is also included, which is initialised to the\ncon\ufb01dence score sdof the object detector (in addition to the joint potential with the Xvariables). This initial con\ufb01dence then changes throughout CRF inference, depending on the\nother potentials. The \ufb01nal probability of the latent Yvariable can thus be viewed as a recalibrated detection score, which gets adjusted during inference based on how much it agrees\nwith the segmentation unaries and pairwise potentials. Note that this \u201crecalibration property\u201d\nof the latent Yvariables was not considered by [2]. The \ufb01nal form of the energy function\n(Eq 1), is thus\nE(x) =\u2211\ni\u03c8U\ni(xi)+\u2211\ni<j\u03c8P\ni,j(xi,xj)+\u2211\nd\u03c8Det"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc28-6db4-11ee-84bc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-17", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "i\u03c8U\ni(xi)+\u2211\ni<j\u03c8P\ni,j(xi,xj)+\u2211\nd\u03c8Det\nd(xd,yd)+\u2211\nd\u03c8U\nd(yd). (3)\n3.3 Instance Identi\ufb01cation and Re\ufb01nement\nOnce we have a category-level segmentation of the image, each pixel still needs to be assigned to an object instance. We assume that each object detection represents a possible\ninstance. Since there are Dobject detections (where Dvaries for every image), and some\npixels do not belong to any object instance, but are part of the background, we have a labelling problem involving D+1 labels. Our set of labels, D, is thus{1,2,...D+1}.\nA na\u00efve solution to the problem of recovering instances from a category-level segmentation would be as follows: If a pixel falls within the bounding box of a detection, and its\ncategory label agrees with the class predicted by the object detector, then it is assigned to the\ninstance represented by that detection. However, this method cannot deal with overlapping\nbounding boxes which are typical when objects occlude one another (as in Figs. 1 and 2).\nInstead, if a pixel falls within the bounding box of a detection, we assign the pixel to"}


