# embeddings_queue

created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a6748-6db4-11ee-af3c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-51", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "Art: Although focused in providing segmented object\nproposals, MCG may also be used to provide boxes\nproposals, by taking the bounding box of the segmented\nproposals and deduplicating them. We add the state\nof the art in boxes proposals [26], [27], [25], and [16]\nto the comparison. Figure 12 shows the recall values\nof the boxes results when compared to the bounding\nboxes of the annotated objects, for three different Jaccard\nthresholds.\nWhile many of the techniques speci\ufb01cally tailored to\nboxes proposals are competitive at J= 0:5, their perfor-mance drops signi\ufb01cantly at higher Jaccard thresholds.\nDespite being tailored to segmented proposals, MCG\nclearly outperforms the state of the art if you look\nfor precise localization of the bounding boxes. Again,\nSCG is very competitive, especially taking its speed into\naccount.\nMCG and SCG Time per Image : Table 3 shows\nthe time per image of the main steps of our approach,\nfrom the raw image to the contours, the hierarchical\nsegmentation, and the proposal generation. All times are\ncomputed using a single core on a Linux machine. Our\nfull MCG takes about 25 s. per image to compute the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a6749-6db4-11ee-aefc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-52", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "segmentation, and the proposal generation. All times are\ncomputed using a single core on a Linux machine. Our\nfull MCG takes about 25 s. per image to compute the\nmultiscale hierarchies, and 17 s. to generate and rank\nthe5 038 proposals on the validation set of SegVOC12.\nOur single-scale SCG produces a segmentation hierarchy\nof better quality than gPb-ucm [20] in less than 3 seconds\nand with signi\ufb01cant less memory footprint.\n12\nFig. 13. COCO Qualitative Results : Image, ground truth, multi-scale UCM and best MCG proposals among the 500\nbest ranked. (More qualitative examples in the supplemental material.)\nContour Hierarchical CandidateTotalDetection Segmentation Generation\nMCG 4.6\u00061.3 20.5\u00065.6 17.0\u00069.8 42.2\u000614.8\nSCG 1.0\u00060.3 2.7\u00060.7 2.6\u00060.4 6.2\u00061.1\nTABLE 3\nTime in seconds per image of MCG and SCG\nTable 4 shows the time-per-image results compared to\nthe rest of state of the art in segmented proposals generation, all run on the same single-core Linux machine.\nProposal\nGeneration"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a674a-6db4-11ee-a0db-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-53", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "the rest of state of the art in segmented proposals generation, all run on the same single-core Linux machine.\nProposal\nGeneration\nMCG (Our) 42.2\u000614.8\nSCG (Our) 6.2\u00061.1\nGOP [13] 1.0\u00060.3\nGLS [14] 7.9\u00061.7\nSeSe [12] 15.9\u00065.2\nRIGOR [15] 31.6\u000616.0\nCPMC [18] \u0015120\nCI [19] \u0015120\nShSh [17] \u0015120\nTABLE 4\nTime comparison for all considered state-of-the-art\ntechniques that produce segmented proposals. All run\non the same single-core Linux machine.Practical considerations : One of the key aspects of\nobject proposals is the size of the pool they generate.\nDepending on the application, one may need more precision and thus a bigger pool, or one might need speed and\nthus a small pool in exchange for some loss of quality.\nMCG and SCG provide a ranked set of around 5 000 and\n2 000 proposals, respectively, and one can take the N\n\ufb01rst in case the speci\ufb01c application needs a smaller pool.\nFrom a practical point of view, this means that one does"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a674b-6db4-11ee-9b37-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-54", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "\ufb01rst in case the speci\ufb01c application needs a smaller pool.\nFrom a practical point of view, this means that one does\nnot need to re-parameterize them for the speci\ufb01c needs\nof a certain application.\nIn contrast, the techniques that do not provide a\nranking of the proposals, need to be re-parameterized\nto adapt them to a different number of proposals, which\nis not desirable in practice.\nOn top of that, the results show that MCG and SCG\nhave outstanding generalization power to unseen images\n(recall that the results for SBD and COCO have been\nobtained using the learnt parameters on SegVOC12),\nmeaning that MCG and SCG offer the best chance to\nobtain competitive results in an unseen database without\nneed to re-train.\nFigure 13 shows some qualitative results on COCO.\n7 C ONCLUSIONS\nWe proposed Multiscale Combinatorial Grouping\n(MCG), a uni\ufb01ed approach for bottom-up segmentation\nand object proposal generation. Our approach produces\n13\nstate-of-the-art contours, hierarchical regions, and object\nproposals. At its core are a fast eigenvector computation\nfor normalized-cut segmentation and an ef\ufb01cient\nalgorithm for combinatorial merging of hierarchical"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a674c-6db4-11ee-bedd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-55", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "proposals. At its core are a fast eigenvector computation\nfor normalized-cut segmentation and an ef\ufb01cient\nalgorithm for combinatorial merging of hierarchical\nregions. We also present Single-scale Combinatorial\nGrouping (SCG), a speeded up version of our technique\nthat produces competitive results in under \ufb01ve seconds\nper image.\nWe perform an extensive validation in BSDS500,\nSegVOC12, SBD, and COCO, showing the quality, robustness and scalability of MCG. Recently, an independent study [45], [46] provided further evidence to the\ninterest of MCG among the current state-of-the-art in\nobject proposal generation. Moreover, our object candidates have already been employed as integral part of\nhigh performing recognition systems [47].\nIn order to promote reproducible research on perceptual grouping, all the resources of this project \u2013 code,\npre-computed results, and evaluation protocols \u2013 are\npublicly available2.\nAcknowledgements : The last iterations of this work\nhave been done while Jordi Pont-Tuset has been at Prof.\nLuc Van Gool\u2019s Computer Vision Lab (CVL) of ETHZ,\nSwitzerland. This work has been partly developed in the\nframework of the project BIGGRAPH-TEC2013-43935-R"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a674d-6db4-11ee-8991-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-56", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "Switzerland. This work has been partly developed in the\nframework of the project BIGGRAPH-TEC2013-43935-R\nand the FPU grant AP2008-01164; \ufb01nanced by the Spanish Ministerio de Econom\u00b4 \u0131a y Competitividad , and the European Regional Development Fund (ERDF). This work\nwas partially supported by ONR MURI N000141010933.\nREFERENCES\n[1] P . Viola and M. Jones, \u201cRobust real-time face detection,\u201d IJCV ,\nvol. 57, no. 2, 2004.\n[2] N. Dalal and B. Triggs, \u201cHistograms of oriented gradients for\nhuman detection,\u201d in CVPR , 2005.\n[3] P . Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan,\n\u201cObject detection with discriminatively trained part based models,\u201d TP AMI , vol. 32, no. 9, 2010.\n[4] T. Malisiewicz and A. A. Efros, \u201cImproving spatial support for\nobjects via multiple segmentations,\u201d in BMVC , 2007.\n[5] C. Gu, J. Lim, P . Arbelaez, and J. Malik, \u201cRecognition using"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a674e-6db4-11ee-adba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-57", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "[5] C. Gu, J. Lim, P . Arbelaez, and J. Malik, \u201cRecognition using\nregions,\u201d in CVPR , 2009.\n[6] J. Carreira, F. Li, and C. Sminchisescu, \u201cObject recognition by\nsequential \ufb01gure-ground ranking,\u201d IJCV , vol. 98, no. 3, pp. 243\u2013\n262, 2012.\n[7] A. Ion, J. Carreira, and C. Sminchisescu, \u201cProbabilistic joint image\nsegmentation and labeling by \ufb01gure-ground composition,\u201d IJCV ,\nvol. 107, no. 1, pp. 40\u201357, 2014.\n[8] P . Arbel \u00b4aez, B. Hariharan, C. Gu, S. Gupta, L. Bourdev, and\nJ. Malik, \u201cSemantic segmentation using regions and parts,\u201d in\nCVPR , 2012.\n[9] J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu, \u201cSemantic\nsegmentation with second-order pooling,\u201d in ECCV , 2012."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d90-6db4-11ee-a1b7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-58", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "segmentation with second-order pooling,\u201d in ECCV , 2012.\n[10] R. Girshick, J. Donahue, T. Darrell, and J. Malik, \u201cRich feature\nhierarchies for accurate object detection and semantic segmentation,\u201d in CVPR , 2014.\n[11] N. Zhang, J. Donahue, R. Girshick, and T. Darrell, \u201cPart-based\nr-cnns for \ufb01ne-grained category detection,\u201d in ECCV , 2014.\n[12] J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M.\nSmeulders, \u201cSelective search for object recognition,\u201d IJCV , vol.\n104, no. 2, pp. 154\u2013171, 2013.\n2.www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/mcg/[13] P . Kr \u00a8ahenb \u00a8uhl and V . Koltun, \u201cGeodesic object proposals,\u201d in\nECCV , 2014.\n[14] P . Rantalankila, J. Kannala, and E. Rahtu, \u201cGenerating object"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d91-6db4-11ee-a196-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-59", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "ECCV , 2014.\n[14] P . Rantalankila, J. Kannala, and E. Rahtu, \u201cGenerating object\nsegmentation proposals using global and local search,\u201d in CVPR ,\n2014.\n[15] A. Humayun, F. Li, and J. M. Rehg, \u201cRIGOR: Recycling Inference\nin Graph Cuts for generating Object Regions,\u201d in CVPR , 2014.\n[16] B. Alexe, T. Deselaers, and V . Ferrari, \u201cMeasuring the objectness\nof image windows,\u201d TP AMI , vol. 34, pp. 2189\u20132202, 2012.\n[17] J. Kim and K. Grauman, \u201cShape sharing for object segmentation,\u201d\ninECCV , 2012.\n[18] J. Carreira and C. Sminchisescu, \u201cCPMC: Automatic object\nsegmentation using constrained parametric min-cuts,\u201d TP AMI ,\nvol. 34, no. 7, pp. 1312\u20131328, 2012.\n[19] I. Endres and D. Hoiem, \u201cCategory-independent object proposals\nwith diverse ranking,\u201d TP AMI , vol. 36, no. 2, pp. 222\u2013234, 2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d92-6db4-11ee-8a15-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-60", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "with diverse ranking,\u201d TP AMI , vol. 36, no. 2, pp. 222\u2013234, 2014.\n[20] P . Arbel \u00b4aez, M. Maire, C. C. Fowlkes, and J. Malik, \u201cContour\ndetection and hierarchical image segmentation,\u201d TP AMI , vol. 33,\nno. 5, pp. 898\u2013916, 2011.\n[21] P . F. Felzenszwalb and D. P . Huttenlocher, \u201cEf\ufb01cient graph-based\nimage segmentation,\u201d IJCV , vol. 59, p. 2004, 2004.\n[22] X. Ren and L. Bo, \u201cDiscriminatively trained sparse code gradients\nfor contour detection,\u201d in NIPS , 2012.\n[23] C. J. Taylor, \u201cTowards fast and accurate segmentation,\u201d CVPR ,\n2013.\n[24] M. Maire and S. X. Yu, \u201cProgressive multigrid eigensolvers for\nmultiscale spectral segmentation,\u201d ICCV , 2013."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d93-6db4-11ee-8027-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-61", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "multiscale spectral segmentation,\u201d ICCV , 2013.\n[25] S. Man \u00b4en, M. Guillaumin, and L. Van Gool, \u201cPrime Object Proposals with Randomized Prim\u2019s Algorithm,\u201d in ICCV , 2013.\n[26] C. L. Zitnick and P . Doll \u00b4ar, \u201cEdge boxes: Locating object proposals\nfrom edges,\u201d in ECCV , 2014.\n[27] M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P . H. S. Torr, \u201cBING:\nBinarized normed gradients for objectness estimation at 300fps,\u201d\ninCVPR , 2014.\n[28] D. Hoiem, A. Efros, and M. Hebert, \u201cRecovering occlusion boundaries from an image,\u201d IJCV , vol. 91, no. 3, pp. 328\u2013346, 2011.\n[29] Z. Ren and G. Shakhnarovich, \u201cImage segmentation by cascaded\nregion agglomeration,\u201d in CVPR , 2013.\n[30] D. Weiss and B. Taskar, \u201cScalpel: Segmentation cascades with"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d94-6db4-11ee-946f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-62", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "region agglomeration,\u201d in CVPR , 2013.\n[30] D. Weiss and B. Taskar, \u201cScalpel: Segmentation cascades with\nlocalized priors and ef\ufb01cient learning,\u201d in CVPR , 2013.\n[31] L. Najman and M. Schmitt, \u201cGeodesic saliency of watershed\ncontours and hierarchical segmentation,\u201d TP AMI , vol. 18, no. 12,\npp. 1163\u20131173, 1996.\n[32] P . Arbel \u00b4aez, \u201cBoundary extraction in natural images using ultrametric contour maps,\u201d in POCV , June 2006.\n[33] D. Martin, C. Fowlkes, and J. Malik, \u201cLearning to detect natural\nimage boundaries using local brightness, color and texture cues,\u201d\nTP AMI , vol. 26, no. 5, pp. 530\u2013549, 2004.\n[34] P . Doll \u00b4ar and C. Zitnick, \u201cStructured forests for fast edge detection,\u201d ICCV , 2013.\n[35] http://www.eecs.berkeley.edu/Research/\nProjects/CS/vision/grouping/resources.html."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d95-6db4-11ee-b6b2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-63", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "[35] http://www.eecs.berkeley.edu/Research/\nProjects/CS/vision/grouping/resources.html.\n[36] J. Pont-Tuset and F. Marques, \u201cSupervised evaluation of image\nsegmentation and object proposal techniques,\u201d P AMI , 2015.\n[37] \u2014\u2014, \u201cMeasures and meta-measures for the supervised evaluation\nof image segmentation,\u201d in CVPR , 2013.\n[38] M. Maire, S. X. Yu, and P . Perona, \u201cHierarchical scene annotation,\u201d\ninBMVC , 2013.\n[39] M. Everingham, H. Muller, and B. Thomas, \u201cEvaluating image\nsegmentation algorithms using the pareto front,\u201d in ECCV , 2006.\n[40] M. Ehrgott, Multicriteria optimization . Springer, 2005.\n[41] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,\nand A. Zisserman, \u201cThe PASCAL Visual Object Classes\nChallenge 2012 (VOC2012) Results,\u201d http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d96-6db4-11ee-ba83-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-64", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "Challenge 2012 (VOC2012) Results,\u201d http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.\n[42] B. Hariharan, P . Arbelaez, L. Bourdev, S. Maji, and J. Malik,\n\u201cSemantic contours from inverse detectors,\u201d in ICCV , 2011.\n[43] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P . Perona, D. Ramanan,\nP . Doll \u00b4ar, and C. Zitnick, \u201cMicrosoft COCO: Common Objects in\nContext,\u201d in ECCV , 2014.\n[44] J. Pont-Tuset and L. V . Gool, \u201cBoosting object proposals: From\npascal to COCO,\u201d in ICCV , 2015.\n[45] J. Hosang, R. Benenson, and B. Schiele, \u201cHow good are detection\nproposals, really?\u201d in BMVC , 2014.\n[46] J. Hosang, R. Benenson, P . Doll \u00b4ar, and B. Schiele, \u201cWhat makes\nfor effective detection proposals?\u201d P AMI , 2015.\n14"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d97-6db4-11ee-915e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-65", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "for effective detection proposals?\u201d P AMI , 2015.\n14\n[47] B. Hariharan, P . Arbel \u00b4aez, R. Girshick, and J. Malik, \u201cSimultaneous detection and segmentation,\u201d in ECCV , 2014.\nJordi Pont-Tuset is a post-doctoral researcher\nat ETHZ, Switzerland, in Prof. Luc Van Gool\u2019s\ncomputer vision group (2015). He received the\ndegree in Mathematics in 2008, the degree in\nElectrical Engineering in 2008, the M.Sc. in\nResearch on Information and Communication\nTechnologies in 2010, and the Ph.D with honors\nin 2014; all from the Universitat Polit `ecnica de\nCatalunya, BarcelonaTech (UPC). He worked at\nDisney Research, Z \u00a8urich (2014).\nPablo Arbel \u00b4aezreceived a PhD with honors in\nApplied Mathematics from the Universit \u00b4e ParisDauphine in 2005. He was a Research Scientist\nwith the Computer Vision Group at UC Berkeley from 2007 to 2014. He currently holds a\nfaculty position at Universidad de los Andes in\nColombia. His research interests are in computer vision, where he has worked on a number\nof problems, including perceptual grouping, object recognition and the analysis of biomedical\nimages."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d98-6db4-11ee-ad2c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-66", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "Colombia. His research interests are in computer vision, where he has worked on a number\nof problems, including perceptual grouping, object recognition and the analysis of biomedical\nimages.\nJonathan T. Barron is a senior research scientist at Google, working on computer vision\nand computational photography. He received a\nPhD in Computer Science from the University\nof California, Berkeley in 2013, where he was\nadvised by Jitendra Malik, and he received a\nHonours BSc in Computer Science from the\nUniversity of Toronto in 2007. His research interests include computer vision, machine learning,\ncomputational photography, shape reconstruction, and biological image analysis. He received\na National Science Foundation Graduate Research Fellowship in 2009,\nand the C.V. Ramamoorthy Distinguished Research Award in 2013.\nFerran Marques received the degree in Electrical Engineering and the Ph.D. from the Universitat Polit `ecnica de Catalunya, BarcelonaT ech (UPC), where he is currently Professor at\nthe department of Signal Theory and Communications. In the term 2002-2004, he served\nas President of the European Association for\nSignal Processing (EURASIP). He has served\nas Associate Editor of the IEEE Transactions\non Image Processing (2009-2012) and as Area\nEditor for Signal Processing: Image Communication, Elsevier (2010-2014). In 2011, he received the Jaume Vicens"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d99-6db4-11ee-b1d4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-67", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "on Image Processing (2009-2012) and as Area\nEditor for Signal Processing: Image Communication, Elsevier (2010-2014). In 2011, he received the Jaume Vicens\nVives distinction for University Teaching Quality. Currently, he serves\nas Dean of the Electrical Engineering School (ETSETB-TelecomBCN)\nat UPC. He has published over 150 conference and journal papers, 2\nbooks, and holds 4 international patents.\nJitendra Malik is Arthur J. Chick Professor in\nthe Department of Electrical Engineering and\nComputer Science at the University of California\nat Berkeley, where he also holds appointments\nin vision science and cognitive science. He received the PhD degree in Computer Science\nfrom Stanford University in 1985. In January\n1986, he joined UC Berkeley as a faculty member in the EECS department where he served as\nChair of the Computer Science Division during\n2002-2006, and of the Department of EECS during 2004-2006. Jitendra Malik\u2019s group has worked on computer vision,\ncomputational modeling of biological vision, computer graphics and\nmachine learning. Several well-known concepts and algorithms arose in\nthis work, such as anisotropic diffusion, normalized cuts, high dynamic\nrange imaging, shape contexts and poselets. According to Google\nScholar, ten of his papers have received more than a thousand citations\neach. He has graduated 33 PhD students. Jitendra was awarded the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d9a-6db4-11ee-bb81-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1503.00848-68", "title": "Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation", "source": "http://arxiv.org/pdf/1503.00848", "start_index": 0, "chroma:document": "Scholar, ten of his papers have received more than a thousand citations\neach. He has graduated 33 PhD students. Jitendra was awarded the\nLonguet-Higgins Award for \u201cA Contribution that has Stood the Test of\nTime\u201d twice, in 2007 and 2008. He is a Fellow of the IEEE and the\nACM, a member of the National Academy of Engineering, and a fellow\nof the American Academy of Arts and Sciences. He received the PAMI\nDistinguished Researcher Award in computer vision in 2013 and the\nK.S. Fu prize in 2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d9b-6db4-11ee-950f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-0", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "High-for-Low and Low-for-High:\nEf\ufb01cient Boundary Detection from Deep Object Features\nand its Applications to High-Level Vision\nGedas Bertasius\nUniversity of Pennsylvania\ngberta@seas.upenn.eduJianbo Shi\nUniversity of Pennsylvania\njshi@seas.upenn.eduLorenzo Torresani\nDartmouth College\nlt@dartmouth.edu\nAbstract\nMost of the current boundary detection systems rely exclusively on low-level features, such as color and texture.\nHowever, perception studies suggest that humans employ\nobject-level reasoning when judging if a particular pixel\nis a boundary. Inspired by this observation, in this work\nwe show how to predict boundaries by exploiting objectlevel features from a pretrained object-classi\ufb01cation network. Our method can be viewed as a High-for-Low approach where high-level object features inform the low-level\nboundary detection process. Our model achieves state-ofthe-art performance on an established boundary detection\nbenchmark and it is ef\ufb01cient to run.\nAdditionally, we show that due to the semantic nature of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d9c-6db4-11ee-8b90-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-1", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "benchmark and it is ef\ufb01cient to run.\nAdditionally, we show that due to the semantic nature of\nour boundaries we can use them to aid a number of highlevel vision tasks. We demonstrate that by using our boundaries we improve the performance of state-of-the-art methods on the problems of semantic boundary labeling, semantic segmentation and object proposal generation. We can\nview this process as a Low-for-High scheme, where lowlevel boundaries aid high-level vision tasks.\nThus, our contributions include a boundary detection\nsystem that is accurate, ef\ufb01cient, generalizes well to multiple datasets, and is also shown to improve existing stateof-the-art high-level vision methods on three distinct tasks.\n1. Introduction\nIn the vision community, boundary detection has always\nbeen considered a low-level problem. However, psychological studies suggest that when a human observer perceives\nboundaries, object level reasoning is used [13, 25, 17]. Despite these \ufb01ndings, most of the boundary detection methods rely exclusively on low-level color and gradient features. In this work, we present a method that uses objectlevel features to detect boundaries. We argue that using\nobject-level information to predict boundaries is more sim-Low-Level Task High-Level Tasks\nBD SBL SS OP\nODS MF AP PI-IOU MR"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d9d-6db4-11ee-83c1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-2", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "object-level information to predict boundaries is more sim-Low-Level Task High-Level Tasks\nBD SBL SS OP\nODS MF AP PI-IOU MR\nSotA 0.76 [27] 28.0 [11] 19.9 [11] 45.8 [5] 0.88 [31]\nHFL 0.77 62.5 54.6 48.8 0.90\nTable 1: Summary of results achieved by our proposed\nmethod (H FL) and state-of-the-art methods (SotA). We provide results on four vision tasks: Boundary Detection (BD),\nSemantic Boundary Labeling (SBL), Semantic Segmentation (SS), and Object Proposal (OP). The evaluation metrics\ninclude ODS F-score for BD task, max F-score (MF) and\naverage precision (AP) for SBL task, per image intersection over union (PI-IOU) for SS task, and max recall (MR)\nfor OP task. Our method produces better results in each of\nthese tasks according to these metrics.\nilar to how humans reason. Our boundary detection scheme\ncan be viewed as a High-for-Low approach where we use\nhigh-level object features as cues for a low-level boundary\ndetection process. Throughout the rest of the paper, we refer to our proposed boundaries as High-for-Low boundaries"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d9e-6db4-11ee-adc7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-3", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "high-level object features as cues for a low-level boundary\ndetection process. Throughout the rest of the paper, we refer to our proposed boundaries as High-for-Low boundaries\n(HFL).\nWe present an ef\ufb01cient deep network that uses objectlevel information to predict the boundaries. Our proposed\narchitecture reuses features from the sixteen convolutional\nlayers of the network of Simonyan et al. [29], which we\nrefer to as VGG net. The VGG net has been trained for\nobject classi\ufb01cation, and therefore, reusing its features allows our method to utilize high-level object information to\npredict H FL boundaries. In the experimental section, we\ndemonstrate that using object-level features produces semantically meaningful boundaries and also achieves above\nstate-of-the-art boundary detection accuracy.\nAdditionally, we demonstrate that we can successfully\napply our H FL boundaries to a number of high-level vision\ntasks. We show that by using H FL boundaries we improve\nthe results of three existing state-of-the-art methods on the\ntasks of semantic boundary labeling, semantic segmenta1arXiv:1504.06201v3  [cs.CV]  21 Sep 2015\ntion and object proposal generation. Therefore, using H FL\nboundaries to boost the results in high level vision tasks"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8d9f-6db4-11ee-8e4e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-4", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "tion and object proposal generation. Therefore, using H FL\nboundaries to boost the results in high level vision tasks\ncan be viewed as a Low-for-High scheme, where boundaries\nserve as low-level cues to aid high-level vision tasks.\nWe present the summarized results for the boundary detection and the three mentioned high-level vision tasks in\nTable 1. Speci\ufb01cally, we compare our proposed method and\nan appropriate state-of-the-art method for that task. As the\nresults indicate, we achieve better results in each of the tasks\nfor each presented evaluation metric. We present more detailed results for each of these tasks in the later sections.\nIn summary, our contributions are as follows. First, we\nshow that using object-level features for boundary detection produces perceptually informative boundaries that outperform prior state-of-the-art boundary detection methods.\nSecond, we demonstrate that we can use H FL boundaries\nto enhance the performance on the high-level vision tasks\nof semantic boundary labeling, semantic segmentation and\nobject proposal. Finally, our method can detect boundaries\nin near-real time. Thus, we present a boundary detection\nsystem that is accurate, ef\ufb01cient, and is also applicable to\nhigh level vision tasks.\n2. Related Work\nMost of the contour detection methods predict boundaries based purely on color, text, or other low-level features."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8da0-6db4-11ee-8986-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-5", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "high level vision tasks.\n2. Related Work\nMost of the contour detection methods predict boundaries based purely on color, text, or other low-level features.\nWe can divide these methods into three broad categories:\nspectral methods, supervised discriminative methods and\ndeep learning based methods.\nSpectral methods formulate contour detection problem\nas an eigenvalue problem. The solution to this problem is\nthen used to reason about the boundaries. The most successful approaches in this genre are the MCG detector [2], gPb\ndetector [1], PMI detector [14], and Normalized Cuts [28].\nSome of the notable discriminative boundary detection\nmethods include sketch tokens (ST) [18], structured edges\n(SE) [6] and sparse code gradients (SCG) [23]. While SCG\nuse supervised SVM learning [4], the latter two methods\nrely on a random forest classi\ufb01er and models boundary detection as a classi\ufb01cation task.\nRecently there have been attempts to apply deep learning\nto the task of boundary detection. SCT [20] is a sparse coding approach that reconstructs an image using a learned dictionary and then detect boundaries. Both N4\ufb01elds [10] and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8da1-6db4-11ee-b051-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-6", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "DeepNet [16] approaches use Convolutional Neural Networks (CNNs) to predict edges. N4\ufb01elds rely on dictionary learning and the use of the Nearest Neighbor algorithm\nwithin a CNN framework while DeepNet uses a traditional\nCNN architecture to predict contours.\nThe most similar to our approach is DeepEdge [3],\nwhich uses a multi-scale bifurcated network to perform contour detection using object-level features. However, we\nshow that our method achieves better results even without\n1100x1100\n512 \n500x375...\n1024 \nInput Image\n5504 Convolutional     Feature Maps 500x375Upsampled     Image\n5504-Dimensional    Feature Vectors  Predicted  Boundaries\n|{z}\nFeature Interpolation & Concatenation\n1100x1100x64- Candidate Contour Points\n35x35x512\nShared Weights for all     Candidate Points\n|{z}\nFigure 1: An illustration of our architecture (best viewed in\ncolor). First we extract a set of candidate contour points.\nThen we upsample the image and feed it through 16convolutional layers pretrained for object classi\ufb01cation. For each\ncandidate point, we \ufb01nd its correspondence in each of the\nfeature maps and perform feature interpolation. This yields"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8da2-6db4-11ee-b1f7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-7", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "candidate point, we \ufb01nd its correspondence in each of the\nfeature maps and perform feature interpolation. This yields\na5504 -dimensional feature vector for each candidate point.\nWe feed each of these vectors to two fully connected layers\nand store the predictions to produce a \ufb01nal boundary map.\nthe complicated multi-scale and bifurcated architecture of\nDeepEdge. Additionally, unlike DeepEdge, our system can\nrun in near-real time.\nIn comparison to prior approaches, we offer several contributions. First, we propose to use object-level information to predict boundaries. We argue that such an approach\nleads to semantic boundaries, which are more consistent\nwith humans reasoning. Second, we avoid feature engineering by learning boundaries from human-annotated data.\nFinally, we demonstrate excellent results for both low-level\nand high-level vision tasks. For the boundary detection task,\nour proposed H FL boundaries outperform all of the prior\nmethods according to both F-score metrics. Additionally,\nwe show that because H FL boundaries are based on objectlevel features, they can be used to improve performance in\nthe high-level vision tasks of semantic boundary labeling,\nsemantic segmentation, and object proposal generation.\n3. Boundary Detection\nIn this section, we describe our proposed architecture\nand the speci\ufb01c details on how we predict H FL boundaries"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8da3-6db4-11ee-a810-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-8", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "3. Boundary Detection\nIn this section, we describe our proposed architecture\nand the speci\ufb01c details on how we predict H FL boundaries\nusing our method. The detailed illustration of our architecture is presented in Figure 1.\nFigure 2: A visualization of selected convolutional feature maps from VGG network (resized to the input image dimension).\nBecause VGG was optimized for an object classi\ufb01cation task, it produces high activation values on objects and their parts.\n3.1. Selection of Candidate Contour Points\nWe \ufb01rst extract a set of candidate contour points with\na high recall. Due to its ef\ufb01ciency and high recall performance, we use the SE edge detector [6]. In practice, we\ncould eliminate this step and simply try to predict boundaries at every pixel. However, selecting a set of initial candidate contour points, greatly reduces the computational cost.\nSince our goal is to build a boundary detector that is both accurate and ef\ufb01cient, we use these candidate points to speed\nup the computation of our method.\n3.2. Object-Level Features\nAfter selecting candidate contour points, we up-sample\nthe original input image to a larger dimension (for example\n1100\u00021100 ). The up-sampling is done to minimize the loss\nof information due to the input shrinkage caused by pooling"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8da4-6db4-11ee-8b09-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-9", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "the original input image to a larger dimension (for example\n1100\u00021100 ). The up-sampling is done to minimize the loss\nof information due to the input shrinkage caused by pooling\nat the different layers. Afterwards, we feed the up-sampled\nimage through 16convolutional layers of the VGG net.\nWe use the VGG net as our model because it has been\ntrained to recognize a large number of object classes (the\n1000 categories of the ImageNet dataset [24]) and thus encodes object-level features that apply to many classes. To\npreserve speci\ufb01c location information we utilize only the\n16convolutional layers of the VGG net. We don\u2019t use fully\nconnected layers because they don\u2019t preserve spatial information, which is crucial for accurate boundary detection.\nWe visualize some of the selected convolutional maps in\nFigure 2. Note the high activation values around the various objects in the images, which con\ufb01rms our hypothesis\nthat the VGG net encodes object speci\ufb01c information in its\nconvolutional feature maps.\n3.3. Feature Interpolation\nSimilarly to [26, 12, 19], we perform feature interpolation in deep layers. After the up-sampled image passes"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8da5-6db4-11ee-9b4c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-10", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "3.3. Feature Interpolation\nSimilarly to [26, 12, 19], we perform feature interpolation in deep layers. After the up-sampled image passes\nthrough all 16convolutional layers, for each selected candidate contour point we \ufb01nd its corresponding point in the\nfeature maps. Due to the dimension differences in convo-lutional maps these correspondences are not exact. Thus\nwe perform feature interpolation by \ufb01nding the four nearest\npoints and averaging their activation values. This is done\nin each of the 5504 feature maps. Thus, this results in a\n5504 -dimensional vector for each candidate point.\nWe note that the interpolation of convolutional feature\nmaps is the crucial component that enables our system to\npredict the boundaries ef\ufb01ciently. Without feature interpolation, our method would need to independently process\nthe candidate edge points by analyzing a small image patch\naround each point, as for example done in DeepEdge [3]\nwhich feeds one patch at a time through a deep network.\nHowever, when the number of candidate points is large\n(e.g., DeepEdge considers about 15K points at each of 4 different scales), their patches overlap signi\ufb01cantly and thus a\nlarge amount of computation is wasted by recalculating \ufb01lter response values over the same pixels. Instead, we can"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8da6-6db4-11ee-9743-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-11", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "large amount of computation is wasted by recalculating \ufb01lter response values over the same pixels. Instead, we can\ncompute the features for all candidate points with a single\npass through the network by performing deep convolution\nover the entire image (i.e., feeding the entire image rather\nthan one patch at a time) and then by interpolating the convolutional feature maps at the location of each candidate\nedge point so as to produce its feature descriptor. Thanks to\nthis speedup, our method has a runtime of 1:2seconds (using a K40 GPU), which is better than the runtimes of prior\ndeep-learning based edge detection methods [27, 10, 16, 3].\n3.4. Learning to Predict Boundaries\nAfter performing feature interpolation, we feed the\n5504 -dimensional feature vectors corresponding to each of\nthe candidate contour points to two fully connected layers\nthat are optimized to the human agreement criterion. To be\nmore precise, we de\ufb01ne our prediction objective as a fraction of human annotators agreeing on the presence of the\nboundary at a particular pixel. Therefore, a learning objective aims at mimicking the judgement of the human labelers.\nFinally, to detect H FL boundaries, we accumulate the\npredictions from the fully connected layers for each of the\ncandidate points and produce a boundary probability map\nas illustrated in Figure 1."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8da7-6db4-11ee-a3fa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-12", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "Finally, to detect H FL boundaries, we accumulate the\npredictions from the fully connected layers for each of the\ncandidate points and produce a boundary probability map\nas illustrated in Figure 1.\n3.5. Implementation Details\nIn this section, we describe the details behind the training\nprocedure of our model. We use the Caffe library [15] to\nimplement our network architecture.\nIn the training stage, we freeze the weights in all of the\nconvolutional layers. To learn the weights in the two fully\nconnected layers we train our model to optimize the least\nsquares error of the regression criterion that we described\nin the previous subsection. To enforce regularization we set\na dropout rate of 0:5in the fully connected layers.\nOur training dataset includes 80Kpoints from the\nBSDS500 dataset [22]. As described in the previous subsection, the labels represent the fraction of human annotators agreeing on the boundary presence. We divide the label space into four quartiles, and select an equal number of\nsamples for each quartile to balance the training dataset. In\naddition to the training dataset, we also sample a hold-out\ndataset of size 40;000. We use this for the hard-positive\nmining [21] in order to reduce the number of false-negative\npredictions."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8da8-6db4-11ee-b2c1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-13", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "dataset of size 40;000. We use this for the hard-positive\nmining [21] in order to reduce the number of false-negative\npredictions.\nFor the \ufb01rst 25epochs we train the network on the original80;000training samples. After the \ufb01rst 25epochs, we\ntest the network on the hold-out dataset and detect false negative predictions made by our network. We then augment\nthe original 80;000 training samples with the false negatives and the same number of randomly selected true negatives. For the remaining 25epochs, we train the network on\nthis augmented dataset.\n3.6. Boundary Detection Results\nIn this section, we present our results on the BSDS500\ndataset [22], which is the most established benchmark for\nboundary detection. The quality of the predicted boundaries is evaluated using three standard measures: \ufb01xed contour threshold (ODS), per-image best threshold (OIS), and\naverage precision (AP).\nWe compare our approach to the state-of-the-art based\non two different sets of BSDS500 ground truth boundaries.\nFirst, we evaluate the accuracy by matching each of the\npredicted boundary pixels with the ground truth boundaries\nthat were annotated by anyof the human annotators. This"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8da9-6db4-11ee-a551-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-14", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "First, we evaluate the accuracy by matching each of the\npredicted boundary pixels with the ground truth boundaries\nthat were annotated by anyof the human annotators. This\nset of ground truth boundaries is referred to as \u201cany\u201d. We\npresent the results for \u201cany\u201d ground truth boundaries in the\nlower half of Table 2. As indicated by the results, H FL\nboundaries outperform all the prior methods according to\nboth F-score measures.\nRecently, there has been some criticism raised about\nthe procedure for boundary detection evaluation on the\nBSDS500 dataset. One issue with the BSDS500 dataset\ninvolves the so called \u201corphan\u201d boundaries: the bound-Consensus GT ODS OIS AP FPS\nSCG [23] 0.6 0.64 0.56 1/280\nDeepNet [16] 0.61 0.64 0.61 1=5z\nPMI [14] 0.61 0.68 0.56 1/900\nDeepEdge [3] 0.62 0.64 0.64 1=1000z\nN4-\ufb01elds [10] 0.64 0.67 0.64 1=6z\nHFL 0.65 0.68 0.67 5=6z\nAny GT ODS OIS AP FPS"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8daa-6db4-11ee-9e26-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-15", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "HFL 0.65 0.68 0.67 5=6z\nAny GT ODS OIS AP FPS\nSE [6] 0.75 0.77 0.80 2.5\nMCG [2] 0.75 0.78 0.76 1/24\nN4-\ufb01elds [10] 0.75 0.77 0.78 1=6z\nDeepEdge [3] 0.75 0.77 0.81 1=1000z\nMSC [30] 0.76 0.78 0.79 DeepContour [27] 0.76 0.77 0.8 1=30z\nHFL 0.77 0.79 0.8 5=6z\nTable 2: Boundary detection results on BSDS500 benchmark. Upper half of the table illustrates the results for \u201cconsensus\u201d ground-truth criterion while the lower half of the\ntable depicts the results for \u201cany\u201d ground-truth criterion. In\nboth cases, our method outperforms all prior methods according to both ODS (optimal dataset scale) and OIS (optimal image scale) metrics. We also report the run-time of\nour method (zGPU time) in the FPS column (frames per\nsecond), which shows that our algorithm is faster than prior\napproaches based on deep learning [27, 10, 16, 3]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dab-6db4-11ee-8e4e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-16", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "second), which shows that our algorithm is faster than prior\napproaches based on deep learning [27, 10, 16, 3].\nFigure 5: Qualitative results on the BSDS benchmark. The\n\ufb01rst column of images represent input images. The second\ncolumn illustrates SE [6], while the third column depicts\nHFL boundaries. Notice that SE boundaries are predicted\nwith low con\ufb01dence if there is no signi\ufb01cant change in color\nbetween the object and the background. Instead, because\nour model is de\ufb01ned in terms of object-level features, it can\npredict object boundaries with high con\ufb01dence even if there\nis no signi\ufb01cant color variation in the scene.\naries that are marked by only one or two human annotators. These \u201corphan\u201d boundaries comprise around 30% of\nBSDS500 dataset but most of them are considered uninformative. However, the standard evaluation benchmark rewards the methods that predict these boundaries. To resolve\n500 1000 1500 2000 2500 3000 3500 4000 4500 5000 5500024x 10\u22123Weight Magnitude\nFeature Map Number (Sorted in the Order from Early to Deep Layers)Figure 6: We train a linear regression model and visualize\nits weight magnitudes in order to understand which features\nare used most heavily in the boundary prediction (this linear"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dac-6db4-11ee-a374-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-17", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "its weight magnitudes in order to understand which features\nare used most heavily in the boundary prediction (this linear\nregression is used only for the visualization purposes and\nnot for the accuracy analysis). Note how heavily weighted\nfeatures lie in the deepest layers of the network, i.e., the layers that are most closely associated with object information.\nthis issue we also evaluate our H FL boundaries on the so\ncalled \u201cconsensus\u201d set of ground truth boundaries. These\n\u201cconsensus\u201d boundaries involve only boundaries that are\nmarked by allof the human annotators and hence, are considered perceptually meaningful. In the upper half of Table 2, we present the results achieved by our method on the\n\u201cconsensus\u201d set of the ground truth boundaries. Our H FL\nboundaries outperform or tie all the prior methods in each\nof the three evaluation metrics, thus suggesting that H FL\nboundaries are similar to the boundaries that humans annotated. We also report the runtimes in Table 2 and note that\nour method runs faster than previous deep-learning based\nedge detection systems [27, 10, 16, 3].\nOur proposed model computes a highly nonlinear function of the 5504-dimensional feature vector of each candidate point. Thus, it is dif\ufb01cult to assess which features are\nused most heavily by our edge predictor. However, we can"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dad-6db4-11ee-bda7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-18", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "used most heavily by our edge predictor. However, we can\ngain a better insight by replacing the nonlinear function with\na simple linear model. In Fig. 6 we show the weight magnitudes of a simple linear regression model (we stress that\nthis linear model is used only for feature visualization purposes). From this Figure, we observe that many important\nfeatures are located in the deepest layers of the VGG network. As shown in [7], these layers encode high-level object information, which con\ufb01rms our hypothesis that highlevel information is useful for boundary detection.\nFinally, we present some qualitative results achieved by\nour method in Figure 5. These examples illustrate the effective advantage that H FL boundaries provide over another\nstate-of-the-art edge detection system, the SE system [6].\nSpeci\ufb01cally, observe the parts of the image where there is a\nboundary that separates an object from the background but\nwhere the color change is pretty small. Notice that because\nthe SE boundary detection is based on low-level color and\ntexture features, it captures these boundaries with very low\ncon\ufb01dence. In comparison, because H FL boundaries rely\non object-level features, it detects these boundaries with\nhigh con\ufb01dence.4. High-Level Vision Applications\nIn this section, we describe our proposed Low-for-High"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dae-6db4-11ee-bb3b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-19", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "high con\ufb01dence.4. High-Level Vision Applications\nIn this section, we describe our proposed Low-for-High\npipeline: using low-level boundaries to aid a number of\nhigh-level vision tasks. We focus on the tasks of semantic\nboundary labeling, semantic segmentation and object proposal generation. We show that using H FL boundaries improves the performance of state-of-the-art methods in each\nof these high-level vision tasks.\n4.1. Semantic Boundary Labeling\nThe task of semantic boundary labeling requires not only\nto predict the boundaries but also to associate a speci\ufb01c\nobject class to each of the boundaries. This implies that\ngiven our predicted boundaries we also need to label them\nwith object class information. We approach this problem\nby adopting the ideas from the recent work on Fully Convolutional Networks (FCN) [19]. Given an input image,\nwe concurrently feed it to our boundary-predicting network\n(described in Section 3), and also through the FCN that was\npretrained for 20Pascal VOC classes and the background\nclass. While our proposed network produces H FL boundaries, the FCN model predicts class probabilities for each\nof the pixels. We can then merge the two output maps as\nfollows. For a given boundary point we consider a 9\u00029"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8daf-6db4-11ee-8158-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-20", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "of the pixels. We can then merge the two output maps as\nfollows. For a given boundary point we consider a 9\u00029\ngrid around that point from each of the 21FCN object-class\nprobability maps. We calculate the maximum value inside\neach grid, and then label the boundary at a given pixel with\nthe object-class that corresponds to the maximum probability across these 21maps. We apply this procedure for each\nof the boundary points, in order to associate object-class\nlabels to the boundaries. Note that we consider the grids\naround the boundary pixel because the output of the FCN\nhas a poor localization, and considering the grids rather than\nindividual pixels leads to higher accuracy.\nWe can also merge H FL boundaries with the state-ofthe-art DeepLab-CRF segmentation [5] to obtain higher accuracy. We do this in a similar fashion as just described.\nFirst, around a given boundary point we extract a 9\u00029\ngrid from the DeepLab-CRF segmentation. We then compute the mode value in the grid (excluding the background\nclass), and use the object-class corresponding to the mode\nvalue as a label for the given boundary point. We do this for\neach of the boundary points. By merging H FL boundaries\nand the output of FCN or DeepLab-CRF, we get semantic"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8db0-6db4-11ee-b803-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-21", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "each of the boundary points. By merging H FL boundaries\nand the output of FCN or DeepLab-CRF, we get semantic\nboundaries that are highly localized and also contain objectspeci\ufb01c information.\n4.1.1 Semantic Boundary Labeling Results\nIn this section, we present semantic boundary labeling results on the SBD dataset [11], which includes ground truth\nboundaries that are also labeled with one of 20Pascal VOC\nclasses. The boundary detection accuracy for each class is\nMethod (Metric) aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean\nInvDet (MF) 42.6 49.5 15.7 16.8 36.7 43.0 40.8 22.6 18.1 26.6 10.2 18.0 35.2 29.4 48.2 14.3 26.8 11.2 22.2 32.0 28.0\nHFL-FC8 (MF) 71.6 59.6 68.0 54.1 57.2 68.0 58.8 69.3 43.3 65.8 33.3 67.9 67.5 62.2 69.0 43.8 68.5 33.9 57.7 54.8 58.7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8db1-6db4-11ee-99b2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-22", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "HFL-CRF (MF) 73.9 61.4 74.6 57.2 58.8 70.4 61.6 71.9 46.5 72.3 36.2 71.1 73.0 68.1 70.3 44.4 73.2 42.6 62.4 60.1 62.5\nInvDet (AP) 38.4 29.6 9.6 9.9 24.2 33.6 31.3 17.3 10.7 16.4 3.7 12.1 28.5 20.4 45.7 7.6 16.1 5.7 14.6 22.7 19.9\nHFL-FC8 (AP) 66.0 50.7 58.9 40.6 47.1 62.9 51.0 59.0 25.6 54.6 15.3 57.8 57.3 55.9 62.2 27.5 55.6 18.0 50.1 40.6 47.8\nHFL-CRF (AP) 71.2 55.2 69.3 45.7 48.9 71.1 56.8 65.7 29.1 65.9 17.7 64.5 68.3 64.7 65.9 29.1 66.5 25.7 60.0 49.8 54.6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8db2-6db4-11ee-9056-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-23", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "Table 3: Results of semantic boundary labeling on the SBD benchmark using the Max F-score (MF) and Average Precision\n(AP) metrics. Our method (H FL) outperforms Inverse Detectors [11] for all 20categories according to both metrics. Note\nthat using the CRF output to label the boundaries produces better results than using the outputs from the FC8 layer of FCN.\nevaluated using the maximum F-score (MF), and average\nprecision (AP) measures.\nLabeling boundaries with the semantic object information is a novel and still relatively unexplored problem.\nTherefore, we found only one other approach (Inverse Detectors) that tried to tackle this problem [11]. The basic idea behind Inverse Detectors consists of several steps.\nFirst, generic boundaries in the image are detected. Then,\na number of object proposal boxes are generated. These\ntwo sources of information are then used to construct the\nfeatures. Finally, a separate classi\ufb01er is used to label the\nboundaries with the object-speci\ufb01c information.\nTable 3 shows that our approach signi\ufb01cantly outperforms Inverse Detectors according to both the maximum Fscore and the average precision metrics for all twenty categories. As described in Section 4.1 we evaluate the two"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8db3-6db4-11ee-8627-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-24", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "variants of our method. Denoted by H FL-FC8 is the variant for which we label H FL boundaries with the outputs\nfrom the last layer (FC8) of the pretrained FCN. We denote\nwith H FL-CRF the result of labeling our boundaries with\nthe output from the DeepLab-CRF [5]. Among these two\nvariants, we show that the latter one produces better results.\nThis is expected since the CRF framework enforces spatial\ncoherence in the semantic segments.\nIn Figure 7, we present some of the qualitative results\nproduced by our method. We note that even with multiple\nobjects in the image, our method successfully recognizes\nand localizes boundaries of each of the classes.\n4.2. Semantic Segmentation\nFor the semantic segmentation task, we propose to\nenhance the DeepLab-CRF [5] with our predicted H FL\nboundaries. DeepLab-CRF is a system comprised of a Fully\nConvolutional Network (described in Section 4.1) and a\ndense CRF applied on top of FCN predictions.\nSpeci\ufb01cally, in the CRF, the authors propose to use\na Gaussian kernel and a bilateral term including position\nand color terms as the CRF features (see [5]). While in\nmost cases the proposed scheme works well, DeepLab-CRF"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8db4-6db4-11ee-b899-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-25", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "a Gaussian kernel and a bilateral term including position\nand color terms as the CRF features (see [5]). While in\nmost cases the proposed scheme works well, DeepLab-CRF\nsometimes produces segmentations that are not spatially coherent, particularly for images containing small object reFigure 7: A visualization of the predicted semantic boundary labels. Images in the \ufb01rst column are input examples.\nColumns two and three show semantic H FL boundaries of\ndifferent object classes. Note that even with multiple objects appearing simultaneously, our method outputs precise\nsemantic boundaries.\ngions.\nWe propose to address this issue by adding features\nbased on our predicted H FL boundaries in the CRF framework. Note that we use predicted boundaries from Section 3\nand not the boundaries labeled with the object information\nthat we obtained in Section 4.1. We use the Normalized\nCut [28] framework to generate our features.\nFirst, we construct a pixel-wise af\ufb01nity matrix Wusing\nour H FL boundaries. We measure the similarity between\ntwo pixels as:\nWij= exp (\u0000max\np2ijfM(p)2\n\u001b2g)\nwhereWijrepresents the similarity between pixels iand\nj,pdenotes the boundary point along the line segment ij\nconnecting pixels iandj,Mdepicts the magnitude of the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8db5-6db4-11ee-aaa1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-26", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "j,pdenotes the boundary point along the line segment ij\nconnecting pixels iandj,Mdepicts the magnitude of the\nboundary at pixel p, and\u001bdenotes the smoothness parameter, which is usually set to 14% of the maximum boundary\nvalue in the image.\nThe intuitive idea is that two pixels are similar (i.e.\nWij= 1) if there is no boundary crossing the line connecting these two pixels (i.e. M(p) = 08p2ij) or if the\nboundary strength is low. We note that it is not necessary\nto build a full af\ufb01nity matrix W. We build a sparse af\ufb01nMetric Method (Dataset) aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean\nPP-IOUDL-CRF (VOC) 78.6 41.1 83.5 75.3 72.9 83.1 76.6 80.8 37.8 72.1 66.5 64.7 65.8 75.7 80.5 34.4 75.9 47.4 86.6 77.9 68.9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8db6-6db4-11ee-a80a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-27", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "DL-CRF+H FL (VOC) 77.9 41.2 83.1 74.4 73.2 85.5 76.1 80.6 35.7 71.0 66.6 64.3 65.9 75.2 80.2 32.8 75.2 47.0 87.1 77.9 68.5\nDL-CRF (SBD) 74.2 68.0 81.9 64.6 71.8 86.3 78.3 84.3 41.6 78.0 49.9 82.0 78.5 77.1 80.1 54.3 75.6 49.8 79.5 70.1 71.4\nDL-CRF+H FL (SBD) 75.1 69.2 81.6 64.8 71.3 86.4 78.1 84.1 41.2 77.8 50.4 81.6 78.2 78.5 80.7 53.8 74.9 49.1 79.5 70.4 71.4\nPI-IOUDL-CRF (VOC) 46.1 28.0 48.5 54.5 45.5 57.6 34.1 47.3 19.5 61.4 41.6 42.5 34.4 61.8 62.1 22.1 50.5 41.0 61.2 31.9 44.6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8db7-6db4-11ee-8094-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-28", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "DL-CRF+H FL (VOC) 47.5 27.6 50.4 63.5 47.7 57.9 38.7 47.2 21.1 57.3 41.2 43.7 36.0 66.4 61.1 21.3 53.9 42.1 70.9 34.6 46.5\nDL-CRF (SBD) 59.4 36.5 58.0 38.6 32.0 58.1 44.7 59.6 25.8 51.8 28.1 59.0 46.9 50.3 61.8 22.2 45.9 33.4 62.1 41.0 45.8\nDL-CRF+H FL (SBD) 63.4 42.5 58.4 41.3 32.5 61.2 45.7 61.4 28.4 55.5 31.5 61.4 51.8 54.6 62.1 24.9 52.6 34.2 67.1 45.1 48.8\nTable 4: Semantic segmentation results on the SBD and VOC 2007 datasets. We measure the results according to PP-IOU\n(per pixel) and PI-IOU (per image) evaluation metrics. We denote the original DeepLab-CRF system and our proposed"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8db8-6db4-11ee-a1fc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-29", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "(per pixel) and PI-IOU (per image) evaluation metrics. We denote the original DeepLab-CRF system and our proposed\nmodi\ufb01cation as DL-CRF and DL-CRF+H FL, respectively. According to the PP-IOU metric, our proposed features (DLCRF+H FL) yield almost equivalent results as the original DeepLab-CRF system. However, based on PI-IOU metric, our\nproposed features improve the mean accuracy by 3%and1:9%on SBD and VOC 2007 datasets respectively.\nity matrix connecting every pair of pixels iandjthat have\ndistance 5or less from each other.\nAfter building a boundary-based af\ufb01nity matrix Wwe\nsetDii=P\ni6=jWijand compute eigenvectors vof the\ngeneralized eigenvalue system:\n(D\u0000W)v=\u0015Dv\nWe then resize the eigenvectors vto the original image\ndimensions, and use them as additional features to the CRF\npart of DeepLab-CRF system. In our experiments, we use\nthe16eigenvectors corresponding to the smallest eigenvalues, which results in 16extra feature channels.\nNote that the eigenvectors contain soft segmentation information. Because H FL boundaries predict object-level"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8db9-6db4-11ee-91d8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-30", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "Note that the eigenvectors contain soft segmentation information. Because H FL boundaries predict object-level\ncontours with high con\ufb01dence, the eigenvectors often capture regions corresponding to objects. We visualize a few\nselected eigenvectors in Figure 8. In the experimental section, we demonstrate that our proposed features make the\noutput produced by DeepLab-CRF more spatially coherent\nand improve the segmentation accuracy according to one of\nthe metrics.\nWe also note that our proposed features are applicable to any generic method that incorporates CRF. For instance, even if DeepLab-CRF used an improved DeepLab\nnetwork architecture, our features would still be bene\ufb01cial\nbecause they contribute directly to the CRF part and not the\nDeepLab network part of the system.\n4.2.1 Semantic Segmentation Results\nIn this section, we present semantic segmentation results\non the SBD [11] and also Pascal VOC 2007 [8] datasets,\nwhich both provide ground truth segmentations for 20Pascal VOC classes. We evaluate the results in terms of two\nmetrics. The \ufb01rst metric measures the accuracy in terms of\npixel intersection-over-union averaged per pixel (PP-IOU)\nacross the 20 classes. According to this metric, the accuracy\nis computed on a per pixel basis. As a result, the images that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dba-6db4-11ee-bd51-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-31", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "across the 20 classes. According to this metric, the accuracy\nis computed on a per pixel basis. As a result, the images that\ncontain large object regions are given more importance.\nFigure 8: In this \ufb01gure, the \ufb01rst column depicts an input\nimage while the second and third columns illustrate two selected eigenvectors for that image. The eigenvectors contain\nsoft segmentation information. Because H FL boundaries\ncapture object-level boundaries, the resulting eigenvectors\nprimarily segment regions corresponding to the objects.\nWe observe that while DeepLab-CRF works well on the\nimages containing large object regions, it produces spatially\ndisjoint outputs for the images with smaller and object regions (see Figure 9). This issue is often being overlooked,\nbecause according to the PP-IOU metric, the images with\nlarge object regions are given more importance and thus\ncontribute more to the accuracy. However, certain applications may require accurate segmentation of small objects.\nTherefore, in addition to PP-IOU, we also consider the PIIOU metric (pixel intersection-over-union averaged per image across the 20 classes), which gives equal weight to each\nof the images.\nFor both of the metrics we compare the semantic segmentation results of a pure DeepLab-CRF [5] and also a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dbb-6db4-11ee-879f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-32", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "of the images.\nFor both of the metrics we compare the semantic segmentation results of a pure DeepLab-CRF [5] and also a\nmodi\ufb01cation of DeepLab-CRF with our proposed features\nadded to the CRF framework. We present the results for\nboth of the metrics in Table 4.\nBased on these results, we observe that according to\nthe \ufb01rst metric (PP-IOU), our proposed features yield almost equivalent results as the original DeepLab-CRF system. However, according to the second metric (PI-IOU) our\nfeatures yield an average improvement of 3%and1:9%in\nFigure 9: An illustration of the more challenging semantic\nsegmentation examples. The \ufb01rst column depicts the predictions achieved by DeepLab-CRF, while the second column illustrates the results after adding our proposed features to the CRF framework. The last column represents\nground truth segmentations. Notice how our proposed features render the predicted semantic segments more spatially\ncoherent and overall more accurate.\nSBD and VOC 2007 datasets respectively.\nWe also visualize the qualitative results produced by both\napproaches in Figure 9. Notice how our proposed features\nmake the segmentations look smoother relative to the segmentations produced by the original DeepLab-CRF system.\nOnce again, we want to stress that our H FL features"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dbc-6db4-11ee-a28e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-33", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "make the segmentations look smoother relative to the segmentations produced by the original DeepLab-CRF system.\nOnce again, we want to stress that our H FL features\nare applicable to any method that uses the CRF. Therefore,\nbased on the results presented in this section, we believe that\nour proposed features could be bene\ufb01cial in a wide array of\nproblems that involve the use of the CRF framework.\n4.3. Object Proposals\nFinally, we show that our method produces object-level\nboundaries that can be successfully exploited in an object\nproposal scheme. Speci\ufb01cally we adopt the EdgeBoxes approach [31], which can be applied to any generic boundaries to produce a list of object proposal boxes. The original EdgeBoxes method uses SE boundaries to generate the\nboxes. However, SE boundaries are predicted using lowlevel color and texture features, rather than object-level\nfeatures. Thus, here we validate the hypothesis that the\nEdgeBoxes proposals can be improved by replacing the SE\nboundaries with our H FL boundaries.\n4.3.1 Object Proposal Results\nIn this section, we present object proposal results on the\nPascal VOC 2012 dataset [9]. We evaluate the quality of\nbounding-box proposals according to three metrics: area\nunder the curve (AUC), the number of proposals needed to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dbd-6db4-11ee-8c58-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-34", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "Pascal VOC 2012 dataset [9]. We evaluate the quality of\nbounding-box proposals according to three metrics: area\nunder the curve (AUC), the number of proposals needed to\nreach recall of 75%, and the maximum recall over 5000 object bounding-boxes. Additionally, we compute the accuracy for each of the metrics for three different intersectionMethodIoU 0.65 IoU 0.7 IoU 0.75\nAUC N@75% Recall AUC N@75% Recall AUC N@75% Recall\nSE 0:52 413 0.93 0:47 658 0.88 0.41 inf 0:75\nHFL 0.53 365 0.95 0.48 583 0.9 0.41 2685 0.77\nTable 5: Comparison of object proposal results. We\ncompare the quality of object proposals using Structured\nEdges [6] and H FL boundaries. We evaluate the performance for three different IOU values and demonstrate that\nusing H FL boundaries produces better results for each evaluation metric and for each IOU value.\nover union (IOU) values: 0:65;0:7, and 0:75. We present\nthese results in Table 5. As described in Section 4.3, we use\nEdgeBoxes [31], a package that uses generic boundaries, to\ngenerate object proposals. We compare the quality of the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dbe-6db4-11ee-9149-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-35", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "EdgeBoxes [31], a package that uses generic boundaries, to\ngenerate object proposals. We compare the quality of the\ngenerated object proposals when using SE boundaries and\nHFL boundaries. We demonstrate that for each IOU value\nand for each of the three evaluation metrics, H FL boundaries produce better or equivalent results. This con\ufb01rms our\nhypothesis that H FL boundaries can be used effectively for\nhigh-level vision tasks such as generating object proposals.\n5. Conclusions\nIn this work, we presented an ef\ufb01cient architecture that\nuses object-level information to predict semantically meaningful boundaries. Most prior edge detection methods rely\nexclusively on low-level features, such as color or texture, to\ndetect the boundaries. However, perception studies suggest\nthat humans employ object-level reasoning when deciding\nwhether a given pixel is a boundary [13, 25, 17]. Thus,\nwe propose a system that focuses on the semantic objectlevel cues rather than low level image information to detect\nthe boundaries. For this reason we refer to our boundary\ndetection scheme as a High-for-Low approach, where highlevel object features inform the low-level boundary detection process. In this paper we demonstrated that our proposed method produces boundaries that accurately separate\nobjects and the background in the image and also achieve\nhigher F-score compared to any prior work."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dbf-6db4-11ee-945b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-36", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "objects and the background in the image and also achieve\nhigher F-score compared to any prior work.\nAdditionally, we showed that, because H FL boundaries\nare based on object-level features, they can be employed to\naid a number of high level vision tasks in a Low-for-High\nfashion. We use our boundaries to boost the accuracy of\nstate-of-the-art methods on the high-level vision tasks of semantic boundary labeling, semantic segmentation, and object proposals generation. We show that using H FL boundaries leads to better results in each of these tasks.\nTo conclude, our boundary detection method is accurate,\nef\ufb01cient, applicable to a variety of datasets, and also useful\nfor multiple high-level vision tasks. We plan to release the\nsource code for H FL upon the publication of the paper .\n6. Acknowledgements\nWe thank Mohammad Haris Baig for the suggestions and\nhelp with the software. This research was funded in part by\nNSF award CNS-1205521.\nReferences\n[1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour\ndetection and hierarchical image segmentation. IEEE Trans.\nPattern Anal. Mach. Intell. , 33(5):898\u2013916, May 2011. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dc0-6db4-11ee-aa39-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-37", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "detection and hierarchical image segmentation. IEEE Trans.\nPattern Anal. Mach. Intell. , 33(5):898\u2013916, May 2011. 2\n[2] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marqu\u00e9s, and J. Malik. Multiscale combinatorial grouping. In Computer Vision\nand Pattern Recognition (CVPR) , 2014. 2, 4\n[3] G. Bertasius, J. Shi, and L. Torresani. Deepedge: A multiscale bifurcated deep network for top-down contour detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015. 2, 3, 4, 5\n[4] C. J. C. Burges. A tutorial on support vector machines for\npattern recognition. Data Mining and Knowledge Discovery ,\n2:121\u2013167, 1998. 2\n[5] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L.\nYuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. CoRR , abs/1412.7062,\n2014. 1, 5, 6, 7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dc1-6db4-11ee-b9f2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-38", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "2014. 1, 5, 6, 7\n[6] P. Doll\u00e1r and C. L. Zitnick. Fast edge detection using structured forests. PAMI , 2015. 2, 3, 4, 5, 8\n[7] J. Donahue, Y . Jia, O. Vinyals, J. Hoffman, N. Zhang,\nE. Tzeng, and T. Darrell. Decaf: A deep convolutional\nactivation feature for generic visual recognition. CoRR ,\nabs/1310.1531, 2013. 5\n[8] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,\nand A. Zisserman. The PASCAL Visual Object Classes\nChallenge 2007 (VOC2007) Results. http://www.pascalnetwork.org/challenges/VOC/voc2007/workshop/index.html.\n7\n[9] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,\nand A. Zisserman. The PASCAL Visual Object Classes\nChallenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.\n8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dc2-6db4-11ee-8079-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-39", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.\n8\n[10] Y . Ganin and V . S. Lempitsky. N4-\ufb01elds: Neural network\nnearest neighbor \ufb01elds for image transforms. ACCV , 2014.\n2, 3, 4, 5\n[11] B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik.\nSemantic contours from inverse detectors. In International\nConference on Computer Vision (ICCV) , 2011. 1, 5, 6, 7\n[12] B. Hariharan, P. A. Arbel\u00e1ez, R. B. Girshick, and J. Malik.\nHypercolumns for object segmentation and \ufb01ne-grained localization. CoRR , abs/1411.5752, 2014. 3\n[13] P.-J. Hsieh, E. Vul, and N. Kanwisher. Recognition alters the\nspatial pattern of fmri activation in early retinotopic cortex.\nJournal of Neurophysiology , 103(3):1501\u20131507, 2010. 1, 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dc3-6db4-11ee-80c4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-40", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "spatial pattern of fmri activation in early retinotopic cortex.\nJournal of Neurophysiology , 103(3):1501\u20131507, 2010. 1, 8\n[14] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Crisp\nboundary detection using pointwise mutual information. In\nECCV , 2014. 2, 4[15] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint\narXiv:1408.5093 , 2014. 4\n[16] J. J. Kivinen, C. K. Williams, N. Heess, and D. Technologies. Visual boundary prediction: A deep neural prediction\nnetwork and quality dissection. AISTATS , 1(2):9, 2014. 2, 3,\n4, 5\n[17] Z. Kourtzi and N. Kanwisher. Representation of perceived\nobject shape by the human lateral occipital complex. Science, 293:1506\u20131509, 2001. 1, 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dc4-6db4-11ee-90bf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-41", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "object shape by the human lateral occipital complex. Science, 293:1506\u20131509, 2001. 1, 8\n[18] J. Lim, C. L. Zitnick, and P. Doll\u00e1r. Sketch tokens: A learned\nmid-level representation for contour and object detection. In\nCVPR , 2013. 2\n[19] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional\nnetworks for semantic segmentation. CVPR (to appear) ,\nNov. 2015. 3, 5\n[20] M. Maire, S. X. Yu, and P. Perona. Reconstructive sparse\ncode transfer for contour detection and semantic labeling. In\nAsian Conference on Computer Vision (ACCV) , 2014. 2\n[21] T. Malisiewicz, A. Gupta, and A. A. Efros. Ensemble of\nexemplar-svms for object detection and beyond. In ICCV ,\n2011. 4\n[22] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database\nof human segmented natural images and its application to\nevaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int\u2019l Conf. Computer Vision , volume 2, pages 416\u2013423, July 2001. 4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dc5-6db4-11ee-9d44-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-42", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int\u2019l Conf. Computer Vision , volume 2, pages 416\u2013423, July 2001. 4\n[23] X. Ren and L. Bo. Discriminatively Trained Sparse Code\nGradients for Contour Detection. In Advances in Neural Information Processing Systems , December 2012. 2, 4\n[24] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,\nS. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,\nA. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual\nRecognition Challenge, 2014. 3\n[25] J. L. Sanguinetti, J. J. Allen, and M. A. Peterson. The ground\nside of an object perceived as shapeless yet processed for\nsemantics. Psychological science , page 0956797613502814,\n2013. 1, 8\n[26] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,\nand Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR ,\nabs/1312.6229, 2013. 3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dc6-6db4-11ee-9b86-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1504.06201-43", "title": "High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision", "source": "http://arxiv.org/pdf/1504.06201", "start_index": 0, "chroma:document": "and Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR ,\nabs/1312.6229, 2013. 3\n[27] W. Shen, X. Wang, Y . Wang, X. Bai, and Z. Zhang. Deepcontour: A deep convolutional feature learned by positivesharing loss for contour detection. June 2015. 1, 3, 4, 5\n[28] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine\nIntelligence , 22:888\u2013905, 1997. 2, 6\n[29] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR ,\nabs/1409.1556, 2014. 1\n[30] A. Sironi, V . Lepetit, and P. Fua. Multiscale centerline detection by learning a scale-space distance transform. June 2014.\n4\n[31] C. L. Zitnick and P. Doll\u00e1r. Edge boxes: Locating object\nproposals from edges. In ECCV , 2014. 1, 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dc7-6db4-11ee-8f73-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-0", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "Semantic Segmentation with Boundary Neural Fields\nGedas Bertasius\nUniversity of Pennsylvania\ngberta@seas.upenn.eduJianbo Shi\nUniversity of Pennsylvania\njshi@seas.upenn.eduLorenzo Torresani\nDartmouth College\nlt@dartmouth.edu\nAbstract\nThe state-of-the-art in semantic segmentation is currently represented by fully convolutional networks (FCNs).\nHowever, FCNs use large receptive \ufb01elds and many pooling layers, both of which cause blurring and low spatial\nresolution in the deep layers. As a result FCNs tend to produce segmentations that are poorly localized around object\nboundaries. Prior work has attempted to address this issue\nin post-processing steps, for example using a color-based\nCRF on top of the FCN predictions. However, these approaches require additional parameters and low-level features that are dif\ufb01cult to tune and integrate into the original\nnetwork architecture. Additionally, most CRFs use colorbased pixel af\ufb01nities, which are not well suited for semantic\nsegmentation and lead to spatially disjoint predictions.\nTo overcome these problems, we introduce a Boundary"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dc8-6db4-11ee-bce7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-1", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "segmentation and lead to spatially disjoint predictions.\nTo overcome these problems, we introduce a Boundary\nNeural Field (BNF), which is a global energy model integrating FCN predictions with boundary cues. The boundary information is used to enhance semantic segment coherence and to improve object localization. Speci\ufb01cally, we\n\ufb01rst show that the convolutional \ufb01lters of semantic FCNs\nprovide good features for boundary detection. We then employ the predicted boundaries to de\ufb01ne pairwise potentials\nin our energy. Finally, we show that our energy decomposes semantic segmentation into multiple binary problems,\nwhich can be relaxed for ef\ufb01cient global optimization. We\nreport extensive experiments demonstrating that minimization of our global boundary-based energy yields results superior to prior globalization methods, both quantitatively\nas well as qualitatively.\n1. Introduction\nThe recent introduction of fully convolutional networks\n(FCNs) [22] has led to signi\ufb01cant quantitative improvements on the task of semantic segmentation. However, despite their empirical success, FCNs suffer from some limitations. Large receptive \ufb01elds in the convolutional layers and\nthe presence of pooling layers lead to blurring and segmentation predictions at a signi\ufb01cantly lower resolution than the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dc9-6db4-11ee-8038-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-2", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "the presence of pooling layers lead to blurring and segmentation predictions at a signi\ufb01cantly lower resolution than the\nFigure 1: Examples illustrating shortcomings of prior semantic segmentation methods: the second column shows\nresults obtained with a FCN [22], while the third column\nshows the output of a Dense-CRF applied to FCN predictions [19, 7]. Segments produced by FCN are blob-like and\nare poorly localized around object boundaries. Dense-CRF\nproduces spatially disjoint object segments due to the use of\na color-based pixel af\ufb01nity function that is unable to measure semantic similarity between pixels.\noriginal image. As a result, their predicted segments tend to\nbe blobby and lack \ufb01ne object boundary details. We report\nin Fig. 1 some examples illustrating typical poor localization of objects in the outputs of FCNs.\nRecently, Chen at al. [7] addressed this issue by applying a Dense-CRF post-processing step [19] on top of coarse\nFCN segmentations. However, such an approach introduces\nseveral problems of its own. First, the Dense-CRF adds new\nparameters that are dif\ufb01cult to tune and integrate into the\noriginal network architecture. Additionally, most methods"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dca-6db4-11ee-9b55-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-3", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "parameters that are dif\ufb01cult to tune and integrate into the\noriginal network architecture. Additionally, most methods\nbased on CRFs or MRFs use low-level pixel af\ufb01nity functions, such as those based on color. These low-level af\ufb01nities often fail to capture semantic relationships between objects and lead to poor segmentation results (see last column\nin Fig. 1).\nWe propose to address these shortcomings by means of\na Boundary Neural Field (BNF), an architecture that employs a single semantic segmentation FCN to predict semantic boundaries and then use them to produce semantic\nsegmentation maps via a global optimization. We demonstrate that even though the semantic segmentation FCN has\nnot been optimized to detect boundaries, it provides good\n1arXiv:1511.02674v2  [cs.CV]  24 May 2016\n...\nConvolutional Feature Maps RGB Input Image\nSegmentation Mask in       Softmax Layer\n   Minimize Global Energy\nBoundaries\n|{z}...\nwn()++()w1\nXi\u2713i(zi)+Xij\u2713ij(zi,zj)\nLinear Combination of Feature MapsFigure 2: The architecture of our system (best viewed in\ncolor). We employ a semantic segmentation FCN [7] for\ntwo purposes: 1) to obtain semantic segmentation unaries"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dcb-6db4-11ee-8476-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-4", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "color). We employ a semantic segmentation FCN [7] for\ntwo purposes: 1) to obtain semantic segmentation unaries\nfor our global energy; 2) to compute object boundaries.\nSpeci\ufb01cally, we de\ufb01ne semantic boundaries as a linear combination of these feature maps (with a sigmoid function applied on top of the sum) and learn individual weights corresponding to each convolutional feature map. We integrate\nthis boundary information in the form of pairwise potentials\n(pixel af\ufb01nities) for our energy model.\nfeatures for boundary detection. Speci\ufb01cally, the contributions of our work are as follows:\n\u000fWe show that semantic boundaries can be expressed\nas a linear combination of interpolated convolutional\nfeature maps inside an FCN. We introduce a boundary\ndetection method that exploits this intuition to predict\nobject boundaries with accuracy superior to the statethe-of-art.\n\u000fWe demonstrate that boundary-based pixel af\ufb01nities\nare better suited for semantic segmentation than the\ncommonly used color af\ufb01nity functions.\n\u000fFinally, we introduce a new global energy that decomposes semantic segmentation into multiple binary\nproblems and relaxes the integrality constraint. We\nshow that minimizing our proposed energy yields better qualitative and quantitative results relative to traditional globalization models such as MRFs or CRFs."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dcc-6db4-11ee-b34a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-5", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "problems and relaxes the integrality constraint. We\nshow that minimizing our proposed energy yields better qualitative and quantitative results relative to traditional globalization models such as MRFs or CRFs.\n2. Related Work\nBoundary Detection. Spectral methods comprise one\nof the most prominent categories for boundary detection.\nIn a typical spectral framework, one formulates a generalized eigenvalue system to solve a low-level pixel groupingproblem. The resulting eigenvectors are then used to predict the boundaries. Some of the most notable approaches\nin this genre are MCG [2], gPb [1], PMI [17], and Normalized Cuts [29]. A weakness of spectral approaches is that\nthey tend to be slow as they perform a global inference over\nthe entire image.\nTo address this issue, recent approaches cast boundary\ndetection as a classi\ufb01cation problem and predict the boundaries in a local manner with high ef\ufb01ciency. The most notable examples in this genre include sketch tokens (ST) [20]\nand structured edges (SE) [9], which employ fast random\nforests. However, many of these methods are based on\nhand-constructed features, which are dif\ufb01cult to tune."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dcd-6db4-11ee-87d6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-6", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "forests. However, many of these methods are based on\nhand-constructed features, which are dif\ufb01cult to tune.\nThe issue of hand-constructed features have been recently addressed by several approaches based on deep learning, such as N4\ufb01elds [11], DeepNet [18], DeepContour [27], DeepEdge [3], H FL [4] and HED [33]. All of\nthese methods use CNNs in some way to predict the boundaries. Whereas DeepNet and DeepContour optimize ordinary CNNs to a boundary based optimization criterion from\nscratch, DeepEdge and H FL employ pretrained models to\ncompute boundaries. The most recent of these methods is\nHED [33], which shows the bene\ufb01t of deeply supervised\nlearning for boundary detection.\nIn comparison to prior deep learning approaches, our\nmethod offers several contributions. First, we exploit the inherent relationship between boundary detection and semantic segmentation to predict semantic boundaries. Specifically, we show that even though the semantic FCN has\nnot been explicitly trained to predict boundaries, the convolutional \ufb01lters inside the FCN provide good features for\nboundary detection. Additionally, unlike DeepEdge [3] and\nHFL [4], our method does not require a pre-processing step\nto select candidate contour points, as we predict boundaries"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dce-6db4-11ee-bde0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-7", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "HFL [4], our method does not require a pre-processing step\nto select candidate contour points, as we predict boundaries\non all pixels in the image. We demonstrate that our approach allows us to achieve state-of-the-art boundary detection results according to both F-score and Average Precision metrics. Additionally, due to the semantic nature of\nour boundaries, we can successfully use them as pairwise\npotentials for semantic segmentation in order to improve\nobject localization and recover \ufb01ne structural details, typically lost by pure FCN-based approaches.\nSemantic Segmentation. We can group most semantic segmentation methods into three broad categories. The\n\ufb01rst category can be described as \u201ctwo-stage\u201d approaches,\nwhere an image is \ufb01rst segmented and then each segment\nis classi\ufb01ed as belonging to a certain object class. Some\nof the most notable methods that belong to this genre include [24, 6, 12, 14].\nThe primary weakness of the above methods is that they\nare unable to recover from errors made by the segmentation\nalgorithm. Several recent papers [15, 10] address this issue\nby proposing to use deep per-pixel CNN features and then\nclassify each pixel as belonging to a certain class. While\nthese approaches partially address the incorrect segmentation problem, they perform predictions independently on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dcf-6db4-11ee-a9eb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-8", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "classify each pixel as belonging to a certain class. While\nthese approaches partially address the incorrect segmentation problem, they perform predictions independently on\neach pixel. This leads to extremely local predictions, where\nthe relationships between pixels are not exploited in any\nway, and thus the resulting segmentations may be spatially\ndisjoint.\nThe third and \ufb01nal group of semantic segmentation\nmethods can be viewed as front-to-end schemes where segmentation maps are predicted directly from raw pixels without any intermediate steps. One of the earliest examples of\nsuch methods is the FCN introduced in [22]. This approach\ngave rise to a number of subsequent related approaches\nwhich have improved various aspects of the original semantic segmentation [7, 34, 8, 16, 21]. There have also been attempts at integrating the CRF mechanism into the network\narchitecture [7, 34]. Finally, it has been shown that semantic\nsegmentation can also be improved using additional training\ndata in the form of bounding boxes [8].\nOur BNF offers several contributions over prior work. To\nthe best of our knowledge, we are the \ufb01rst to present a model\nthat exploits the relationship between boundary detection\nand semantic segmentation within a FCN framework . We\nintroduce pairwise pixel af\ufb01nities computed from semantic boundaries inside an FCN, and use these boundaries to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dd0-6db4-11ee-85cd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-9", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "and semantic segmentation within a FCN framework . We\nintroduce pairwise pixel af\ufb01nities computed from semantic boundaries inside an FCN, and use these boundaries to\npredict the segmentations in a global fashion. Unlike [21],\nwhich requires a large number of additional parameters to\nlearn for the pairwise potentials, our global model only\nneeds\u00195Kextra parameters, which is about 3orders of\nmagnitudes less than the number of parameters in a typical deep convolutional network (e.g. VGG [30]). We empirically show that our proposed boundary-based af\ufb01nities\nare better suited for semantic segmentation than color-based\naf\ufb01nities. Additionally, unlike in [7, 34, 21], the solution to\nour proposed global energy can be obtained in closed-form,\nwhich makes global inference easier. Finally we demonstrate that our method produces better results than traditional globalization models such as CRFs or MRFs.\n3. Boundary Neural Fields\nIn this section, we describe Boundary Neural Fields.\nSimilarly to traditional globalization methods, Boundary\nNeural Fields are de\ufb01ned by an energy including unary\nand pairwise potentials. Minimization of the global energy yields the semantic segmentation. BNFs build both\nunary and pairwise potentials from the input RGB image"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dd1-6db4-11ee-ab15-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-10", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "and pairwise potentials. Minimization of the global energy yields the semantic segmentation. BNFs build both\nunary and pairwise potentials from the input RGB image\nand then combine them in a global manner. More precisely,\nthe coarse segmentations predicted by a semantic FCN are\nused to de\ufb01ne the unary potentials of our BNF. Next, we\nshow that the convolutional feature maps of the FCN can\nbe used to accurately predict semantic boundaries. These\nboundaries are then used to build pairwise pixel af\ufb01nities,\nwhich are used as pairwise potentials by the BNF. Finally,we introduce a global energy function, which minimizes the\nenergy corresponding to the unary and pairwise terms and\nimproves the initial FCN segmentation. The detailed illustration of our architecture is presented in Figure 2. We now\nexplain each of these steps in more detail.\n3.1. FCN Unary Potentials\nTo predict semantic unary potentials we employ the\nDeepLab model [7], which is a fully convolutional adaptation of the VGG network [30]. The FCN consists of 16convolutional layers and 3fully convolutional layers. There are\nmore recent FCN-based methods that have demonstrated\neven better semantic segmentation results [8, 34, 16, 21]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dd2-6db4-11ee-986a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-11", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "more recent FCN-based methods that have demonstrated\neven better semantic segmentation results [8, 34, 16, 21].\nAlthough these more advanced architectures could be integrated into our framework to improve our unary potentials, in this work we focus on two aspects orthogonal to\nthis prior work: 1) demonstrating that our boundary-based\naf\ufb01nity function is better suited for semantic segmentation\nthan the common color-based af\ufb01nities and 2) showing that\nour proposed global energy achieves better qualitative and\nquantitative semantic segmentation results in comparison to\nprior globalization models.\n3.2. Boundary Pairwise Potentials\nIn this section, we describe our approach for building\npairwise pixel af\ufb01nities using semantic boundaries. The basic idea behind our boundary detection approach is to express semantic boundaries as a function of convolutional\nfeature maps inside the FCN. Due to the close relationship\nbetween the tasks of semantic segmentation and boundary\ndetection, we hypothesize that convolutional feature maps\nfrom the semantic segmentation FCN can be employed as\nfeatures for boundary detection.\n3.2.1 Learning to Predict Semantic Boundaries.\nWe propose to express semantic boundaries as a linear combination of interpolated FCN feature maps with a non-linear\nfunction applied on top of this sum. We note that interpolation of feature maps has been successfully used in prior"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dd3-6db4-11ee-b516-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-12", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "function applied on top of this sum. We note that interpolation of feature maps has been successfully used in prior\nwork (see e.g. [15]) in order to obtain dense pixel-level features from the low-resolution outputs of deep convolutional\nlayers. Here we adopt interpolation to produce pixel-level\nboundary predictions. There are several advantages to our\nproposed formulation. First, because we express boundaries\nas a linear combination of feature maps, we only need to\nlearn a small number of parameters, corresponding to the\nindividual weight values of each feature map in the FCN.\nThis amounts to\u00195Klearning parameters, which is much\nsmaller than the number of parameters in the entire network\n(\u001915M). In comparison, DeepEdge [3] and HFL [4] need\n17M and 6M additional parameters to predict boundaries.\nFurthermore, expressing semantic boundaries as a linear\ncombination of FCN feature maps allows us to ef\ufb01ciently\npredict boundary probabilities for all pixels in the image\n(we resize the FCN feature maps to the original image dimensions). This eliminates the need to select candidate\nboundary points in a pre-processing stage, which was instead required in prior boundary detection work [3, 4].\nOur boundary prediction pipeline can be described as"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dd4-6db4-11ee-87f2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-13", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "boundary points in a pre-processing stage, which was instead required in prior boundary detection work [3, 4].\nOur boundary prediction pipeline can be described as\nfollows. First we use use SBD segmentations [13] to optimize our FCN for semantic segmentation task. We then\ntreat FCN convolutional maps as features for the boundary detection task and use the boundary annotations from\nBSDS 500 dataset [23] to learn the weights for each feature\nmap. BSDS 500 dataset contains 200training, 100validation, 200testing images, and ground truth annotations by 5\nhuman labelers for each of these images.\nTo learn the weights corresponding to each convolutional\nfeature map we \ufb01rst sample 80Kpoints from the dataset.\nWe de\ufb01ne the target labels for each point as the fraction of\nhuman annotators agreeing on that point being a boundary.\nTo \ufb01x the issue of label imbalance (there are many more\nnon-boundaries than boundaries), we divide the label space\ninto four quartiles, and select an equal number of samples\nfor each quartile to balance the training dataset. Given these\nsampled points, we then de\ufb01ne our features as the values in\nthe interpolated convolutional feature maps corresponding\nto these points. To predict semantic boundaries we weigh\neach convolutional feature map by its weight, sum them up"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dd5-6db4-11ee-9326-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-14", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "the interpolated convolutional feature maps corresponding\nto these points. To predict semantic boundaries we weigh\neach convolutional feature map by its weight, sum them up\nand apply a sigmoid function on top of it. We obtain the\nweights corresponding to each convolutional feature map by\nminimizing the cross-entropy loss using a stochastic batch\ngradient descent for 50epochs. To obtain crisper boundaries at test-time we post-process the boundary probabilities\nusing non-maximum suppression.\nTo give some intuition on how FCN feature maps contribute to boundary detection, in Fig. 3 we visualize the feature maps corresponding to the highest weight magnitudes.\nIt is clear that many of these maps contain highly localized\nboundary information.\nBoundary Detection Results Before discussing how\nboundary information is integrated in our energy for semantic segmentation, here we present experimental results assessing the accuracy of our boundary detection scheme. We\ntested our boundary detector on the BSDS500 dataset [23],\nwhich is the standard benchmark for boundary detection.\nThe quality of the predicted boundaries is evaluated using\nthree standard measures: \ufb01xed contour threshold (ODS),\nper-image best threshold (OIS), and average precision (AP).\nIn Table 1 we show that our algorithm outperforms all\nprior methods according to both F-score measures and the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dd6-6db4-11ee-adec-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-15", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "per-image best threshold (OIS), and average precision (AP).\nIn Table 1 we show that our algorithm outperforms all\nprior methods according to both F-score measures and the\nAverage Precision metric. In Fig. 4, we also visualize our\npredicted boundaries. The second column shows the pixellevel softmax output computed from the linear combination of feature maps, while the third column depicts our \ufb01Figure 3: An input image and convolutional feature maps\ncorresponding to the largest weight magnitude values. Intuitively these are the feature maps that contribute most heavily to the task of boundary detection.\nMethod ODS OIS AP\nSCG [25] 0.739 0.758 0.773\nSE [9] 0.746 0.767 0.803\nMCG [2] 0.747 0.779 0.759\nN4-\ufb01elds [11] 0.753 0.769 0.784\nDeepEdge [3] 0.753 0.772 0.807\nDeepContour [27] 0.756 0.773 0.797\nHFL [4] 0.767 0.788 0.795\nHED [33] 0.782 0.804 0.833\nBNF 0.788 0.807 0.851"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dd7-6db4-11ee-9118-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-16", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "HED [33] 0.782 0.804 0.833\nBNF 0.788 0.807 0.851\nTable 1: Boundary detection results on BSDS500 benchmark. Our proposed method outperforms all prior algorithms according to all three evaluation metrics.\nnal boundaries after applying a non-maximum suppression\npost-processing step.\nWe note that our predicted boundaries achieve highcon\ufb01dence predictions around objects. This is important as\nwe employ these boundaries to improve semantic segmentation results, as discussed in the next subsection.\n3.2.2 Constructing Pairwise Pixel Af\ufb01nities.\nWe can use the predicted boundaries to build pairwise pixel\naf\ufb01nities. Intuitively, we declare two pixels as similar (i.e.,\nlikely to belong to the same segment) if there is no boundary crossing the straight path between these two pixels.\nConversely, two pixels are dissimilar if there is a boundary crossing their connecting path. The larger the boundary\nmagnitude of the crossed path, the more dissimilar the two\npixels should be, since a strong boundary is likely to mark\nthe separation of two distinct segments. Similarly to [1], we\nencode this intuition with a following formulation:\nwsb\nij= exp (\u0000Mij\n\u001bsb) (1)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dd8-6db4-11ee-96b7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-17", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "encode this intuition with a following formulation:\nwsb\nij= exp (\u0000Mij\n\u001bsb) (1)\nFigure 4: A \ufb01gure illustrating our boundary detection results. In the second column, we visualize the raw probability output of our boundary detector. In the third column,\nwe present the \ufb01nal boundary maps after non-maximum\nsuppression. While most prior methods predict the boundaries where the sharpest change in color occurs, our method\ncaptures semantic object-level boundaries, which we subsequently use to aid semantic segmentation.\nwhereMijdenotes the maximum boundary value that\ncrosses the straight line path between pixels iandj,\u001bsbdepicts the smoothing parameter and wsb\nijdenotes the semantic\nboundary-based af\ufb01nity between pixels iandj.\nSimilarly, we want to exploit high-level object information in the network to de\ufb01ne another type of pixel similarity.\nSpeci\ufb01cally, we use object class probabilities from the softmax (SM) layer to achieve this goal. Intuitively, if pixels i\nandjhave different hard segmentation labels from the softmax layer, we set their similarity ( wsm\nij) to0. Otherwise,\nwe compute their similarity using the following equation:\nwsm\nij= exp (\u0000Dij\n\u001bsm) (2)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dd9-6db4-11ee-8aed-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-18", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "ij) to0. Otherwise,\nwe compute their similarity using the following equation:\nwsm\nij= exp (\u0000Dij\n\u001bsm) (2)\nwhereDijdenotes the difference in softmax output values corresponding to the most likely object class for pixels\niandj, and\u001bsmis a smoothing parameter. Then we can\nwrite the \ufb01nal af\ufb01nity measure as:\nwij= exp (wsm\nij)wsb\nij (3)\nWe exponentiate the term corresponding to the objectlevel af\ufb01nity because our boundary-based af\ufb01nity may be\ntoo aggressive in declaring two pixels as dissimilar. To address this issue, we increase the importance of the objectlevel af\ufb01nity in (3) using the exponential function. However, in the experimental results section, we demonstrate\nthat most of the bene\ufb01t from modeling pairwise potentials\ncomes from wsb\nijrather thanwsm\nij.We then use this pairwise pixel af\ufb01nity measure to build\na global af\ufb01nity matrix Wthat encodes relationships between pixels in the entire image. For a given pixel, we\nsample\u001910% of points in the neighborhood of radius 20\naround that pixel, and store the resulting af\ufb01nities into W."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dda-6db4-11ee-8ad1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-19", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "sample\u001910% of points in the neighborhood of radius 20\naround that pixel, and store the resulting af\ufb01nities into W.\n3.3. Global Inference\nThe last step in our proposed method is to combine semantic boundary information with the coarse segmentation\nfrom the FCN softmax layer to produce an improved segmentation. We do this by introducing a global energy function that utilizes the af\ufb01nity matrix constructed in the previous section along with the segmentation from the FCN\nsoftmax layer. Using this energy, we perform a global inference to get segmentations that are well localized around the\nobject boundaries and that are also spatially smooth.\nTypical globalization models such as MRFs [31],\nCRFs [19] or Graph Cuts [5] produce a discrete label assignment for the segmentation problem by jointly modeling a multi-label distribution and solving a non-convex optimization. The common problem in doing so is that the\noptimization procedure may get stuck in local optima.\nWe introduce a new global energy function, which overcomes this issue and achieves better segmentation in comparison to prior globalization models. Similarly to prior\nglobalization approaches, our goal is to minimize the energy\ncorresponding to the sum of unary and pairwise potentials.\nHowever, the key difference in our approach comes from the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8ddb-6db4-11ee-909b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-20", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "globalization approaches, our goal is to minimize the energy\ncorresponding to the sum of unary and pairwise potentials.\nHowever, the key difference in our approach comes from the\nrelaxation of some of the constraints. Speci\ufb01cally, instead\nof modeling our problem as a joint multi-label distribution,\nwe propose to decompose it into multiple binary problems,\nwhich can be solved concurrently. This decomposition can\nbe viewed as assigning pixels to foreground and background\nlabels for each of the different object classes. Additionally,\nwe relax the integrality constraint. Both of these relaxations\nmake our problem more manageable and allow us to formulate a global energy function that is differentiable, and has\na closed form solution.\nIn [35], the authors introduce the idea of learning\nwith global and local consistency in the context of semisupervised problems. Inspired by this work, we incorporate\nsome of these ideas in the context of semantic segmentation. Before de\ufb01ning our proposed global energy function,\nwe introduce some relevant notation.\nFor the purpose of illustration, suppose that we only have\ntwo classes: foreground and background. Then we can denote an optimal continuous solution to such a segmentation\nproblem with variable z\u0003. To denote similarity between pixelsiandjwe usewij. Then,diindicates the degree of a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8ddc-6db4-11ee-ba55-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-21", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "problem with variable z\u0003. To denote similarity between pixelsiandjwe usewij. Then,diindicates the degree of a\npixeli. In graph theory, the degree of a node denotes the\nnumber of edges incident to that node. Thus, we set the degree of a pixel to di=Pn\nj=1wijfor alljexcepti6=j.\nFinally, with fiwe denote an initial segmentation probaInput\n Softmax\n Dense-CRF\n BNF Boundaries\n BNF Segmentation\nFigure 5: A \ufb01gure illustrating semantic segmentation results. Images in columns two and three represent FCN softmax and\nDense-CRF predictions, respectively. Note that all methods use the same FCN unary potentials. Additionally, observe that\nunlike FCN and Dense-CRF, our methods predicts segmentation that are both well localized around object boundaries and\nthat are also spatially smooth.\nbility, which in our case is obtained from the FCN softmax\nlayer.\nUsing this notation, we can then formulate our global\ninference objective as:\nz\u0003= argmin\nz\u0016\n2X\nidi(zi\u0000fi\ndi)2+1\n2X\nijwij(zi\u0000zj)2(4)\nThis energy consists of two different terms. Similar to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8ddd-6db4-11ee-9e3b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-22", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "idi(zi\u0000fi\ndi)2+1\n2X\nijwij(zi\u0000zj)2(4)\nThis energy consists of two different terms. Similar to\nthe general globalization framework, our \ufb01rst term encodes\nthe unary energy while the second term includes the pairwise energy. We now explain the intuition behind each of\nthese terms. The unary term attempts to \ufb01nd a segmentation\nassignment ( zi) that deviates little from the initial candidate\nsegmentation computed from the softmax layer (denoted by\nfi). Theziin the unary term is weighted by the degree di\nof the pixel in order to produce larger unary costs for pixels\nthat have many similar pixels within the neighborhood. Instead, the pairwise term ensures that pixels that are similar\nshould be assigned similar zvalues. To balance the energies of the two terms we introduce a parameter \u0016and set it\nto0:025throughout all our experiments.\nWe can also express the same global energy function in\nmatrix notation:\nz\u0003= argmin\nz\u0016\n2D(z\u0000D\u00001f)T(z\u0000D\u00001f)+1\n2zT(D\u0000W)z\n(5)\nwhere z\u0003is an\u00021vector containing an optimal continuous assignment for all npixels, Dis a diagonal degree"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dde-6db4-11ee-bd59-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-23", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "2zT(D\u0000W)z\n(5)\nwhere z\u0003is an\u00021vector containing an optimal continuous assignment for all npixels, Dis a diagonal degree\nmatrix, and Wis then\u0002npixel af\ufb01nity matrix. Finally, f\ndenotes an\u00021vector containing the probabilities from the\nsoftmax layer corresponding to a particular object class.\nAn advantage of our energy is that it is differentiable. Ifwe denote the above energy as E(z)then the derivative of\nthis energy can be written as follows:\n@E(z)\n@z=\u0016D(z\u0000D\u00001f) + (D\u0000W)z=0 (6)\nWith simple algebraic manipulations we can then obtain\na closed form solution to this optimization:\nz\u0003= (D\u0000\u000bW)\u00001\ff (7)\nwhere\u000b=1\n1+\u0016and\f=\u0016\n1+\u0016. In the general case where\nwe havekobject classes we can write the solution as:\nZ\u0003= (D\u0000\u000bW)\u00001\fF (8)\nwhere Znow depicts a n\u0002kmatrix containing assignments for all kobject classes, while Fdenotesn\u0002kmatrix\nwith object class probabilities from softmax layer. Due to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8ddf-6db4-11ee-bf7a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-24", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "where Znow depicts a n\u0002kmatrix containing assignments for all kobject classes, while Fdenotesn\u0002kmatrix\nwith object class probabilities from softmax layer. Due to\nthe large size of D\u0000\u000bWit is impractical to invert it. However, if we consider an image as a graph where each pixel\ndenotes a vertex in the graph, we can observe that the term\nD\u0000Win our optimization is equivalent to a Laplacian matrix of such graph. Since we know that a Laplacian matrix is\npositive semi-de\ufb01nite, we can use the preconditioned conjugate gradient method [28] to solve the system in Eq. (9).\nAlternatively, because our de\ufb01ned global energy in Eq. (5)\nis differentiable, we can ef\ufb01ciently solve this optimization\nproblem using stochastic gradient descent. We choose the\nformer option and solve the following system:\n(D\u0000\u000bW)z\u0003=\ff (9)\nTo obtain the \ufb01nal discrete segmentation, for each pixel\nwe assign the object class that corresponds to the largest\ncolumn value in the row of Z(note that each row in Zrepresents a single pixel in the image, and each column in Z"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8de0-6db4-11ee-b70a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-25", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "we assign the object class that corresponds to the largest\ncolumn value in the row of Z(note that each row in Zrepresents a single pixel in the image, and each column in Z\nMetric Inference Method RGB Af\ufb01nity BNF Af\ufb01nity\nPP-IOUBelief Propagation [31] 75.4 75.6\nICM 74.2 75.8\nTRWS [32] 75.9 76.7\nQPBO [26] 76.9 77.2\nBNF 74.6 77.6\nPI-IOUBelief Propagation [31] 45.9 46.2\nICM 45.7 48.8\nTRWS [32] 51.5 52.0\nQPBO [26] 55.3 57.2\nBNF 53.0 58.5\nTable 2: We compare semantic segmentation results\nwhen using a color-based pixel af\ufb01nity and our proposed\nboundary-based af\ufb01nity. We note that our proposed af\ufb01nity\nimproves the performance of all globalization techniques.\nNote that all of the inference methods use the same FCN\nunary potentials . This suggests that for every method our"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8de1-6db4-11ee-aa83-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-26", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "improves the performance of all globalization techniques.\nNote that all of the inference methods use the same FCN\nunary potentials . This suggests that for every method our\nboundary-based af\ufb01nity is more bene\ufb01cial for semantic segmentation than the color-based af\ufb01nity.\nrepresents one of the object classes). In the experimental\nsection, we show that this solution produces better quantitative and qualitative results in comparison to commonly used\nglobalization techniques.\n4. Experimental Results\nIn this section we present quantitative and qualitative results for semantic segmentation on the SBD [13] dataset,\nwhich contains objects and their per-pixel annotations for\n20Pascal VOC classes. We evaluate semantic segmentation results using two evaluation metrics. The \ufb01rst metric\nmeasures accuracy based on pixel intersection-over-union\naveraged per pixels (PP-IOU) across the 20 classes. According to this metric, the accuracy is computed on a perpixel basis. As a result, the images that contain large object regions are given more importance. However, for certain applications we may need to accurately segment small\nobjects. Therefore, similar to [4] we also consider the PIIOU metric (pixel intersection-over-union averaged per image across the 20 classes), which gives equal weight to each\nof the images."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8de2-6db4-11ee-8376-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-27", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "of the images.\nWe compare Boundary Neural Fields with other commonly used global inference methods. These methods include Belief Propagation [31], Iterated Conditional Mode\n(ICM), Graph Cuts [5], and Dense-CRF [19]. Note that in\nall of our evaluations we use the same FCN unary potentials\nfor every model.\nOur evaluations provide evidence for three conclusions:\n\u000fIn Subsection 4.1, we show that our boundary-based\npixel af\ufb01nities are better suited for semantic segmentation than the traditional color-based af\ufb01nities.\n\u000fIn Subsection 4.2, we demonstrate that our global minimization leads to better results than those achieved byother inference schemes.\n\u000fIn Fig. 5, we qualitatively compare the outputs of FCN\nand Dense-CRF to our predicted segmentations. This\ncomparison shows that the BNF segments are better\nlocalized around the object boundaries and that they\nare also spatially smooth.\n4.1. Comparing Af\ufb01nity Functions for Semantic\nSegmentation\nIn Table 2, we consider two global models. Both models use the same unary potentials obtained from the FCN\nsoftmax layer. However, the \ufb01rst model uses the popular\ncolor-based pairwise af\ufb01nities, while the second employs"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8de3-6db4-11ee-8462-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-28", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "softmax layer. However, the \ufb01rst model uses the popular\ncolor-based pairwise af\ufb01nities, while the second employs\nour boundary-based af\ufb01nities. Each of these two models\nis optimized using several inference strategies. The table\nshows that using our boundary based-af\ufb01nity function improves the results of all global inference methods according to both evaluation metrics. Note that we cannot include Dense-CRF [19] in this comparison because it employs an ef\ufb01cient message-passing technique and integrating our af\ufb01nities into this technique is a non-trivial task.\nHowever, we compare our method with Dense-CRF in Subsection 4.2.\nThe results in Table 2 suggest that our semantic boundary based pixel af\ufb01nity function yields better semantic segmentation results compared to the commonly-used color\nbased af\ufb01nities. We note that we also compared the results\nof our inference technique using other edge detectors, notably UCM [1] and H FL [4]. In comparison to UCM edges,\nwe observed that our boundaries provide 1:0%and6:0%\naccording to both evaluation metrics respectively. When\ncomparing our boundaries with H FL method, we observed\nsimilar segmentation performance, which suggests that our\nmethod works best with the high quality semantic boundaries."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8de4-6db4-11ee-abf4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-29", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "comparing our boundaries with H FL method, we observed\nsimilar segmentation performance, which suggests that our\nmethod works best with the high quality semantic boundaries.\n4.2. Comparing Inference Methods for Semantic\nSegmentation\nAdditionally, we also present semantic segmentation results for both of the metrics (PP-IOU and PI-IOU) in Table 3. In this comparison, all the techniques use the same\nFCN unary potentials. Additionally, all inference methods\nexcept Dense-CRF use our af\ufb01nity measure (since the previous analysis suggested that our af\ufb01nities yield better performance). We use BNF-SB to denote the variant of our\nmethod that uses only semantic boundary based af\ufb01nities.\nAdditionally, we use BNF-SB-SM to indicate the version\nof our method that uses both boundary and softmax -based\naf\ufb01nities (see Eq. (3)).\nBased on these results, we observe that our proposed\ntechnique outperforms all the other globalization methods\naccording to both metrics, by 0:3%and1:3%respectively.\nMetric Method aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8de5-6db4-11ee-bb08-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-30", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "Metric Method aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean\nPP-IOUFCN-Softmax 80.7 71.6 80.7 71.3 72.9 88.1 81.8 86.6 47.4 82.9 57.9 83.9 79.6 80.4 81.0 64.7 78.2 54.5 80.9 69.9 74.8\nBelief Propagation [31] 81.4 72.2 82.4 72.2 74.3 88.8 82.4 87.2 48.4 83.8 58.4 84.6 80.5 80.9 81.5 65.1 79.5 55.5 81.5 71.2 75.6\nICM 81.7 72.2 82.8 72.1 75.3 89.6 83.4 87.7 46.3 83.3 58.4 84.6 80.6 81.4 81.5 65.8 79.5 56.0 80.7 74.1 75.8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8de6-6db4-11ee-b0be-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-31", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "TRWS [32] 81.6 70.9 83.8 72.0 75.1 89.5 82.5 88.0 51.7 86.6 61.9 85.8 83.3 80.8 81.1 65.3 81.5 58.8 77.6 75.9 76.7\nGraph Cuts [5] 82.5 72.4 84.6 73.3 77.2 89.7 83.3 88.8 49.3 84.0 60.3 85.4 82.2 81.2 81.9 66.7 79.8 58.0 82.3 74.9 76.9\nQPBO [26] 82.6 72.3 84.7 73.1 76.7 89.9 83.6 89.3 49.7 85.0 61.1 86.2 82.9 81.3 82.3 67.1 80.5 58.8 82.2 75.1 77.2\nDense-CRF [19] 83.4 71.5 84.9 72.6 76.2 89.5 83.3 89.1 50.4 86.7 61.0 86.8 83.5 81.8 82.3 66.9 82.2 58.2 81.9 75.1 77.3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8de7-6db4-11ee-bcd0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-32", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "BNF-SB 81.9 72.5 84.9 73.3 76.0 90.3 83.1 89.2 51.2 86.7 61.5 86.6 83.2 81.3 81.9 66.2 81.7 58.6 81.6 75.8 77.4\nBNF-SB-SM 82.2 73.1 85.1 73.8 76.7 90.6 83.4 89.5 51.3 86.7 61.4 86.8 83.3 81.7 82.3 67.7 81.9 58.4 82.4 75.4 77.6\nPI-IOUFCN-Softmax 56.9 35.1 47.8 41.1 27.4 51.1 43.4 52.7 22.2 43.1 29.2 54.2 40.5 45.6 59.1 24.2 43.6 24.8 55.9 37.2 41.8\nBelief Propagation [31] 68.0 38.6 52.9 45.8 31.9 55.9 47.2 58.2 24.6 49.9 31.7 60.2 44.9 50.1 62.4 25.2 49.9 27.6 62.3 42.2 46.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8de8-6db4-11ee-a73e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-33", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "ICM 65.3 40.9 56.4 45.3 33.7 58.9 49.5 61.9 25.8 53.5 33.2 62.1 48.0 53.2 63.4 24.1 54.8 34.0 63.7 47.7 48.8\nTRWS [32] 67.5 40.7 60.3 46.3 35.6 63.4 49.6 69.3 29.7 58.9 37.8 67.4 57.3 53.8 64.1 26.3 62.0 36.9 63.1 49.9 52.0\nGraph Cuts [5] 72.1 47.8 64.5 50.8 36.0 70.8 51.4 71.6 31.7 65.8 34.4 71.8 62.0 59.4 64.8 29.0 60.9 38.7 70.3 51.6 55.3\nQPBO [26] 71.6 46.8 65.6 49.6 38.0 72.6 52.7 76.7 32.5 69.6 38.9 74.4 61.4 61.0 66.2 30.3 68.7 41.4 72.2 52.8 57.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8de9-6db4-11ee-970c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-34", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "Dense-CRF [19] 68.0 39.5 58.0 45.0 33.4 62.8 47.7 66.0 29.4 60.9 36.0 68.5 54.6 51.4 63.7 28.3 57.6 37.1 65.9 48.2 51.1\nBNF-SB 71.6 48.1 67.2 52.3 37.8 79.5 52.9 80.8 33.3 71.5 39.5 75.1 65.7 63.4 65.1 31.1 67.5 39.6 73.2 54.7 58.5\nBNF-SB-SM 72.0 48.9 66.5 52.9 39.1 79.0 53.4 78.6 32.9 72.2 39.4 74.6 65.9 64.2 65.8 31.7 66.9 39.0 73.1 53.9 58.5\nTable 3: Semantic segmentation results on the SBD dataset according to PP-IOU (per pixel) and PI-IOU (per image) evaluation metrics. We use BNF-SB to denote the variant of our method that uses only semantic boundary based af\ufb01nities."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dea-6db4-11ee-80d7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-35", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "Additionally, we use BNF-SB-SM to indicate our method that uses boundary and softmax based af\ufb01nities (See Eq. (3)). We\nobserve that our proposed globalization method outperforms other globalization techniques according to both metrics by\nat least 0:3%and1:3%respectively. Note that in this experiment, all of the inference methods use the same FCN unary\npotentials . Additionally, for each method except Dense-CRF (it is challenging to incorporate boundary based af\ufb01nities into\nthe Dense-CRF framework) we use our boundary based af\ufb01nities, since those lead to better results.\nAdditionally, these results indicate that most bene\ufb01t comes\nfrom the semantic boundary af\ufb01nity term rather than the\nsoftmax af\ufb01nity term.\nIn Fig. 5, we also present qualitative semantic segmentation results. Note that, compared to the segmentation output from the softmax layer, our segmentation is much better localized around the object boundaries. Additionally,\nin comparison to Dense-CRF predictions, our method produces segmentations that are much spatially smoother.\n4.3. Semantic Boundary Classi\ufb01cation\nWe can also label our boundaries with a speci\ufb01c object"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8deb-6db4-11ee-83b8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-36", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "4.3. Semantic Boundary Classi\ufb01cation\nWe can also label our boundaries with a speci\ufb01c object\nclass, using the same classi\ufb01cation strategy as in the H FL\nsystem [4]. Since the SBD dataset provides annotations for\nsemantic boundary classi\ufb01cation, we can test our results\nagainst the state-of-the-art H FL [4] method for this task.\nDue to the space limitation, we do not include full results for\neach category. However, we observe that our produced results achieve mean Max F-Score of 54:5%(averaged across\nall20classes) whereas H FL method obtains 51:7%.\n5. Conclusions\nIn this work we introduced a Boundary Neural Field\n(BNF), an architecture that employs a semantic segmentation FCN to predict semantic boundaries and then uses the\npredicted boundaries and the FCN output to produce an improved semantic segmentation maps a global optimization.\nWe showed that our predicted boundaries are better suited\nfor semantic segmentation than the commonly used low-level color based af\ufb01nities. Additionally, we introduced a\nglobal energy function that decomposes semantic segmentation into multiple binary problems and relaxes an integrality constraint. We demonstrated that the minimization\nof this global energy allows us to predict segmentations\nthat are better localized around the object boundaries and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dec-6db4-11ee-9341-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-37", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "of this global energy allows us to predict segmentations\nthat are better localized around the object boundaries and\nthat are spatially smoother compared to the segmentations\nachieved by prior methods. We made the code of our globalization technique available at http://www.seas.upenn.\nedu/~gberta/publications.html .\nThe main goal of this work was to show the effectiveness of boundary-based af\ufb01nities for semantic segmentation. However, due to differentiability of our global energy,\nit may be possible to add more parameters inside the BNFs\nand learn them in a front-to-end fashion. We believe that\noptimizing the entire architecture jointly could capture the\ninherent relationship between semantic segmentation and\nboundary detection even better and further improve the performance of BNFs. We will investigate this possibility in\nour future work.\n6. Acknowledgements\nThis research was funded in part by NSF award CNS1205521.\nReferences\n[1] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE\nTrans. Pattern Anal. Mach. Intell. , 33(5):898\u2013916, May 2011. 2, 4, 7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8ded-6db4-11ee-ae44-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-38", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "Trans. Pattern Anal. Mach. Intell. , 33(5):898\u2013916, May 2011. 2, 4, 7\n[2] Pablo Arbelaez, J. Pont-Tuset, Jon Barron, F. Marqu\u00e9s, and Jitendra\nMalik. Multiscale combinatorial grouping. In Computer Vision and\nPattern Recognition (CVPR) , 2014. 2, 4\n[3] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Deepedge: A\nmulti-scale bifurcated deep network for top-down contour detection.\nInThe IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015. 2, 3, 4\n[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. High-for-low\nand low-for-high: Ef\ufb01cient boundary detection from deep object features and its applications to high-level vision. In The IEEE International Conference on Computer Vision (ICCV) , December 2015. 2,\n3, 4, 7, 8\n[5] Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach.\nIntell. , 23(11):1222\u20131239, November 2001. 5, 7, 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dee-6db4-11ee-9173-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-39", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "Intell. , 23(11):1222\u20131239, November 2001. 5, 7, 8\n[6] Jo\u00e3o Carreira, Rui Caseiro, Jorge Batista, and Cristian Sminchisescu.\nSemantic segmentation with second-order pooling. In Proceedings\nof the 12th European Conference on Computer Vision - Volume Part\nVII, ECCV\u201912, pages 430\u2013443, Berlin, Heidelberg, 2012. SpringerVerlag. 2\n[7] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin\nMurphy, and Alan L. Yuille. Semantic image segmentation with deep\nconvolutional nets and fully. In ICLR , 2015. 1, 2, 3\n[8] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In The IEEE International Conference on Computer Vision\n(ICCV) , December 2015. 3\n[9] Piotr Doll\u00e1r and C. Lawrence Zitnick. Fast edge detection using\nstructured forests. PAMI , 2015. 2, 4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8def-6db4-11ee-ac33-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-40", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "[9] Piotr Doll\u00e1r and C. Lawrence Zitnick. Fast edge detection using\nstructured forests. PAMI , 2015. 2, 4\n[10] Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun. Learning hierarchical features for scene labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence , August 2013.\n2\n[11] Yaroslav Ganin and Victor S. Lempitsky. N4-\ufb01elds: Neural network\nnearest neighbor \ufb01elds for image transforms. ACCV , 2014. 2, 4\n[12] Saurabh Gupta, Ross Girshick, Pablo Arbel\u00e1ez, and Jitendra Malik.\nLearning rich features from RGB-D images for object detection and\nsegmentation. In Proceedings of the European Conference on Computer Vision (ECCV) , 2014. 2\n[13] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu\nMaji, and Jitendra Malik. Semantic contours from inverse detectors.\nInInternational Conference on Computer Vision (ICCV) , 2011. 4, 7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8df0-6db4-11ee-a936-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-41", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "Maji, and Jitendra Malik. Semantic contours from inverse detectors.\nInInternational Conference on Computer Vision (ICCV) , 2011. 4, 7\n[14] Bharath Hariharan, Pablo Arbel\u00e1ez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In European Conference on Computer Vision (ECCV) , 2014. 2\n[15] Bharath Hariharan, Pablo Andr\u00e9s Arbel\u00e1ez, Ross B. Girshick, and\nJitendra Malik. Hypercolumns for object segmentation and \ufb01negrained localization. In The IEEE Conference on Computer Vision\nand Pattern Recognition (CVPR) , June 2015. 2, 3\n[16] Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled\ndeep neural network for semi-supervised semantic segmentation. In\nNIPS) , December 2015. 3\n[17] Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H. Adelson.\nCrisp boundary detection using pointwise mutual information. In\nECCV , 2014. 2\n[18] Jyri J Kivinen, Christopher KI Williams, and Nicolas Heess. Visual\nboundary prediction: A deep neural prediction network and quality"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8df1-6db4-11ee-aa38-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-42", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "[18] Jyri J Kivinen, Christopher KI Williams, and Nicolas Heess. Visual\nboundary prediction: A deep neural prediction network and quality\ndissection. AISTATS , 1(2):9, 2014. 2\n[19] Philipp Kr\u00e4henb\u00fchl and Vladlen Koltun. Ef\ufb01cient inference in fully\nconnected crfs with gaussian edge potentials. In J. Shawe-Taylor,\nR.S. Zemel, P.L. Bartlett, F. Pereira, and K.Q. Weinberger, editors,\nAdvances in Neural Information Processing Systems 24 , pages 109\u2013\n117. Curran Associates, Inc., 2011. 1, 5, 7, 8[20] Joseph Lim, C. Lawrence Zitnick, and Piotr Doll\u00e1r. Sketch tokens:\nA learned mid-level representation for contour and object detection.\nInCVPR , 2013. 2\n[21] Guosheng Lin, Chunhua Shen, Ian D. Reid, and Anton van den Hengel. Ef\ufb01cient piecewise training of deep structured models for semantic segmentation. CoRR , abs/1504.01013, 2015. 3\n[22] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In The IEEE Conference"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8df2-6db4-11ee-8af9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-43", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "[22] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In The IEEE Conference\non Computer Vision and Pattern Recognition (CVPR) , June 2015. 1,\n3\n[23] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human\nsegmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th\nInt\u2019l Conf. Computer Vision , volume 2, pages 416\u2013423, July 2001. 4\n[24] Mohammadreza Mostajabi, Payman Yadollahpour, and Gregory\nShakhnarovich. Feedforward semantic segmentation with zoom-out\nfeatures. CoRR , abs/1412.0774, 2014. 2\n[25] X. Ren and L. Bo. Discriminatively Trained Sparse Code Gradients\nfor Contour Detection. In Advances in Neural Information Processing Systems , December 2012. 4\n[26] Carsten Rother, Vladimir Kolmogorov, Victor Lempitsky, and Martin\nSzummer. Optimizing binary mrfs via extended roof duality. In Proc\nComp. Vision Pattern Recogn. (CVPR) , June 2007. 7, 8\n[27] Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8df3-6db4-11ee-ae44-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-44", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "Comp. Vision Pattern Recogn. (CVPR) , June 2007. 7, 8\n[27] Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang\nZhang. Deepcontour: A deep convolutional feature learned by\npositive-sharing loss for contour detection. June 2015. 2, 4\n[28] Jonathan R Shewchuk. An introduction to the conjugate gradient\nmethod without the agonizing pain. Technical report, Pittsburgh, PA,\nUSA, 1994. 6\n[29] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 22:888\u2013905, 1997. 2\n[30] K. Simonyan and A. Zisserman. Very deep convolutional networks\nfor large-scale image recognition. CoRR , abs/1409.1556, 2014. 3\n[31] Marshall F. Tappen and William T. Freeman. Comparison of graph\ncuts with belief propagation for stereo, using identical mrf parameters. In Proceedings of the Ninth IEEE International Conference on\nComputer Vision - Volume 2 , ICCV \u201903, pages 900\u2013, Washington,\nDC, USA, 2003. IEEE Computer Society. 5, 7, 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8df4-6db4-11ee-b2d9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-45", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "Computer Vision - Volume 2 , ICCV \u201903, pages 900\u2013, Washington,\nDC, USA, 2003. IEEE Computer Society. 5, 7, 8\n[32] Martin Wainwright, Tommi Jaakkola, and Alan Willsky. Map estimation via agreement on (hyper)trees: Message-passing and linear\nprogramming approaches. IEEE Transactions on Information Theory, 51:3697\u20133717, 2002. 7, 8\n[33] Saining Xie and Zhuowen Tu. Holistically-nested edge detection.\nInThe IEEE International Conference on Computer Vision (ICCV) ,\nDecember 2015. 2, 4\n[34] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip Torr.\nConditional random \ufb01elds as recurrent neural networks. In International Conference on Computer Vision (ICCV) , 2015. 3\n[35] Dengyong Zhou, Olivier Bousquet, Thomas N. Lal, Jason Weston,\nand Bernhard Sch\u00f6lkopf. Learning with local and global consistency.\nIn S. Thrun, L.K. Saul, and B. Sch\u00f6lkopf, editors, Advances in Neural Information Processing Systems 16 , pages 321\u2013328. MIT Press,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8df5-6db4-11ee-b800-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.02674-46", "title": "Semantic Segmentation with Boundary Neural Fields", "source": "http://arxiv.org/pdf/1511.02674", "start_index": 0, "chroma:document": "2004. 5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8df6-6db4-11ee-a147-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-0", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "Unitary Evolution Recurrent Neural Networks\nMartin Arjovsky\u0003MARJOVSKY @DC.UBA.AR\nAmar Shah\u0003AS793@ CAM .AC.UK\nYoshua Bengio\nUniversidad de Buenos Aires, University of Cambridge,\nUniversit \u00b4e de Montr \u00b4eal. Yoshua Bengio is a CIFAR Senior Fellow.\n\u0003Indicates \ufb01rst authors. Ordering determined by coin \ufb02ip.\nAbstract\nRecurrent neural networks (RNNs) are notoriously dif\ufb01cult to train. When the eigenvalues\nof the hidden to hidden weight matrix deviate\nfrom absolute value 1, optimization becomes dif\ufb01cult due to the well studied issue of vanishingandexploding gradients, especially when trying to learn long-term dependencies. To circumvent this problem, we propose a new architecture\nthat learns a unitary weight matrix, with eigenvalues of absolute value exactly 1. The challenge we address is that of parametrizing unitary matrices in a way that does not require expensive computations (such as eigendecomposition) after each weight update. We construct an\nexpressive unitary weight matrix by composing\nseveral structured matrices that act as building\nblocks with parameters to be learned. Optimization with this parameterization becomes feasible"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8df7-6db4-11ee-a9ad-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-1", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "expressive unitary weight matrix by composing\nseveral structured matrices that act as building\nblocks with parameters to be learned. Optimization with this parameterization becomes feasible\nonly when considering hidden states in the complex domain. We demonstrate the potential of\nthis architecture by achieving state of the art results in several hard tasks involving very longterm dependencies.\n1. Introduction\nDeep Neural Networks have shown remarkably good performance on a wide range of complex data problems including speech recognition (Hinton et al., 2012), image\nrecognition (Krizhevsky et al., 2012) and natural language\nprocessing (Collobert et al., 2011). However, training very\ndeep models remains a dif\ufb01cult task. The main issue surrounding the training of deep networks is the vanishing\nandexploding gradients problems introduced by HochreProceedings of the 33rdInternational Conference on Machine\nLearning , New York, NY , USA, 2016. JMLR: W&CP volume\n48. Copyright 2016 by the author(s).iter (1991) and shown by Bengio et al. (1994) to be necessarily arising when trying to learn to reliably store bits\nof information in any parametrized dynamical system. If\ngradients propagated back through a network vanish, the\ncredit assignment role of backpropagation is lost, as information about small changes in states in the far past has no"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8df8-6db4-11ee-9569-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-2", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "gradients propagated back through a network vanish, the\ncredit assignment role of backpropagation is lost, as information about small changes in states in the far past has no\nin\ufb02uence on future states. If gradients explode, gradientbased optimization algorithms struggle to traverse down a\ncost surface, because gradient-based optimization assumes\nsmall changes in parameters yield small changes in the objective function. As the number of time steps considered\nin the sequence of states grows, the shrinking or expanding\neffects associated with the state-to-state transformation at\nindividual time steps can grow exponentially, yielding respectively vanishing or exploding gradients. See Pascanu\net al. (2010) for a review.\nAlthough the long-term dependencies problem appears\nintractable in the absolute (Bengio et al., 1994) for\nparametrized dynamical systems, several heuristics have\nrecently been found to help reduce its effect, such as the\nuse of self-loops and gating units in the LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014) recurrent architectures. Recent work also supports the idea\nof using orthogonal weight matrices to assist optimization\n(Saxe et al., 2014; Le et al., 2015).\nIn this paper, we explore the use of orthogonal and unitary"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8df9-6db4-11ee-9c02-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-3", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "(Saxe et al., 2014; Le et al., 2015).\nIn this paper, we explore the use of orthogonal and unitary\nmatrices in recurrent neural networks. We start in Section 2\nby showing a novel bound on the propagated gradients in\nrecurrent nets when the recurrent matrix is orthogonal. Section 3 discusses the dif\ufb01culties of parameterizing real valued orthogonal matrices and how they can be alleviated by\nmoving to the complex domain.\nWe discuss a novel approach to constructing expressive\nunitary matrices as the composition of simple unitary matrices which require at most O(nlogn)computation and\nO(n)memory, when the state vector has dimension n.\nThese are unlike general matrices, which require O(n2)\ncomputation and memory. Complex valued representationsarXiv:1511.06464v4  [cs.LG]  25 May 2016\nUnitary Evolution Recurrent Neural Networks\nhave been considered for neural networks in the past, but\nwith limited success and adoption (Hirose, 2003; Zimmermann et al., 2011). We hope our \ufb01ndings will change this.\nWhilst our model uses complex valued matrices and parameters, all implementation and optimization is possible\nwith real numbers and has been done in Theano (Bergstra"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dfa-6db4-11ee-8e83-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-4", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "Whilst our model uses complex valued matrices and parameters, all implementation and optimization is possible\nwith real numbers and has been done in Theano (Bergstra\net al., 2010). This along with other implementation details\nare discussed in Section 4, and the code used for the experiments is available online. The potential of the developed\nmodel for learning long term dependencies with relatively\nfew parameters is explored in Section 5. We \ufb01nd that the\nproposed architecture generally outperforms LSTMs and\nprevious approaches based on orthogonal initialization.\n2. Orthogonal Weights and Bounding the\nLong-Term Gradient\nA matrix, W, is orthogonal if W>W=WW>=I.\nOrthogonal matrices have the property that they preserve\nnorm (i.e.kWhk2=khk2) and hence repeated iterative\nmultiplication of a vector by an orthogonal matrix leaves\nthe norm of the vector unchanged.\nLethTandhtbe the hidden unit vectors for hidden layers\nTandtof a neural network with Thidden layers and T\u001d\nt. IfCis the objective we are trying to minimize, then\nthevanishing andexploding gradient problems refer to the\ndecay or growth of@C\n@htas the number of layers, T, grows.\nLet\u001bbe a pointwise nonlinearity function, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dfb-6db4-11ee-ab53-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-5", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "decay or growth of@C\n@htas the number of layers, T, grows.\nLet\u001bbe a pointwise nonlinearity function, and\nzt+1=Wtht+Vtxt+1\nht+1=\u001b(zt+1) (1)\nthen by the chain rule\n@C\n@ht=@C\n@hT@hT\n@ht\n=@C\n@hTT\u00001Y\nk=t@hk+1\n@hk=@C\n@hTT\u00001Y\nk=tDk+1WT\nk (2)\nwhere Dk+1=diag(\u001b0(zk+1))is the Jacobian matrix of\nthe pointwise nonlinearity.\nIn the following we de\ufb01ne the norm of a matrix to refer to\nthe spectral radius norm (or operator 2-norm) and the norm\nof a vector to mean L2-norm. By de\ufb01nition of the operator norms, for any matrices A;Band vectorvwe have\nkAvk\u0014k AkkvkandkABk\u0014k AkkBk. If the weight\nmatrices Wkare norm preserving (i.e. orthogonal), thenweprove\n\r\r\r\r@C"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dfc-6db4-11ee-99a2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-6", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "matrices Wkare norm preserving (i.e. orthogonal), thenweprove\n\r\r\r\r@C\n@ht\r\r\r\r=\r\r\r\r\r@C\n@hTT\u00001Y\nk=tDk+1WT\nk\r\r\r\r\r\n\u0014\r\r\r\r@C\n@hT\r\r\r\rT\u00001Y\nk=t\r\rDk+1WT\nk\r\r\n=\r\r\r\r@C\n@hT\r\r\r\rT\u00001Y\nk=tkDk+1k: (3)\nSince Dkis diagonal,kDkk= max j=1;:::;nj\u001b0(z(j)\nk)j,\nwithz(j)\nkthej-th pre-activation of the k-th hidden layer.\nIf the absolute value of the derivative \u001b0can take some\nvalue\u001c > 1, then this bound is useless, since k@C\n@htk\u0014\r\r\r@C\n@hT\r\r\r\u001cT\u0000twhich grows exponentially in T. We therefore cannot effectively bound@C\n@htfor deep networks, resulting potentially in exploding gradients.\nIn the casej\u001b0j< \u001c < 1, equation 3 proves that that@C"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dfd-6db4-11ee-bdab-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-7", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "@htfor deep networks, resulting potentially in exploding gradients.\nIn the casej\u001b0j< \u001c < 1, equation 3 proves that that@C\n@httends to 0 exponentially fast as Tgrows, resulting in guaranteed vanishing gradients. This argument makes the recti\ufb01ed linear unit (ReLU) nonlinearity an attractive choice\n(Glorot et al., 2011; Nair & Hinton, 2010). Unless all the\nactivations are killed at one layer, the maximum entry of\nDkis 1, resulting inkDkk= 1 for all layers k. With\nReLU nonlinearities, we thus have\n\r\r\r\r@C\n@ht\r\r\r\r\u0014\r\r\r\r@C\n@hT\r\r\r\rT\u00001Y\nk=tkDk+1k=\r\r\r\r@C\n@hT\r\r\r\r: (4)\nMost notably, this result holds for a network of arbitrary\ndepth and renders engineering tricks like gradient clipping\nunnecessary (Pascanu et al., 2010).\nTo the best of our knowledge, this analysis is a novel contribution and the \ufb01rst time a neural network architecture has\nbeen mathematically proven to avoid exploding gradients.\n3. Unitary Evolution RNNs\nUnitary matrices generalize orthogonal matrices to the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dfe-6db4-11ee-95c0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-8", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "been mathematically proven to avoid exploding gradients.\n3. Unitary Evolution RNNs\nUnitary matrices generalize orthogonal matrices to the\ncomplex domain. A complex valued, norm preserving matrix,U, is called a unitary matrix and is such that U\u0003U=\nUU\u0003=I, where U\u0003is the conjugate transpose of U. Directly parametrizing the set of unitary matrices in such a\nway that gradient-based optimization can be applied is not\nstraightforward because a gradient step will typically yield\na matrix that is not unitary, and projecting on the set of unitary matrices (e.g., by performing an eigendecomposition)\ngenerally costsO(n3)computation when Uisn\u0002n.\nThe most important feature of unitary and orthogonal matrices for our purpose is that they have eigenvalues \u0015jwith\nUnitary Evolution Recurrent Neural Networks\nabsolute value 1. The following lemma, proved in (Hoffman & Kunze, 1971), may shed light on a method which\ncan be used to ef\ufb01ciently span a large set of unitary matrices.\nLemma 1. A complex square matrix Wis unitary if and\nonly if it has an eigendecomposition of the form W="}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8dff-6db4-11ee-ae6d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-9", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "Lemma 1. A complex square matrix Wis unitary if and\nonly if it has an eigendecomposition of the form W=\nVDV\u0003, where\u0003denotes the conjugate transpose. Here,\nV;D2Cn\u0002nare complex matrices, where Vis unitary,\nandDis a diagonal such that jDj;jj= 1. Furthermore, W\nis a real orthogonal matrix if and only if for every eigenvalue Dj;j=\u0015jwith eigenvector vj, there is also a complex conjugate eigenvalue \u0015k=\u0015jwith corresponding\neigenvectorvk=vj.\nWriting\u0015j=eiwjwithwj2R, a naive method to learn a\nunitary matrix would be to \ufb01x a basis of eigenvectors V2\nCn\u0002nand set\nW=VDV\u0003; (5)\nwhere Dis a diagonal such that Dj;j=\u0015j.\nLemma 1 informs us how to construct a real orthogonal\nmatrix, W. We must (i) ensure the columns of Vcome\nin complex conjugate pairs, vk=vj, and (ii) tie weights"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e00-6db4-11ee-9f44-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-10", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "matrix, W. We must (i) ensure the columns of Vcome\nin complex conjugate pairs, vk=vj, and (ii) tie weights\nwk=\u0000wjin order to achieve eiwj=eiwk. Most neural network objective functions are differentiable with respect to the weight matrices, and consequently wjmay be\nlearned by gradient descent.\nUnfortunately the above approach has undesirable properties. Fixing Vand learning wrequiresO\u0000\nn2\u0001\nmemory,\nwhich is unacceptable given that the number of learned parameters isO(n). Further note that calculating Vufor an\narbitrary vector urequiresO(n2)computation. Setting V\nto the identity would satisfy the conditions of the lemma,\nwhilst reducing memory and computation requirements to\nO(n), however, Wwould remain diagonal, and have poor\nrepresentation capacity.\nWe propose an alternative strategy to parameterize unitary\nmatrices. Since the product of unitary matrices is itself a\nunitary matrix, we compose several simple, parameteric,\nunitary matrices to construct a single, expressive unitary\nmatrix. The four unitary building blocks considered are\n\u000fD, a diagonal matrix with Dj;j=eiwj, with parameterswj2R,\n\u000fR=I\u00002vv\u0003"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e01-6db4-11ee-98a8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-11", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "\u000fD, a diagonal matrix with Dj;j=eiwj, with parameterswj2R,\n\u000fR=I\u00002vv\u0003\nkvk2, a re\ufb02ection matrix in the complex\nvectorv2Cn,\n\u000f\u0005, a \ufb01xed random index permutation matrix, and\n\u000f F andF\u00001, the Fourier and inverse Fourier transforms.Appealingly, D,Rand\u0005all permitO(n)storage and\nO(n)computation for matrix vector products. FandF\u00001\nrequire no storage and O(nlogn)matrix vector multiplication using the Fast Fourier Transform algorithm. A major\nadvantage of composing unitary matrices of the form listed\nabove, is that the number of parameters, memory and computational cost increase almost linearly in the size of the\nhidden layer. With such a weight matrix, immensely large\nhidden layers are feasible to train, whilst being impossible\nin traditional neural networks.\nWith this in mind, in this work we choose to consider recurrent neural networks with unitary hidden to hidden weight\nmatrices. Our claim is that the ability to have large hidden\nlayers where hidden states norms are preserved provides\na powerful tool for modeling long term dependencies in\nsequence data. (Bengio et al., 1994) suggest that having\na large memory may be crucial for solving dif\ufb01cult tasks"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e02-6db4-11ee-a6ee-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-12", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "sequence data. (Bengio et al., 1994) suggest that having\na large memory may be crucial for solving dif\ufb01cult tasks\nwith long ranging dependencies: the smaller the state dimension, the more information necessarily has to be eliminated when mapping a long sequence to a \ufb01xed-dimension\nstate.\nWe call any RNN architecture which uses a unitary hidden\nto hidden matrix a unitary evolution RNN (uRNN). After\nexperimenting with several structures, we settled on the following composition\nW=D3R2F\u00001D2\u0005R 1FD1: (6)\nWhilst each but the permutation matrix is complex, we\nparameterize and represent them with real numbers for implementation purposes. When the \ufb01nal cost is real and differentiable, we may perform gradient descent optimization\nto learn the parameters. (Yang et al., 2015) construct a real\nvalued, non-orthogonal matrix using a similar parameterization with the motivation of parameter reduction by an\norder of magnitude on an industrial sized network. This\ncombined with earlier work (Le et al., 2010) suggests that it\nis possible to create highly expressive matrices by composing simple matrices with few parameters. In the following\nsection, we explain details on how to implement our model\nand illustrate how we bypass the potential dif\ufb01culties of\nworking in the complex domain."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e03-6db4-11ee-8a84-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-13", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "section, we explain details on how to implement our model\nand illustrate how we bypass the potential dif\ufb01culties of\nworking in the complex domain.\n4. Architecture details\nIn this section, we describe the nonlinearity we used, how\nwe incorporate real valued inputs with complex valued hidden units and map from complex hidden states to real outputs.\n4.1. Complex hidden units\nOur implementation represents all complex numbers using real values in terms of their real and imaginary parts.\nUnitary Evolution Recurrent Neural Networks\nUnder this framework, we sidestep the lack of support\nfor complex numbers by most deep learning frameworks.\nConsider multiplying the complex weight matrix W=\nA+iBby the complex hidden vector h=x+iy,\nwhere A;B;x;y are real. It is trivially true that Wh=\n(Ax\u0000By) +i(Ay+Bx). When we represent v2Cnas\u0000\nRe(v)>;Im(v)>\u0001>2R2n, we compute complex matrix\nvector products with real numbers as follows\n\u0012Re(Wh)\nIm(Wh)\u0013\n=\u0012A\u0000B\nB A\u0013\u0012Re(h)\nIm(h)\u0013\n: (7)\nMore generally, let f:Cn!Cnbe any complex function andz=x+iyany complex vector. We may write"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e04-6db4-11ee-9ef9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-14", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "Im(h)\u0013\n: (7)\nMore generally, let f:Cn!Cnbe any complex function andz=x+iyany complex vector. We may write\nf(z) =\u000b(x;y) +i\f(x;y)where\u000b;\f :Rn!Rn. This\nallows us to implement everything using real valued operations, compatible with any any deep learning framework\nwith automatic differentiation such as Theano.\n4.2. Input to Hidden, Nonlinearity, Hidden to Output\nAs is the case with most recurrent networks, our uRNN follows the same hidden to hidden mapping as equation 1 with\nVt=VandWt=W. Denote the size of the complex\nvalued hidden states as nh. The input to hidden matrix is\ncomplex valued, V2Cnh\u0002nin. We learn the initial hidden\nstateh02Cnhas a parameter of the model.\nChoosing an appropriate nonlinearity is not trivial in the\ncomplex domain. As discussed in the introduction, using a\nReLU is a natural choice in combination with a norm preserving weight matrix. We \ufb01rst experimented with placing\nseparate ReLU activations on the real and imaginary parts"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e05-6db4-11ee-85e2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-15", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "ReLU is a natural choice in combination with a norm preserving weight matrix. We \ufb01rst experimented with placing\nseparate ReLU activations on the real and imaginary parts\nof the hidden states. However, we found that such a nonlinearity usually performed poorly. Our intuition is that applying separate ReLU nonlinearities to the real and imaginary parts brutally impacts the phase of a complex number,\nmaking it dif\ufb01cult to learn structure.\nWe speculate that maintaining the phase of hidden states\nmay be important for storing information across a large\nnumber of time steps, and our experiments supported this\nclaim. A variation of the ReLU that we name modReLU ,\nis what we \ufb01nally chose. It is a pointwise nonlinearity,\n\u001bmodReLU (z) :C!C, which affects only the absolute\nvalue of a complex number, de\ufb01ned as\n\u001bmodReLU (z) =\u001a(jzj+b)z\njzjifjzj+b\u00150\n0 ifjzj+b<0(8)\nwhereb2Ris a bias parameter of the nonlinearity. For a\nnhdimensional hidden space we learn nhnonlinearity bias\nparameters, one per dimension. Note that the modReLU\nis similar to the ReLU in spirit, in fact more concretely"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e06-6db4-11ee-bc9b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-16", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "parameters, one per dimension. Note that the modReLU\nis similar to the ReLU in spirit, in fact more concretely\n\u001bmodReLU (z) =\u001bReLU (jzj+b)z\njzj.To map hidden states to output, we de\ufb01ne a matrix U2\nRno\u00022nh, wherenois the output dimension. We calculate\na linear output as\not=U\u0012Re(ht)\nIm(ht)\u0013\n+bo; (9)\nwherebo2Rnois the output bias. The linear output is real\nvalued (ot2Rno) and can be used for prediction and loss\nfunction calculation akin to typical neural networks (e.g. it\nmay be passed through a softmax which is used for cross\nentropy calculation for classi\ufb01cation tasks).\n4.3. Initialization\nDue to the stability of the norm preserving operations of\nour network, we found that performance was not very sensitive to initialization of parameters. For full disclosure\nand reproducibility, we explain our initialization strategy\nfor each parameter below.\n\u000fWe initialize VandU(the input and output matrices) as in (Glorot & Bengio, 2010),\nwith weights sampled independently from uniforms,\nUh\n\u0000p\n6pnin+nout;p\n6pnin+nouti\n."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e07-6db4-11ee-b459-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-17", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "with weights sampled independently from uniforms,\nUh\n\u0000p\n6pnin+nout;p\n6pnin+nouti\n.\n\u000fThe biases,bandboare initialized to 0. This implies\nthat at initialization, the network is linear with unitary\nweights, which seems to help early optimization (Saxe\net al., 2014).\n\u000fThe re\ufb02ection vectors for R1andR2are initialized\ncoordinate-wise from a uniform U[\u00001;1]. Note that\nthe re\ufb02ection matrices are invariant to scalar multiplication of the parameter vector, hence the width of the\nuniform initialization is unimportant.\n\u000fThe diagonal weights for D1;D2andD3are sampled from a uniform, U[\u0000\u0019;\u0019]. This ensures that the\ndiagonal entries Dj;jare sampled uniformly over the\ncomplex unit circle.\n\u000fWe initialize h0with a uniform,Uh\n\u0000q\n3\n2nh;q\n3\n2nhi\n,\nwhich results in E\u0002\nkh0k2\u0003\n= 1. Since the norm of\nthe hidden units are roughly preserved through unitary\nevolution and inputs are typically whitened to have\nnorm 1, we have hidden states, inputs and linear outputs of the same order of magnitude, which seems to\nhelp optimization.\n5. Experiments"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e08-6db4-11ee-8479-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-18", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "norm 1, we have hidden states, inputs and linear outputs of the same order of magnitude, which seems to\nhelp optimization.\n5. Experiments\nIn this section we explore the performance of our uRNN\nin relation to (a) RNN with tanh activations, (b) IRNN (Le\net al., 2015), that is an RNN with ReLU activations and\nwith the recurrent weight matrix initialized to the identity,\nUnitary Evolution Recurrent Neural Networks\nFigure 1. Results of the copying memory problem for time lags of 100;200;300;500. The LSTM is able to beat the baseline only for\n100times steps. Conversely the uRNN is able to completely solve each time length in very few training iterations, without getting stuck\nat the baseline.\nand (c) LSTM (Hochreiter & Schmidhuber, 1997) models. We show that the uRNN shines quantitatively when\nit comes to modeling long term dependencies and exhibits\nqualitatively different learning properties to the other models.\nWe chose a handful of tasks to evaluate the performance\nof the various models. The tasks were especially created\nto be be pathologically hard, and have been used as benchmarks for testing the ability of a model to capture long-term\nmemory (Hochreiter & Schmidhuber, 1997; Le et al., 2015;"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e09-6db4-11ee-93de-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-19", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "memory (Hochreiter & Schmidhuber, 1997; Le et al., 2015;\nGraves et al., 2014; Martens & Sutskever, 2011)\nOf the handful of optimization algorithms we tried on the\nvarious models, RMSProp (Tieleman & Hinton, 2012) lead\nto fastest convergence and is what we stuck to for all experiments here on in. However, we found the IRNN to be\nparticularly unstable; it only ran without blowing up with\nincredibly low learning rates and gradient clipping. Since\nthe performance was so poor relative to other models we\ncompare against, we do not show IRNN curves in the \ufb01gures. In each experiment we use a learning rate of 10\u00003\nand a decay rate of 0:9. For the LSTM and RNN models,\nwe had to clip gradients at 1 to avoid exploding gradients.Gradient clipping was unnecessary for the uRNN.\n5.1. Copying memory problem\nRecurrent networks have been known to have trouble remembering information about inputs seen many time steps\npreviously (Bengio et al., 1994; Pascanu et al., 2010). We\ntherefore want to test the uRNN\u2019s ability to recall exactly\ndata seen a long time ago.\nFollowing a similar setup to (Hochreiter & Schmidhuber,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e0a-6db4-11ee-a072-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-20", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "data seen a long time ago.\nFollowing a similar setup to (Hochreiter & Schmidhuber,\n1997), we outline the copy memory task. Consider 10 categories,faig9\ni=0. The input takes the form of a T+20 length\nvector of categories, where we test over a range of values\nofT. The \ufb01rst 10entries are sampled uniformly, independently and with replacement from faig7\ni=0, and represent\nthe sequence which will need to be remembered. The next\nT\u00001entries are set to a8, which can be thought of as the\n\u2019blank\u2019 category. The next single entry is a9, which represents a delimiter, which should indicate to the algorithm\nthat it is now required to reproduce the initial 10categories\nin the output. The remaining 10entries are set to a8. The\nrequired output sequence consists of T+ 10 repeated entries ofa8, followed by the \ufb01rst 10categories of the input\nUnitary Evolution Recurrent Neural Networks\nFigure 2. Results of the adding problem for T= 100 ;200;400;750. The RNN with tanh is not able to beat the baseline for any time\nlength. The LSTM and the uRNN show similar performance across time lengths, consistently beating the baseline."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e0b-6db4-11ee-a749-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-21", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "length. The LSTM and the uRNN show similar performance across time lengths, consistently beating the baseline.\nsequence in exactly the same order. The goal is to minimize the average cross entropy of category predictions at\neach time step of the sequence. The task amounts to having to remember a categorical sequence of length 10, for T\ntime steps.\nA simple baseline can be established by considering an\noptimal strategy when no memory is available, which we\ndeem the memoryless strategy. The memoryless strategy\nwould be to predict a8forT+ 10 entries and then predict\neach of the \ufb01nal 10categories from the set faig7\ni=0independently and uniformly at random. The categorical cross\nentropy of this strategy is10 log(8)\nT+20.\nWe ran experiments where the RNN with tanh activations,\nIRNN, LSTM and uRNN had hidden layers of size 80, 80,\n40 and 128 respectively. This equates to roughly 6500 parameters per model. In Figure 1, we see that aside from the\nsimplest case, both the RNN with tanh and more surprisingly the LSTMs get almost exactly the same cost as the\nmemoryless strategy. This behaviour is consistent with the\nresults of (Graves et al., 2014), in which poor performance"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e0c-6db4-11ee-974d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-22", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "memoryless strategy. This behaviour is consistent with the\nresults of (Graves et al., 2014), in which poor performance\nis reported for the LSTM for a very similar long term memory problem.\nThe uRNN consistently achieves perfect performance inrelatively few iterations, even when having to recall sequences after 500 time steps. What is remarkable is that\nthe uRNN does not get stuck at the baseline at all, whilst\nthe LSTM and RNN do. This behaviour suggests that the\nrepresentations learned by the uRNN have qualitatively different properties from both the LSTM and classical RNNs.\n5.2. Adding Problem\nWe closely follow the adding problem de\ufb01ned in (Hochreiter & Schmidhuber, 1997) to explain the task at hand. Each\ninput consists of two sequences of length T. The \ufb01rst sequence, which we denote x, consists of numbers sampled\nuniformly at random U[0;1]. The second sequence is an indicator sequence consisting of exactly two entries of 1 and\nremaining entries 0. The \ufb01rst 1 entry is located uniformly\nat random in the \ufb01rst half of the sequence, whilst the second 1 entry is located uniformly at random in the second\nhalf. The output is the sum of the two entries of the \ufb01rst sequence, corresponding to where the 1 entries are located in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e0d-6db4-11ee-b110-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-23", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "half. The output is the sum of the two entries of the \ufb01rst sequence, corresponding to where the 1 entries are located in\nthe second sequence. A naive strategy of predicting 1 as the\noutput regardless of the input sequence gives an expected\nmean squared error of 0:167, the variance of the sum of two\nindependent uniform distributions. This is our baseline to\nbeat.\nUnitary Evolution Recurrent Neural Networks\nFigure 3. Results on pixel by pixel MNIST classi\ufb01cation tasks. The uRNN is able to converge in a fraction of the iterations that the\nLSTM requires. The LSTM performs better on MNIST classi\ufb01cation, but the uRNN outperforms on the more complicated task of\npermuted pixels.\nWe chose to use 128 hidden units for the RNN with tanh,\nIRNN and LSTM and 512 for the uRNN. This equates to\nroughly 16K parameters for the RNN with tanh and IRNN,\n60K for the LSTM and almost 9K for the uRNN. All models were trained using batch sizes of 20 and 50 with the best\nresults being reported. Our results are shown in Figure 2.\nThe LSTM and uRNN models are able to convincingly beat\nthe baseline up to T= 400 time steps. Both models do well"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e0e-6db4-11ee-9da0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-24", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "The LSTM and uRNN models are able to convincingly beat\nthe baseline up to T= 400 time steps. Both models do well\nwhenT= 750 , but the mean squared error does not reach\nclose to 0. The uRNN achieves lower test error, but it\u2019s\ncurve is more noisy. Despite having vastly more parameters, we monitored the LSTM performance to ensure no\nover\ufb01tting.\nThe RNN with tanh and IRNN were not able to beat the\nbaseline for any number of time steps. (Le et al., 2015) report that their RNN solve the problem for T= 150 and the\nIRNN forT= 300 , but they require over a million iterations before they start learning. Neither of the two models came close to either the uRNN or the LSTM in performance. The stark difference in our \ufb01ndings are best explained by our use of RMSprop with signi\ufb01cantly higher\nlearning rates ( 10\u00003as opposed to 10\u00008) than (Le et al.,\n2015) use for SGD with momentum.\n5.3. Pixel-by-pixel MNIST\nIn this task, suggested by (Le et al., 2015), algorithms are\nfed pixels of MNIST (LeCun et al., 1998) sequentially and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e0f-6db4-11ee-b491-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-25", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "In this task, suggested by (Le et al., 2015), algorithms are\nfed pixels of MNIST (LeCun et al., 1998) sequentially and\nrequired to output a class label at the end. We consider two\ntasks: one where pixels are read in order (from left to right,\nbottom to top) and one where the pixels are all randomly\npermuted using the same randomly generated permutation\nmatrix. The same model architectures as for the addingproblem were used for this task, except we now use a softmax for category classi\ufb01cation. We ran the optimization\nalgorithms until convergence of the mean categorical cross\nentropy on test data, and plot test accuracy in Figure 3.\nBoth the uRNN and LSTM perform applaudably well here.\nOn the correct unpermuted MNIST pixels, the LSTM performs better, achieving 98.2 % test accurracy versus 95.1%\nfor the uRNN. However, when we permute the ordering of\nthe pixels, the uRNN dominates with 91.4% of accuracy in\ncontrast to the 88% of the LSTM, despite having less than\na quarter of the parameters. This result is state of the art on\nthis task, beating the IRNN (Le et al., 2015), which reaches\nclose to 82% after 1 million training iterations. Notice that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e10-6db4-11ee-be2c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-26", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "this task, beating the IRNN (Le et al., 2015), which reaches\nclose to 82% after 1 million training iterations. Notice that\nuRNN reaches convergence in less than 20 thousand iterations, while it takes the LSTM from 5 to 10 times as many\nto \ufb01nish learning.\nPermuting the pixels of MNIST images creates many\nlonger term dependencies across pixels than in the original pixel ordering, where a lot of structure is local. This\nmakes it necessary for a network to learn and remember\nmore complicated dependencies across varying time scales.\nThe results suggest that the uRNN is better able to deal\nwith such structure over the data, where the LSTM is better\nsuited to more local sequence structure tasks.\n5.4. Exploratory experiments\nNorms of hidden state gradients. As discussed in Section 2, key to being able to learn long term dependencies\nis in controlling@C\n@ht. With this in mind, we explored how\neach model propagated gradients, by examining\r\r\r@C\n@ht\r\r\ras\na function of t. Gradient norms were computed at the beUnitary Evolution Recurrent Neural Networks\nFigure 4. From left to right. Norms of the gradients with respect to hidden states i.e.\r\r\r@C"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e11-6db4-11ee-a77c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-27", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "Figure 4. From left to right. Norms of the gradients with respect to hidden states i.e.\r\r\r@C\n@ht\r\r\rat (i) beginning of training, (ii) after 100\niterations. (iii) Norms of the hidden states and (iv) L2distance between hidden states and \ufb01nal hidden state. The gradient norms of\nuRNNs do not decay as fast as for other models as training progresses. uRNN hidden state norms stay much more consistent over time\nthan the LSTM. LSTM hidden states stay almost the same after a number of time steps, suggesting that it is not able to use new input\ninformation.\nginning of training and again after 100 iterations of training\non the adding problem. The curves are plotted in Figure 4.\nIt is clear that at \ufb01rst, the uRNN propagates gradients perfectly, while each other model has exponentially vanishing\ngradients. After 100 iterations of training, each model experiences vanishing gradients, but the uRNN is best able to\npropagate information, having much less decay.\nHidden state saturation. We claim that typical recurrent\narchitectures saturate, in the sense that after they acquire\nsome information, it becomes much more dif\ufb01cult to acquire further information pertaining to longer dependencies. We took the uRNN and LSTM models trained on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e12-6db4-11ee-a988-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-28", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "some information, it becomes much more dif\ufb01cult to acquire further information pertaining to longer dependencies. We took the uRNN and LSTM models trained on\nthe adding problem with T= 200 , and computed a forward pass with newly generated data for the adding problem withT= 1000 . In order to show saturation effects,\nwe plot the norms of the hidden states and the L2distance\nbetween each state and the last in Figure 4.\nIn our experiments, it is clear that the uRNN does not suffer\nas much as other models do. Notice that whilst the norms of\nhidden states in the uRNN grow very steadily over time, in\nthe LSTM they grow very fast, and then stay constant after\nabout 500time steps. This behaviour may suggest that the\nLSTM hidden states saturate in their ability to incorporate\nnew information, which is vital for modeling long complicated sequences. It is interesting to see that the LSTM\nhidden state at t= 500 , is close to that of t= 1000 , whilst\nthis is far from the case in the uRNN. Again, this suggests\nthat the LSTM\u2019s capacity to use new information to alter\nits hidden state severly degrades with sequence length. The\nuRNN does not suffer from this dif\ufb01culty nearly as badly.\nA clear example of this phenomenon was observed in the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e13-6db4-11ee-a585-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-29", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "uRNN does not suffer from this dif\ufb01culty nearly as badly.\nA clear example of this phenomenon was observed in the\nadding problem with T= 750 . We found that the Pearson\ncorrelation between the LSTM output prediction and the\n\ufb01rst of the two uniform samples (whose sum is the target\noutput) was\u001a= 0:991. This suggests that the LSTM learnt\nto simply \ufb01nd and store the \ufb01rst sample, as it was unable\nto incorporate any more information by the time it reachedthe second, due to saturation of the hidden states.\n6. Discussion\nThere are a plethora of further ideas that may be explored\nfrom our \ufb01ndings, both with regards to learning representation and ef\ufb01cient implementation. For example, one hurdle\nof modeling long sequences with recurrent networks is the\nrequirement of storing all hidden state values for the purpose of gradient backpropagation. This can be prohibitive,\nsince GPU memory is typically a limiting factor of neural\nnetwork optimization. However, since our weight matrix is\nunitary, its inverse is its conjugate transpose, which is just\nas easy to operate with. If further we were to use an invertible nonlinearity function, we would no longer need to store"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e14-6db4-11ee-abd7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-30", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "as easy to operate with. If further we were to use an invertible nonlinearity function, we would no longer need to store\nhidden states, since they can be recomputed in the backward pass. This could have potentially huge implications,\nas we would be able to reduce memory usage by an order\nofT, the number of time steps. This would make having\nimmensely large hidden layers possible, perhaps enabling\nvast memory representations.\nIn this paper we demonstrate state of the art performance\non hard problems requiring long term reasoning and memory. These results are based on a novel parameterization\nof unitary matrices which permit ef\ufb01cient matrix computations and parameter optimization. Whilst complex domain modeling has been widely succesful in the signal processing community (e.g. Fourier transforms, wavelets), we\nhave yet to exploit the power of complex valued representation in the deep learning community. Our hope is that\nthis work will be a step forward in this direction. We motivate the idea of unitary evolution as a novel way to mitigate\nthe problems of vanishing and exploding gradients. Empirical evidence suggests that our uRNN is better able to pass\ngradient information through long sequences and does not\nsuffer from saturating hidden states as much as LSTMs,\ntypical RNNs, or RNNs initialized with the identity weight\nmatrix (IRNNs)."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e15-6db4-11ee-b050-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-31", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "suffer from saturating hidden states as much as LSTMs,\ntypical RNNs, or RNNs initialized with the identity weight\nmatrix (IRNNs).\nUnitary Evolution Recurrent Neural Networks\nAcknowledgments : We thank the developers of\nTheano (Bergstra et al., 2010) for their great work.\nWe thank NSERC, Compute Canada, Canada Research\nChairs and CIFAR for their support. We would also like\nto thank C \u00b8 aglar Gulc \u00b8ehre, David Krueger, Soroush Mehri,\nMarcin Moczulski, Mohammad Pezeshki and Saizheng\nZhang for helpful discussions, comments and code sharing.\nReferences\nBengio, Yoshua, Simard, Patrice, and Frasconi, Paolo.\nLearning long-term dependencies with gradient descent\nis dif\ufb01cult. IEE Transactions on Neural Networks , 5,\n1994.\nBergstra, James, Breuleux, Olivier, Bastien, Fr \u00b4ed\u00b4eric,\nLamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua. Theano: a CPU and GPU math expression"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e16-6db4-11ee-bf99-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-32", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "compiler. Proceedings of the Python for Scienti\ufb01c Computing Conference (SciPy) , 2010.\nCho, Kyunghyun, van Merri \u00a8enboer, Bart, Bahdanau,\nDzmitry, and Bengio, Yoshua. On the properties of neural machine translation: Encoder\u2013Decoder approaches.\nInEighth Workshop on Syntax, Semantics and Structure\nin Statistical Translation , October 2014.\nCollobert, Ronan, Weston, Jason, Bottou, L \u00b4eon, Karlen,\nMichael, Kavukcuoglu, Koray, and Kuksa, Pavel. Natural language processing (almost) from scratch. Journal\nof Machine Learning Research , 12:2493\u20132537, 2011.\nGlorot, Xavier and Bengio, Yoshua. Understanding the\ndif\ufb01culty of training deep feedforward neural networks.\nInternational Conference on Arti\ufb01cial Intelligence and\nStatistics (AISTATS) , 2010.\nGlorot, Xavier, Bordes, Antoine, and Bengio, Yoshua.\nDeep sparse recti\ufb01er neural networks. International\nConference on Arti\ufb01cial Intelligence and Statistics (AISTATS) , 2011.\nGraves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e17-6db4-11ee-9821-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-33", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "Conference on Arti\ufb01cial Intelligence and Statistics (AISTATS) , 2011.\nGraves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural\nturing machines. arXiv preprint arXiv:1410.5401 , 2014.\nHinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew,\nVanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara, and\nKingsbury, Brian. Deep neural networks for acoustic\nmodeling in speech recognition. Signal Processing Magazine , 2012.\nHirose, Akira. Complex-valued neural networks: theories\nand applications , volume 5. World Scienti\ufb01c Publishing\nCompany Incorporated, 2003.Hochreiter, S. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, T.U. M \u00a8unich, 1991.\nHochreiter, Sepp and Schmidhuber, J \u00a8urgen. Long shortterm memory. Neural Computation , 8(9):1735\u20131780,\n1997.\nHoffman, Kenneth and Kunze, Ray. Linear Algebra . Pearson, second edition, 1971."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e18-6db4-11ee-bc07-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-34", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "1997.\nHoffman, Kenneth and Kunze, Ray. Linear Algebra . Pearson, second edition, 1971.\nKrizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.\nImagenet classi\ufb01cation with deep convolutional neural networks. Neural Information Processing Systems ,\n2012.\nLe, Quoc, Sarl \u00b4os, Tam \u00b4as, and Smola, Alex. Fastfood - approximating kernel expansions in loglinear time. International Conference on Machine Learning , 2010.\nLe, Quoc V ., Navdeep, Jaitly, and Hinton, Geoffrey E. A\nsimple way to initialize recurrent networks of recti\ufb01ed\nlinear units. arXiv preprint arXiv:1504.00941 , 2015.\nLeCun, Yann, Bottou, L \u00b4eon, Bengio, Yoshua, and Haffner,\nPatrick. Gradient-based learning applied to document\nrecognition. Proceedings of the IEEE , 1998.\nMartens, James and Sutskever, Ilya. Learning recurrent\nneural networks with hessian-free optimization. International Conference on Machine Learning , 2011.\nNair, Vinod and Hinton, Geoffrey E. Recti\ufb01ed linear units\nimprove restricted boltzmann machines. International\nConference on Machine Learning , 2010."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e19-6db4-11ee-a16e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.06464-35", "title": "Unitary Evolution Recurrent Neural Networks", "source": "http://arxiv.org/pdf/1511.06464", "start_index": 0, "chroma:document": "Nair, Vinod and Hinton, Geoffrey E. Recti\ufb01ed linear units\nimprove restricted boltzmann machines. International\nConference on Machine Learning , 2010.\nPascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua.\nOn the dif\ufb01culty of training recurrent neural networks.\nInternational Conference on Machine Learning , 2010.\nSaxe, Andrew M., McLelland, James L., and Ganguli,\nSurya. Exact solutions to the nonlinear dynamics of\nlearning in deep linear neural networks. International\nConference in Learning Representations , 2014.\nTieleman, Tijmen and Hinton, Geoffrey. Lecture 6.5rmsprop: Divide the gradient by a running average of\nits recent magnitude. Coursera: Neural Networks for\nMachine Learning , 2012.\nYang, Zichao, Moczulski, Marcin, Denil, Misha, de Freitas, Nando, Smola, Alex, Song, Le, and Wang, Ziyu.\nDeep fried convnets. International Conference on Computer Vision (ICCV) , 2015.\nZimmermann, Hans-Georg, Minin, Alexey, and\nKusherbaeva, Victoria. Comparison of the complex valued and real valued neural networks trained\nwith gradient descent and random search algorithms. In\nESANN , 2011."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e1a-6db4-11ee-987c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-0", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "Higher Order Conditional Random Fields in Deep\nNeural Networks\nAnurag Arnab, Sadeep Jayasumana, Shuai Zheng, Philip H.S. Torr\nUniversity of Oxford\n{firstname.lastname }@eng.ox.ac.uk\nAbstract. We address the problem of semantic segmentation using deep learning.\nMost segmentation systems include a Conditional Random Field (CRF) to produce\na structured output that is consistent with the image\u2019s visual features. Recent deep\nlearning approaches have incorporated CRFs into Convolutional Neural Networks\n(CNNs), with some even training the CRF end-to-end with the rest of the network.\nHowever, these approaches have not employed higher order potentials, which\nhave previously been shown to signi\ufb01cantly improve segmentation performance.\nIn this paper, we demonstrate that two types of higher order potential, based on\nobject detections and superpixels, can be included in a CRF embedded within a\ndeep network. We design these higher order potentials to allow inference with\nthe differentiable mean \ufb01eld algorithm. As a result, all the parameters of our\nricher CRF model can be learned end-to-end with our pixelwise CNN classi\ufb01er.\nWe achieve state-of-the-art segmentation performance on the PASCAL VOC"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e1b-6db4-11ee-be70-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-1", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "We achieve state-of-the-art segmentation performance on the PASCAL VOC\nbenchmark with these trainable higher order potentials.\nKeywords: Semantic Segmentation, Conditional Random Fields, Deep Learning,\nConvolutional Neural Networks\n1 Introduction\nSemantic segmentation involves assigning a visual object class label to every pixel in an\nimage, resulting in a segmentation with a semantic meaning for each segment. While a\nstrong pixel-level classi\ufb01er is critical for obtaining high accuracy in this task, it is also\nimportant to enforce the consistency of the semantic segmentation output with visual\nfeatures of the image. For example, segmentation boundaries should usually coincide\nwith strong edges in the image, and regions in the image with similar appearance should\nhave the same label.\nRecent advances in deep learning have enabled researchers to create stronger classi\ufb01ers, with automatically learned features, within a Convolutional Neural Network\n(CNN) [1 \u20133]. This has resulted in large improvements in semantic segmentation accuracy on widely used benchmarks such as PASCAL VOC [4]. CNN classi\ufb01ers are now\nconsidered the standard choice for pixel-level classi\ufb01ers used in semantic segmentation.\nOn the other hand, probabilistic graphical models have long been popular for structured prediction of labels, with constraints enforcing label consistency. Conditional"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e1c-6db4-11ee-a339-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-2", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "On the other hand, probabilistic graphical models have long been popular for structured prediction of labels, with constraints enforcing label consistency. Conditional\nRandom Fields (CRFs) have been the most common framework, and various rich and\narXiv:1511.08119v4  [cs.CV]  29 Jul 2016"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e1d-6db4-11ee-8a63-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-3", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "Higher Order Conditional Random Fields in Deep Neural Networks 3\nWe evaluate our higher order potentials on the PASCAL VOC 2012 semantic segmentation benchmark as well as the PASCAL Context dataset, to show signi\ufb01cant\nimprovements over our baseline and achieve state-of-the art results.\n2 Related Work\nBefore deep learning became prominent, semantic segmentation was performed with\ndense hand-crafted features which were fed into a per-pixel or region classi\ufb01er [14]. The\nindividual predictions made by these classi\ufb01ers were often noisy as they lacked global\ncontext, and were thus post-processed with a CRF, making use of prior knowledge such\nas the fact that nearby pixels, as well as pixels of similar appearance, are likely to share\nthe same class label [14, 15].\nThe CRF model of [14] initially contained only unary and pairwise terms in an\n8-neighbourhood, which [16] showed can result in shrinkage bias. Numerous improvements to this model were subsequently proposed including: densely connected pairwise\npotentials facilitating interactions between all pairs of image pixels [17], formulating\nhigher order potentials de\ufb01ned over cliques larger than two nodes [5, 16] in order to\ncapture more context, modelling co-occurrence of object classes [18 \u201320], and utilising"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e1e-6db4-11ee-9619-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-4", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "capture more context, modelling co-occurrence of object classes [18 \u201320], and utilising\nthe results of object detectors [6, 21, 22].\nRecent advances in deep learning have allowed us to replace hand-crafted features\nwith features learned speci\ufb01cally for semantic segmentation. The strength of these\nrepresentations was illustrated by [3] who achieved signi\ufb01cant improvements over\nprevious hand-crafted methods without using any CRF post-processing. Chen et al. [12]\nshowed further improvements by post-processing the results of a CNN with a CRF.\nSubsequent works [9 \u201311, 23] have taken this idea further by incorporating a CRF as\nlayers within a deep network and then learning parameters of both the CRF and CNN\ntogether via backpropagation.\nIn terms of enhancements to conventional CRF models, Ladicky et al. [6] proposed using an off-the-shelf object detector to provide additional cues for semantic\nsegmentation. Unlike other approaches that re\ufb01ne a bounding-box detection to produce\na segmentation [8, 24], this method used detector outputs as a soft constraint and can\nthus recover from object detection errors. Their formulation, however, used graph-cut\ninference, which was only tractable due to the absence of dense pairwise potentials."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e1f-6db4-11ee-a96e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-5", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "thus recover from object detection errors. Their formulation, however, used graph-cut\ninference, which was only tractable due to the absence of dense pairwise potentials.\nObject detectors have also been used by [21, 25], who also modelled variables that\ndescribe the degree to which an object hypothesis is accepted.\nWe formulate the detection potential in a different manner to [6, 21, 25] so that it is\namenable to mean \ufb01eld inference. Mean \ufb01eld permits inference with dense pairwise connections, which results in substantial accuracy improvements [10,12, 17]. Furthermore,\nmean \ufb01eld updates related to our potentials are differentiable and its parameters can thus\nbe learned in our end-to-end trainable architecture.\nWe also note that while the semantic segmentation problem has mostly been formulated in terms of pixels [3,10,14], some have expressed it in terms of superpixels [26 \u201328].\nSuperpixels can capture more context than a single pixel and computational costs can\nalso be reduced if one considers pairwise interactions between superpixels rather than individual pixels [21]. However, such superpixel representations assume that the segments\n4 Arnab et al.\nshare boundaries with objects in an image, which is not always true. As a result, several"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e20-6db4-11ee-aaa1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-6", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "4 Arnab et al.\nshare boundaries with objects in an image, which is not always true. As a result, several\nauthors [5, 7] have employed higher order potentials de\ufb01ned over superpixels that encourage label consistency over regions, but do not strictly enforce it. This approach also\nallows multiple, non-hierarchical layers of superpixels to be integrated. Our formulation\nuses this kind of higher order potential, but in an end-to-end trainable CNN.\nGraphical models have been used with CNNs in other areas besides semantic segmentation, such as in pose-estimation [29] and group activity recognition [30]. Alternatively,\nIonescu et al. [31] incorporated structure into a deep network with structured matrix\nlayers and matrix backpropagation. However, the nature of models used in these works\nis substantially different to ours. Some early works that advocated gradient backpropagation through graphical model inference for parameter optimisation include [32, 33]\nand [34].\nOur work differentiates from the above works since, to our knowledge, we are the \ufb01rst\nto propose and conduct a thorough experimental investigation of higher order potentials\nthat are based on detection outputs and superpixel segmentation in a CRF which is\nlearned end-to-end in a deep network. Note that although [7] formulated mean \ufb01eld"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e21-6db4-11ee-bb57-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-7", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "learned end-to-end in a deep network. Note that although [7] formulated mean \ufb01eld\ninference with higher order potentials, they did not consider object detection potentials\nat all, nor were the parameters learned.\n3 Conditional Random Fields\nWe now review conditional random \ufb01elds used in semantic segmentation and introduce\nthe notation used in the paper. Take an image IwithNpixels, indexed 1,2,...,N . In\nsemantic segmentation, we attempt to assign every pixel a label from a prede\ufb01ned set of\nlabels L={l1,l2,...,l L}. De\ufb01ne a set of random variables X1,X2,...,X N, one for\neach pixel, where each Xi\u2208 L. LetX= [X1X2... X N]T. Any particular assignment\nxtoXis thus a solution to the semantic segmentation problem.\nWe use notations {V}, andV(i)to represent the set of elements of a vector V, and\ntheithelement of V, respectively. Given a graph Gwhere the vertices are from {X}\nand the edges de\ufb01ne connections among these variables, the pair (I,X)is modelled as a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e22-6db4-11ee-b5fb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-8", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "and the edges de\ufb01ne connections among these variables, the pair (I,X)is modelled as a\nCRF characterised by Pr(X=x|I) = (1/Z(I)) exp( \u2212E(x|I)), whereE(x|I)is the\nenergy of the assignment xandZ(I)is the normalisation factor known as the partition\nfunction. We drop the conditioning on Ihereafter to keep the notation uncluttered. The\nenergyE(x)of an assignment is de\ufb01ned using the set of cliques Cin the graph G. More\nspeci\ufb01cally, E(x) =/summationtext\nc\u2208C\u03c8c(xc),where xcis a vector formed by selecting elements\nofxthat correspond to random variables belonging to the clique c, and\u03c8c(.)is the\ncost function for the clique c. The function, \u03c8c(.), usually uses prior knowledge about a\ngood segmentation, as well as information from the image, the observation the CRF is\nconditioned on.\nMinimising the energy yields the maximum a posteriori (MAP) labelling of the\nimage i.e.the most probable label assignment given the observation (image). When\ndense pairwise potentials are used in the CRF to obtain higher accuracy, exact inference"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e23-6db4-11ee-a7b8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-9", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "image i.e.the most probable label assignment given the observation (image). When\ndense pairwise potentials are used in the CRF to obtain higher accuracy, exact inference\nis impracticable, and one has to resort to an approximate inference method such as mean\n\ufb01eld inference [17]. Mean \ufb01eld inference is particularly appealing in a deep learning\nsetting since it is possible to formulate it as a Recurrent Neural Network [10].\nHigher Order Conditional Random Fields in Deep Neural Networks 5\n4 CRF with Higher Order Potentials\nMany CRF models that have been incorporated into deep learning frameworks [10, 12]\nhave so far used only unary and pairwise potentials. However, potentials de\ufb01ned on\nhigher order cliques have been shown to be useful in previous works such as [7, 16]. The\nkey contribution of this paper is to show that a number of explicit higher order potentials\ncan be added to CRFs to improve image segmentation, while staying compatible with\ndeep learning. We formulate these higher order potentials in a manner that mean \ufb01eld\ninference can still be used to solve the CRF. Advantages of mean \ufb01eld inference are\ntwofold: First, it enables ef\ufb01cient inference when using densely-connected pairwise"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e24-6db4-11ee-93d3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-10", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "twofold: First, it enables ef\ufb01cient inference when using densely-connected pairwise\npotentials. Multiple works, [10,33] have shown that dense pairwise connections result in\nsubstantial accuracy improvements, particularly at image boundaries [12, 17]. Secondly,\nwe keep all our mean \ufb01eld updates differentiable with respect to their inputs as well\nas the CRF parameters introduced. This design enables us to use backpropagation to\nautomatically learn all the parameters in the introduced potentials.\nWe use two types of higher order potential, one based on object detections and the\nother based on superpixels. These are detailed in Sections 4.1 and 4.2 respectively. Our\ncomplete CRF model is represented by\nE(x) =/summationdisplay\ni\u03c8U\ni(xi) +/summationdisplay\ni < j\u03c8P\nij(xi,xj) +/summationdisplay\nd\u03c8Det\nd(xd) +/summationdisplay\ns\u03c8SP\ns(xs),(1)\nwhere the \ufb01rst two terms \u03c8U\ni(.)and\u03c8P\nij(.,.)are the usual unary and densely-connected\npairwise energies [17] and the last two terms are the newly introduced higher order"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e25-6db4-11ee-983d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-11", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "i(.)and\u03c8P\nij(.,.)are the usual unary and densely-connected\npairwise energies [17] and the last two terms are the newly introduced higher order\nenergies. Energies from the object detection take the form \u03c8Det\nd(xd), where vector xd\nis formed by elements of xthat correspond to the foreground pixels of the dthobject\ndetection. Superpixel label consistency based energies take the form \u03c8SP\ns(xs), where xs\nis formed by elements of xthat correspond to the pixels belonging to the sthsuperpixel.\n4.1 Object Detection Based Potentials\nSemantic segmentation errors can be classi\ufb01ed into two broad categories [35]: recognition\nand boundary errors. Boundary errors occur when semantic labels are incorrect at the\nedges of objects, and it has been shown that densely connected CRFs with appearanceconsistency terms are effective at combating this problem [17]. On the other hand,\nrecognition errors occur when object categories are recognised incorrectly or not at\nall. A CRF with only unary and pairwise potentials cannot effectively correct these\nerrors since they are caused by poor unary classi\ufb01cation. However, we propose that a\nstate-of-the-art object detector [36,37] capable of recognising and localising objects, can"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e26-6db4-11ee-bd12-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-12", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "state-of-the-art object detector [36,37] capable of recognising and localising objects, can\nprovide important information in this situation and help reduce the recognition error, as\nshown in Fig. 2.\nA key challenge in feeding-in object-detection potentials to semantic segmentation\nare false detections. A na \u00a8\u0131ve approach of adding an object detector\u2019s output to a CRF\nformulated to solve the problem of semantic segmentation would confuse the CRF\ndue to the presence of the false positives in the detector\u2019s output. Therefore, a robust\nformulation, which can automatically reject object detection false positives when they"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e27-6db4-11ee-9c4f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-13", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "Higher Order Conditional Random Fields in Deep Neural Networks 7\npositive: some of the detections obtained from the object detector may be identi\ufb01ed to\nbe false following CRF inference.\nAllYdvariables are added to the CRF which previously contained only Xivariables.\nLet each (Xd,Yd), where {Xd}={Xi\u2208 {X}|i\u2208Fd}, form a clique cdin the\nCRF. We de\ufb01ne the detection-based higher order energy associated with a particular\nassignment (xd,yd)to the clique (Xd,Yd)as follows:\n\u03c8Det\nd(Xd=xd,Yd=yd) =\uf8f1\n\uf8f4\uf8f2\n\uf8f4\uf8f3wDetsd\nnd/summationtextnd\ni=1[x(i)\nd=ld]ifyd= 0,\nwDetsd\nnd/summationtextnd\ni=1[x(i)\nd/ne}ationslash=ld]ifyd= 1,(2)\nwherend=|Fd|is the number of foreground pixels in the dthdetection,x(i)\ndis theith\nelement of the vector xd,wDetis a learnable weight parameter, and [.]is the Iverson"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e28-6db4-11ee-beac-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-14", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "dis theith\nelement of the vector xd,wDetis a learnable weight parameter, and [.]is the Iverson\nbracket. Note that this potential encourages X(i)\nds to take the value ldwhenYdis1, and\nat the same time encourages Ydto be 0when manyX(i)\nds do not take ld. In other words,\nit enforces the consistency among X(i)\nds andYd.\nAn important property of the above de\ufb01nition of \u03c8Det\nd(.)is that it can be simpli\ufb01ed\nas a sum of pairwise potentials between Ydand eachX(i)\ndfori= 1,2,...,n d. That is,\n\u03c8Det\nd(Xd=xd,Yd=yd) =nd/summationdisplay\ni=1fd(x(i)\nd,yd), where,\nfd(x(i)\nd,yd) =\uf8f1\n\uf8f2\n\uf8f3wDetsd\nnd[x(i)\nd=ld] ifyd= 0,\nwDetsd\nnd[x(i)\nd/ne}ationslash=ld] ifyd= 1.(3)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e29-6db4-11ee-baec-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-15", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "d=ld] ifyd= 0,\nwDetsd\nnd[x(i)\nd/ne}ationslash=ld] ifyd= 1.(3)\nWe make use of this simpli\ufb01cation in Section 5 when deriving the mean \ufb01eld updates\nassociated with this potential.\nFor the latent Yvariables, in addition to the joint potentials with Xvariables,\ndescribed in Eq. (2)and(3), we also include unary potentials, which are initialised\nfrom the score sdof the object detection. The underlying idea is that if the object\ndetector detects an object with high con\ufb01dence, the CRF in turn starts with a high initial\ncon\ufb01dence about the validity of that detection. This con\ufb01dence can, of course, change\nduring the CRF inference depending on other information ( e.g.segmentation unary\npotentials) available to the CRF.\nExamples of input images with multiple detections and GrabCut foreground masks\nare shown in Figure 3. Note how false detections are ignored and erroneous parts of the\nforeground mask are also largely ignored.\n4.2 Superpixel Based Potentials\nThe next type of higher order potential we use is based on the idea that superpixels"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e2a-6db4-11ee-91d6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-16", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "foreground mask are also largely ignored.\n4.2 Superpixel Based Potentials\nThe next type of higher order potential we use is based on the idea that superpixels\nobtained from oversegmentation [39, 40] quite often contain pixels from the same visual\nobject. It is therefore natural to encourage pixels inside a superpixel to have the same\nsemantic label. Once again, this should not be a hard constraint in order to keep the\n8 Arnab et al.\n(a) (b) (c) (d)\nFig. 3: Effects of imperfect foreground segmentation (a,b) Detected objects, as well\nas the foreground masks obtained from GrabCut. (c,d) Output using detection potentials.\nIncorrect parts of the foreground segmentation of the main aeroplane, and entire TV\ndetection have been ignored by CRF inference as they did not agree with the other energy\nterms. The person is a failure case though as the detection has caused part of the sofa to\nbe erroneously labelled.\n(a) (b) (c)\nFig. 4: Segmentation enhancement from superpixel based potentials (a) The output\nof our system without any superpixel potentials. (b) Superpixels obtained from the image\nusing the method of [39]. Only one \u201clayer\u201d of superpixels is shown. In practice, we"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e2b-6db4-11ee-b834-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-17", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "using the method of [39]. Only one \u201clayer\u201d of superpixels is shown. In practice, we\nused four. (c) The output using superpixel potentials. The result has improved as we\nencourage consistency over superpixel regions. This removes some of the spurious noise\nthat was present previously.\nalgorithm robust to initial superpixel segmentation errors and to violations of this key\nassumption.\nWe use two types of energies in the CRF to encourage superpixel consistency in\nsemantic segmentation. Firstly, we use the Pn-Potts model type energy [41], which is\ndescribed by,\n\u03c8SP\ns(Xs=xs) =\uf8f1\n\uf8f2\n\uf8f3wLow(l) if allx(i)\ns=l,\nwHigh otherwise,(4)\nwherewLow(l)<w High for alll, and{Xs} \u2282 {X}is a clique de\ufb01ned by a superpixel.\nThe primary idea is that assigning different labels to pixels in the same superpixel incurs\na higher cost, whereas one obtains a lower cost if the labelling is consistent throughout\nthe superpixel. Costs wLow(l)andwHigh are learnable during the end-to-end training of\nthe network."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e2c-6db4-11ee-8be2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-18", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "the superpixel. Costs wLow(l)andwHigh are learnable during the end-to-end training of\nthe network.\nSecondly, to make this potential stronger, we average initial unary potentials from the\nclassi\ufb01er (the CNN in our case), across all pixels in the superpixel and use the average\nHigher Order Conditional Random Fields in Deep Neural Networks 9\nas an additional unary potential for those pixels. During experiments, we observed that\nsuperpixel based higher order energy helps in getting rid of small spurious regions of\nwrong labels in the segmentation output, as shown in Fig. 4.\n5 Mean Field Updates and Their Differentials\nThis section discusses the mean \ufb01eld updates for the higher order potentials previously\nintroduced. These update operations are differentiable with respect to the Qi(Xi)distribution inputs at each iteration, as well as the parameters of our higher order potentials.\nThis allows us to train our CRF end-to-end as another layer of a neural network.\nTake a CRF with random variables V1,V2,...,V Nand a set of cliques C, which\nincludes unary, pairwise and higher order cliques. Mean \ufb01eld inference approximates\nthe joint distribution Pr(V=v)with the product of marginals/producttext\niQ(Vi=vi). We"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e2d-6db4-11ee-af16-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-19", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "the joint distribution Pr(V=v)with the product of marginals/producttext\niQ(Vi=vi). We\nuseQ(Vc=vc)to denote the marginal probability mass for a subset {Vc}of these\nvariables. Where there is no ambiguity, we use the short-hand notation Q(vc)to represent\nQ(Vc=vc). General mean \ufb01eld updates of such a CRF take the form [13]\nQt+1(Vi=v) =1\nZiexp\uf8eb\n\uf8ed\u2212/summationdisplay\nc\u2208C/summationdisplay\n{vc|vi=v}Qt(vc\u2212i)\u03c8c(vc)\uf8f6\n\uf8f8, (5)\nwhereQtis the marginal after the tthiteration, vcan assignment to all variables in\ncliquec,vc\u2212ian assignment to all variables in cexcept forVi,\u03c8c(vc)is the cost of\nassigning vcto the clique c, andZiis the normalisation constant that makes Q(Vi=v)\na probability mass function after the update.\nUpdates from Detection Based Potentials Following Eq. (3)above, we now use\nEq.(5)to derive the mean \ufb01eld updates related to \u03c8Det"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e2e-6db4-11ee-9e39-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-20", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "Updates from Detection Based Potentials Following Eq. (3)above, we now use\nEq.(5)to derive the mean \ufb01eld updates related to \u03c8Det\nd. The contribution from \u03c8Det\ndto\nthe update of Q(X(i)\nd=l)takes the form\n/summationdisplay\n{(xd,yd)|x(i)\nd=l}Q(xd\u2212i,yd)\u03c8Det\nd(xd,yd) =\uf8f1\n\uf8f2\n\uf8f3wDetsd\nndQ(Yd= 0) ifl=ld,\nwDetsd\nndQ(Yd= 1) otherwise,(6)\nwhere xd\u2212iis an assignment to Xdwith theithelement deleted. Using the same equations, we derive the contribution from the energy \u03c8Det\ndto the update of Q(Yd=b)to\ntake the form\n/summationdisplay\n{(xd,yd)|yd=b}Q(xd)\u03c8Det\nd(xd,yd) =\uf8f1\n\uf8f2\n\uf8f3wDetsd\nnd/summationtextnd\ni=1Q(X(i)\nd=ld) ifb= 0,\nwDetsd\nnd/summationtextnd"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e2f-6db4-11ee-a03b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-21", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "nd/summationtextnd\ni=1Q(X(i)\nd=ld) ifb= 0,\nwDetsd\nnd/summationtextnd\ni=1(1\u2212Q(X(i)\nd=ld))otherwise.\n(7)\nIt is possible to increase the number of parameters in \u03c8Det\nd(.). Since we use backpropagation to learn these parameters automatically during end-to-end training, it is desirable\nto have a high number of parameters to increase the \ufb02exibility of the model. Following\n10 Arnab et al.\nthis idea, we made the weight wDetclass speci\ufb01c, that is, a function wDet(ld)is used\ninstead ofwDetin Eqs. (2),(6)and (7). The underlying assumption is that detector\noutputs can be very helpful for certain classes, while being not so useful for classes that\nthe detector performs poorly on, or classes for which foreground segmentation is often\ninaccurate.\nNote that due to the presence of detection potentials in the CRF, error differentials\ncalculated with respect to the Xvariable unary potentials and pairwise parameters will\nno longer be valid in the forms described in [10]. The error differentials with respect to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e30-6db4-11ee-98b0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-22", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "calculated with respect to the Xvariable unary potentials and pairwise parameters will\nno longer be valid in the forms described in [10]. The error differentials with respect to\ntheXandYvariables, as well as class-speci\ufb01c detection potential weights wDet(l)are\nincluded in the supplementary material.\nUpdates for Superpixel Based Potentials The contribution from the Pn-Potts type\npotential to the mean \ufb01eld update of Q(xi=l), where pixel iis in the superpixel clique\ns, was derived in [7] as\n/summationdisplay\n{xs|x(i)\ns=l}Q(xs\u2212i)\u03c8SP\ns(xs) =wLow(l)/productdisplay\nj\u2208c,j/ne}ationslash=iQ(Xj=l)+wHigh\uf8eb\n\uf8ed1\u2212/productdisplay\nj\u2208c\u2212iQ(Xj=l)\uf8f6\n\uf8f8.\n(8)\nThis update operation is differentiable with respect to the parameters wLow(l)andwHigh,\nallowing us to optimise them via backpropagation, and also with respect to the Q(X)\nvalues enabling us to optimise previous layers in the network."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e31-6db4-11ee-935b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-23", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "allowing us to optimise them via backpropagation, and also with respect to the Q(X)\nvalues enabling us to optimise previous layers in the network.\nConvergence of parallel mean \ufb01eld updates Mean \ufb01eld with parallel updates, as\nproposed in [17] for speed, does not have any convergence guarantees in the general case.\nHowever, we usually empirically observed convergence with higher order potentials,\nwithout damping the mean \ufb01eld update as described in [7, 42]. This may be explained by\nthe fact that the unaries from the initial pixelwise-prediction part of our network provide\na good initialisation. In cases where the mean \ufb01eld energy did not converge, we still\nempirically observed good \ufb01nal segmentations.\n6 Experiments\nWe evaluate our new CRF formulation on two different datasets using the CRF-RNN\nnetwork [10] as the main baseline, since we are essentially enriching the CRF model\nof [10]. We then present ablation studies on our models.\n6.1 Experimental set-up and results\nOur deep network consists of two conceptually different, but jointly trained stages. The\n\ufb01rst, \u201cunary\u201d part of our network is formed by the FCN-8s architecture [3]. It is initialised"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e32-6db4-11ee-a1be-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-24", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "\ufb01rst, \u201cunary\u201d part of our network is formed by the FCN-8s architecture [3]. It is initialised\nfrom the Imagenet-trained VGG-16 network [2], and then \ufb01ne-tuned with data from\nthe VOC 2012 training set [4], extra VOC annotations of [43] and the MS COCO [44]\ndataset.\nHigher Order Conditional Random Fields in Deep Neural Networks 11\nTable 1: Comparison of each higher order\npotential with baseline on VOC 2012 reduced validation set\nMethodReduced\nval set(%)\nBaseline (unary + pairwise) [10] 72.9\nSuperpixels only 74.0\nDetections only 74.9\nDetections and Superpixels 75.8Table 2: Mean IoU accuracy on VOC 2012\ntest set. All methods are trained with MS\nCOCO [44] data\nMethod Test set(%)\nOurs 77.9\nDPN [9] 77.5\nCentrale Super Boundaries [45] 75.7\nDilated Convolutions [46] 75.3\nBoxSup [35] 75.2\nDeepLab Attention [47] 75.1\nCRF-RNN (baseline) [10] 74.7\nDeepLab WSSL [48] 73.9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e33-6db4-11ee-a304-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-25", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "DeepLab Attention [47] 75.1\nCRF-RNN (baseline) [10] 74.7\nDeepLab WSSL [48] 73.9\nDeepLab [12] 72.7\nTable 3: Mean Intersection over Union (IoU) results on PASCAL Context validation set\ncompared to other current methods.\nMethod Ours BoxSup [35] ParseNet [49] CRF-RNN [10] FCN-8s [3] CFM [28]\nMean IoU (%) 41.3 40.5 40.4 39.3 37.8 34.4\nThe output of the \ufb01rst stage is fed into our CRF inference network. This is implemented using the mean \ufb01eld update operations and their differentials described in\nSection 5. Five iterations of mean \ufb01eld inference were performed during training. Our\nCRF network has two additional inputs in addition to segmentation unaries obtained from\nthe FCN-8s network: data from the object detector and superpixel oversegmentations of\nthe image.\nWe used the publicly available code and model of the Faster R-CNN [37] object detector. The fully automated version of GrabCut [38] was then used to obtain foregrounds\nfrom the detection bounding boxes. These choices were made after conducting preliminary experiments with alternate detection and foreground segmentation algorithms."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e34-6db4-11ee-ae2a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-26", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "from the detection bounding boxes. These choices were made after conducting preliminary experiments with alternate detection and foreground segmentation algorithms.\nFour levels of superpixel oversegmentations were used, with increasing superpixel\nsize to de\ufb01ne the cliques used in this potential. Four levels were used since performance\non the VOC validation set stopped increasing after this number. We used the superpixel\nmethod of [39] as it was shown to adhere to object boundaries the best [40], but our\nmethod generalises to any oversegmentation algorithm.\nWe trained the full network end-to-end, optimising the weights of the CNN classi\ufb01er\n(FCN-8s) and CRF parameters jointly. We initialised our network using the publicly\navailable weights of [10], and trained with a learning rate of 10\u221210and momentum of\n0.99. The learning rate is low because the loss was not normalised by the number of\npixels in the training image. This is to have a larger loss for images with more pixels.\nWhen training our CRF, we only used VOC 2012 data [4] as it has the most accurate\nlabelling, particularly around boundaries.\n12 Arnab et al.\nPASCAL VOC 2012 Dataset The improvement obtained by each higher order potential\nwas evaluated on the same reduced validation set [3] used by our baseline [10]. As Table"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e35-6db4-11ee-84cb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-27", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "PASCAL VOC 2012 Dataset The improvement obtained by each higher order potential\nwas evaluated on the same reduced validation set [3] used by our baseline [10]. As Table\n1 shows, each new higher order potential improves the mean IoU over the baseline. We\nonly report test set results for our best method since the VOC guidelines discourage\nthe use of the test set for ablation studies. On the test set (Table 2), we outperform our\nbaseline by 3.2% which equates to a 12.6% reduction in the error rate. This sets a new\nstate-of-the-art on the VOC dataset. Qualitative results highlighting success and failure\ncases of our algorithm, as well as more detailed results, are shown in our supplementary\nmaterial.\nPASCAL Context Table 3 shows our state-of-the-art results on the recently released\nPASCAL Context dataset [50]. We trained on the provided training set of 4998 images,\nand evaluated on the validation set of 5105 images. This dataset augments VOC with\nannotations for all objects in the scene. As a result, there are 59 classes as opposed to\nthe 20 in the VOC dataset. Many of these new labels are \u201cstuff\u201d classes such as \u201cgrass\u201d\nand \u201csky\u201d. Our object detectors are therefore only trained for 20 of the 59 labels in this"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e36-6db4-11ee-a6bd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-28", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "and \u201csky\u201d. Our object detectors are therefore only trained for 20 of the 59 labels in this\ndataset. Nevertheless, we improve by 0.8% over the previous state-of-the-art [35] and\n2% over our baseline [10].\n6.2 Ablation Studies\nWe perform additional experiments to determine the errors made by our system, show\nthe bene\ufb01ts of end-to-end training and compare our detection potentials to a simpler\nbaseline. Unless otherwise stated, these experiments are performed on the VOC 2012\nreduced validation set.\nError Analysis To analyse the improvements made by our higher order potentials, we\nseparately evaluate the performance on the \u201cboundary\u201d and \u201cinterior\u201d regions in a similar\nmanner to [35]. As shown in Fig. 5 c) and d), we consider a narrow band (trimap [16])\naround the \u201cvoid\u201d labels annotated in the VOC 2012 reduced validation set. The mean\nIoU of pixels lying within this band is termed the \u201cBoundary IoU\u201d whilst the \u201cInterior\nIoU\u201d is evaluated outside this region.\nFig. 5 shows our results as the trimap width is varied. Adding the detection potentials"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e37-6db4-11ee-beda-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-29", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "IoU\u201d is evaluated outside this region.\nFig. 5 shows our results as the trimap width is varied. Adding the detection potentials\nimproves the Interior IoU over our baseline (only pairwise potentials [10]) as the object\ndetector may recognise objects in the image which the pixelwise classi\ufb01cation stage of\nour network may have missed out. However, the detection potentials also improve the\nBoundary IoU for all tested trimap widths as well. Improving the recognition of pixels\nin the interior of an object also helps with delineating the boundaries since the strength\nof the pairwise potentials exerted by the Qdistributions at each of the correctly-detected\npixels increase.\nOur superpixel priors also increase the Interior IoU with respect to the baseline.\nEncouraging consistency over regions helps to get rid of spurious regions of wrong\nlabels (as shown in Fig. 4). Fig. 5 suggests that most of this improvement occurs in the\ninterior of an object. The Boundary IoU is slightly lower than the baseline, and this may\nHigher Order Conditional Random Fields in Deep Neural Networks 13\na) Image (b) Ground truth (c) Boundary (d) Interior\n0 5 10 15 20 25 30 35 406668707274767880\nTrimap WidthInterior IoU\n  \nPairwise only"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e38-6db4-11ee-b46e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-30", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "0 5 10 15 20 25 30 35 406668707274767880\nTrimap WidthInterior IoU\n  \nPairwise only\nSuperpixels\nDetections\nDetections and Superpixels\n0 5 10 15 20 25 30 35 40505560657075\nTrimap WidthBoundary IoU\n  \nPairwise only\nSuperpixels\nDetections\nDetections and Superpixels\n(e) Interior IoU (f) Boundary IoU\nFig. 5: Error analysis on VOC 2012 reduced validation set The IoU is computed for\nboundary and interior regions for various trimap widths. An example of the Boundary\nand Interior regions for a sample image using a width of 9 pixels is shown in white in\nthe top row. Black regions are ignored in the IoU calculation.\nbe due to the fact that superpixels do not always align correctly with the edges of an\nobject (the \u201cboundary recall\u201d of various superpixel methods are evaluated in [40]).\nWe can see that the combination of detection and superpixel potentials results in a\nsubstantial improvement in our Interior IoU. This is the primary reason our overall IoU\non the VOC benchmark increases with higher order potentials.\nBene\ufb01ts of end-to-end training Table 4 shows how end-to-end training outperforms"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e39-6db4-11ee-b59c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-31", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "on the VOC benchmark increases with higher order potentials.\nBene\ufb01ts of end-to-end training Table 4 shows how end-to-end training outperforms\npiecewise training. We trained the CRF piecewise by freezing the weights of the unary\npart of the network, and only learning the CRF parameters.\nOur results in Table 2 used the FCN-8s [3] architecture to generate unaries. To show\nthat our higher order potentials improve performance regardless of the underlying CNN\nused for producing unaries, we also perform an experiment using our reimplementation\nTable 4: Comparison of mean IoU (%) obtained on VOC 2012 reduced validation set\nfrom end-to-end and piecewise training\nMethod FCN-8s DCN\nUnary only, \ufb01ne-tuned on COCO 68.3 68.6\nPairwise CRF trained piecewise 69.5 70.7\nPairwise CRF trained end-to-end 72.9 72.5\nHigher Order CRF trained piecewise 73.6 73.5\nHigher Order CRF trained end-to-end 75.8 75.0\nTest set performance of best model 77.9 76.9\n14 Arnab et al.\nof the \u201cfront-end\u201d module proposed in the Dilated Convolution Network (DCN) of [46]"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e3a-6db4-11ee-b475-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-32", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "14 Arnab et al.\nof the \u201cfront-end\u201d module proposed in the Dilated Convolution Network (DCN) of [46]\ninstead of FCN-8s.\nTable 4 shows that end-to-end training of the CRF yields considerable improvements\nover piecewise training. This was the case when using either FCN-8s or DCN for\nobtaining the initial unaries before performing CRF inference with higher order potentials.\nThis suggests that our CRF network module can be plugged into different architectures\nand achieve performance improvements.\nBaseline for detections To evaluate the ef\ufb01cacy of our detection potentials, we formulate a simpler baseline since no other methods use detection information at inference\ntime (BoxSup [35] derives ground truth for training using ground-truth bounding boxes).\nOur baseline is similar to CRF-RNN [10], but prior to CRF inference, we take the\nsegmentation mask from the object detection and add a unary potential proportional to\nthe detector\u2019s con\ufb01dence to the unary potentials for those pixels. We then perform mean\ufb01eld inference (with only pairwise terms) on these \u201caugmented\u201d unaries. Using this"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e3b-6db4-11ee-b176-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-33", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "method, the mean IoU increases from 72.9% to 73.6%, which is signi\ufb01cantly less than\nthe 74.9% which we obtained using only our detection potentials without superpixels\n(Table 1).\nOur detection potentials perform better since our latent Ydetection variables model\nwhether the detection hypothesis is accepted or not. Our CRF inference is able to evaluate\nobject detection inputs in light of other potentials. Inference increases the relative score\nof detections which agree with the segmentation, and decreases the score of detections\nwhich do not agree with other energies in the CRF. Figures 2 b) and d) show examples\nof false-positive detections that have been ignored and correct detections that have been\nused to re\ufb01ne our segmentation. Our baseline, on the other hand, is far more sensitive to\nerroneous detections as it cannot adjust the weight given to them during inference.\n7 Conclusion\nWe presented a CRF model with two different higher order potentials to tackle the\nsemantic segmentation problem. The \ufb01rst potential is based on the intuitive idea that\nobject detection can provide useful cues for semantic segmentation. Our formulation is\ncapable of automatically rejecting false object detections that do not agree at all with\nthe semantic segmentation. Secondly, we used a potential that encourages superpixels to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e3c-6db4-11ee-a250-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-34", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "capable of automatically rejecting false object detections that do not agree at all with\nthe semantic segmentation. Secondly, we used a potential that encourages superpixels to\nhave consistent labelling. These two new potentials can co-exist with the usual unary\nand pairwise potentials in a CRF.\nImportantly, we showed that ef\ufb01cient mean \ufb01eld inference is still possible in the\npresence of the new higher order potentials and derived the explicit forms of the mean\n\ufb01eld updates and their differentials. This enabled us to implement the new CRF model\nas a stack of CNN layers and to train it end-to-end in a uni\ufb01ed deep network with a\npixelwise CNN classi\ufb01er. We experimentally showed that the addition of higher order\npotentials results in a signi\ufb01cant increase in semantic segmentation accuracy allowing us\nto reach state-of-the-art performance.\nThis work was supported by ERC grant ERC-2012-AdG 321162-HELIOS, EPSRC grant\nSeebibyte EP/M013774/1, EPSRC/MURI grant EP/N019474/1 and the Clarendon Fund.\nHigher Order Conditional Random Fields in Deep Neural Networks 15\nReferences"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e3d-6db4-11ee-98e6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-35", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "Higher Order Conditional Random Fields in Deep Neural Networks 15\nReferences\n1.Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classi\ufb01cation with deep convolutional\nneural networks. In: NIPS. (2012) 1097\u20131105\n2.Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image\nrecognition. In: ICLR. (2015)\n3.Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation.\nIn: CVPR. (2015)\n4.Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual\nobject classes (voc) challenge. IJCV (2010)\n5.Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Associative hierarchical crfs for object class\nimage segmentation. In: ICCV . (2009) 739\u2013746\n6.Ladick \u00b4y, L., Sturgess, P., Alahari, K., Russell, C., Torr, P.H.S.: What, where and how many?"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e3e-6db4-11ee-9606-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-36", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "combining object detectors and crfs. In: ECCV . (2010) 424\u2013437\n7.Vineet, V., Warrell, J., Torr, P.H.: Filter-based mean-\ufb01eld inference for random \ufb01elds with\nhigher-order terms and product label-spaces. IJCV (2014)\n8.Hariharan, B., Arbel \u00b4aez, P., Girshick, R., Malik, J.: Simultaneous detection and segmentation.\nIn: ECCV . Springer (2014) 297\u2013312\n9.Liu, Z., Li, X., Luo, P., Loy, C.C., Tang, X.: Semantic image segmentation via deep parsing\nnetwork. In: ICCV . (2015)\n10.Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., Torr,\nP.: Conditional random \ufb01elds as recurrent neural networks. In: ICCV . (2015)\n11.Lin, G., Shen, C., Reid, I.: Ef\ufb01cient piecewise training of deep structured models for semantic\nsegmentation. In: CVPR. (2016)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e3f-6db4-11ee-bd65-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-37", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "segmentation. In: CVPR. (2016)\n12.Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR (2015)\n13.Koller, D., Friedman, N.: Probabilistic graphical models: principles and techniques. MIT\npress (2009)\n14.Shotton, J., Winn, J., Rother, C., Criminisi, A.: Textonboost for image understanding: Multiclass object recognition and segmentation by jointly modeling texture, layout, and context.\nIJCV (2009)\n15.He, X., Zemel, R.S., Carreira-Perpi \u02dcn\u00b4an, M. \u00b4A.: Multiscale conditional random \ufb01elds for image\nlabeling. In: CVPR, IEEE (2004)\n16.Kohli, P., Ladicky, L., Torr, P.: Robust higher order potentials for enforcing label consistency.\nIJCV 82(3) (2009) 302\u2013324\n17.Kr\u00a8ahenb \u00a8uhl, P., Koltun, V.: Ef\ufb01cient inference in fully connected CRFs with Gaussian edge"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e40-6db4-11ee-99d2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-38", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "17.Kr\u00a8ahenb \u00a8uhl, P., Koltun, V.: Ef\ufb01cient inference in fully connected CRFs with Gaussian edge\npotentials. In: NIPS. (2011)\n18.Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Graph cut based inference with co-occurrence\nstatistics. In: ECCV . (2010) 239\u2013253\n19.Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., Belongie, S.: Objects in context.\nIn: ICCV . (2007) 1\u20138\n20.Gonfaus, J.M., Boix, X., Van de Weijer, J., Bagdanov, A.D., Serrat, J., Gonzalez, J.: Harmony\npotentials for joint classi\ufb01cation and segmentation. In: CVPR, IEEE (2010) 3280\u20133287\n21.Yao, J., Fidler, S., Urtasun, R.: Describing the Scene as a Whole: Joint Object Detection,\nScene Classi\ufb01cation and Semantic Segmentation. In: CVPR. (2012) 702\u2013709"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e41-6db4-11ee-909e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-39", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "Scene Classi\ufb01cation and Semantic Segmentation. In: CVPR. (2012) 702\u2013709\n22.Wojek, C., Schiele, B.: A dynamic conditional random \ufb01eld model for joint labeling of object\nand scene classes. In: ECCV , Springer (2008) 733\u2013747\n23.Lin, G., Shen, C., Reid, I., van den Hengel, A.: Deeply learning the messages in message\npassing inference. In: NIPS. (2015) 361\u2013369\n16 Arnab et al.\n24.Yang, Y., Hallman, S., Ramanan, D., Fowlkes, C.C.: Layered object models for image\nsegmentation. PAMI (2012)\n25.Sun, M., Kim, B.s., Kohli, P., Savarese, S.: Relating things and stuff via objectproperty\ninteractions. PAMI 36(7) (2014) 1370\u20131383\n26.Carreira, J., Caseiro, R., Batista, J., Sminchisescu, C.: Semantic segmentation with secondorder pooling. In: ECCV . (2012) 430\u2013443\n27.Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e42-6db4-11ee-ba2e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-40", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "27.Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene\nlabeling. PAMI (2013)\n28.Dai, J., He, K., Sun, J.: Convolutional feature masking for joint object and stuff segmentation.\nCVPR (2015)\n29.Tompson, J.J., Jain, A., LeCun, Y ., Bregler, C.: Joint training of a convolutional network and\na graphical model for human pose estimation. In: NIPS. (2014) 1799\u20131807\n30.Deng, Z., Zhai, M., Chen, L., Liu, Y., Muralidharan, S., Roshtkhari, M.J., Mori, G.: Deep\nstructured models for group activity recognition. In: BMVC. (2015)\n31.Ionescu, C., Vantzos, O., Sminchisescu, C.: Matrix backpropagation for deep networks with\nstructured layers. In: ICCV . (2015) 2965\u20132973\n32.Domke, J.: Learning graphical model parameters with approximate marginal inference. PAMI\n(2013)\n33.Kr\u00a8ahenb \u00a8uhl, P., Koltun, V.: Parameter learning and convergent inference for dense random"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e43-6db4-11ee-9820-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-41", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "(2013)\n33.Kr\u00a8ahenb \u00a8uhl, P., Koltun, V.: Parameter learning and convergent inference for dense random\n\ufb01elds. In: ICML. (2013)\n34.Ross, S., Munoz, D., Hebert, M., Bagnell, J.A.: Learning message-passing inference machines\nfor structured prediction. In: CVPR. (2011)\n35.Dai, J., He, K., Sun, J.: Boxsup: Exploiting bounding boxes to supervise convolutional\nnetworks for semantic segmentation. In: ICCV . (2015)\n36. Girshick, R.: Fast r-cnn. In: ICCV . (2015)\n37.Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with\nregion proposal networks. In: NIPS. (2015)\n38.Rother, C., Kolmogorov, V., Blake, A.: Grabcut: Interactive foreground extraction using\niterated graph cuts. ACM TOG (2004)\n39.Felzenszwalb, P.F., Huttenlocher, D.P.: Ef\ufb01cient graph-based image segmentation. IJCV\n(2004)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e44-6db4-11ee-ab1e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-42", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "(2004)\n40.Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: Slic superpixels compared\nto state-of-the-art superpixel methods. PAMI 34(11) (2012) 2274\u20132282\n41. Kohli, P., Kumar, M.P., Torr, P.H.: P3 & beyond: Solving energies with higher order cliques.\nIn: CVPR. (2007)\n42.Baqu, P., Bagautdinov, T., Fleuret, F., Fua, P.: Principled Parallel Mean-Field Inference for\nDiscrete Random Fields. In: CVPR. (2016)\n43.Hariharan, B., Arbel \u00b4aez, P., Bourdev, L., Maji, S., Malik, J.: Semantic contours from inverse\ndetectors. In: ICCV , IEEE (2011) 991\u2013998\n44.Lin, T.Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \u00b4ar, P., Zitnick, C.L.:\nMicrosoft coco: Common objects in context. In: ECCV . Springer (2014) 740\u2013755"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e45-6db4-11ee-9e29-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-43", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "Microsoft coco: Common objects in context. In: ECCV . Springer (2014) 740\u2013755\n45.Kokkinos, I.: Pushing the boundaries of boundary detection using deep learning. In: ICLR.\n(2016)\n46.Yu, F., Koltun, V .: Multi-scale context aggregation by dilated convolutions. In: ICLR. (2016)\n47.Chen, L.C., Yang, Y ., Wang, J., Xu, W., Yuille, A.L.: Attention to scale: Scale-aware semantic\nimage segmentation. In: CVPR. (2016)\n48.Papandreou, G., Chen, L., Murphy, K., Yuille, A.L.: Weakly- and semi-supervised learning of\na DCNN for semantic image segmentation. In: ICCV . (2015)\n49.Liu, W., Rabinovich, A., Berg, A.C.: Parsenet: Looking wider to see better. arXiv preprint\narXiv:1506.04579 (2015)\nHigher Order Conditional Random Fields in Deep Neural Networks 17\n50.Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., et al.: The"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e46-6db4-11ee-9735-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-44", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "role of context for object detection and semantic segmentation in the wild. In: CVPR, IEEE\n(2014) 891\u2013898\n51.Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In:\nICCV . (2015)\n52.Mostajabi, M., Yadollahpour, P., Shakhnarovich, G.: Feedforward semantic segmentation\nwith zoom-out features. In: CVPR. (2015)\n53.Dong, J., Chen, Q., Yan, S., Yuille, A.: Towards uni\ufb01ed object detection and semantic\nsegmentation. In: ECCV . (2014) 299\u2013314\n54.Carreira, J., Caseiro, R., Batista, J., Sminchisescu, C.: Free-form region description with\nsecond-order pooling. PAMI (2014)\nAppendix\nAppendix A of this supplementary material presents the derivatives of the mean \ufb01eld\nupdates which we use for inference in our Conditional Random Field (CRF). Appendix\nB shows detailed qualitative results for the experiments described in our main paper.\nA Derivatives of Mean Field Updates\nThe pseudocode for the mean \ufb01eld inference algorithm with latent Ydetection variables"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e47-6db4-11ee-bb00-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-45", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "A Derivatives of Mean Field Updates\nThe pseudocode for the mean \ufb01eld inference algorithm with latent Ydetection variables\nis shown below in Algorithm 1. We use the same notation used in the main paper.\nAlgorithm 1 Mean Field Inference\nQ0(Xi=l)\u21901\nZiexp/parenleftbig\n\u2212\u03c8U\ni(l)/parenrightbig\n,\u2200i,l\nQ0(Yd=b)\u2190sb\nd(1\u2212sd)(1\u2212b),\u2200d,b\n\u22bfInitialisation\nfort = 0 :T\u22121do\nEt(Xi=l)\u2190UnaryUpdate + PairwiseUpdate +\nDetectionUpdate + SuperpixelUpdate, \u2200i,l\nEt(Yd=b)\u2190YUnaryUpdate + Y DetectionUpdate\n\u22bfMean \ufb01eld updates\nQt+1(Xi=l)\u21901\nZiexp/parenleftbig\n\u2212Et(Xi=l)/parenrightbig\n,\u2200i,l\nQt+1(Yd=b)\u21901\nZdexp/parenleftbig\n\u2212Et(Yd=b)/parenrightbig\n,\u2200d,b\n\u22bfNormalising\nend for"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e48-6db4-11ee-bad4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-46", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "Zdexp/parenleftbig\n\u2212Et(Yd=b)/parenrightbig\n,\u2200d,b\n\u22bfNormalising\nend for\nFor the explicit forms of the UnaryUpdate andPairwiseUpdate above, and their\ndifferentials, we refer the reader to [10] and discuss the terms DetectionUpdate and\nSuperpixelUpdate in detail below.\nLet us assume that only one object detection of the form (ld,sd,Fd)is available for\nthe image under consideration. When multiple detections are present, simply a summation of the updates and differentials discussed below apply. Therefore, no generality\n18 Arnab et al.\nis lost with this assumption. Similarly, we can assume that only one superpixel clique\n{Xs}is present, without a loss of generality.\nAssuming that pixel iin Algorithm 1 belongs to Fd, Eq. (6) in the main paper\ndescribed the exact form of DetectionUpdate . Similarly, assuming that pixel ibelongs\nto{Xs}Eq. (8) described the form of SuperpixelUpdate .\nLetLdenote the value of the loss function calculated at the output of the deep\nnetwork. This could be the softmax loss or any other appropriate loss function. During\nbackpropagation, we get the error signal\u2202L"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e49-6db4-11ee-866b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-47", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "network. This could be the softmax loss or any other appropriate loss function. During\nbackpropagation, we get the error signal\u2202L\n\u2202QTat the output of the mean \ufb01eld inference.\nUsing this error information, we need to compute the derivative of the loss Lwith respect\nto theXunaries and various CRF parameters. Note that, if we compute the relevent\ndifferentials for only one iteration of the mean \ufb01eld algorithm, it is possible to calculate\nthem for multiple iterations using the recurrent behaviour of the iterations.\nNote that, by looking at Normalising step of Algorithm 1, it is trivial to calculate\n\u2202Qt+1\n\u2202Et. Therefore, we can then calculate\u2202L\n\u2202Etusing the chain rule. This is same as\nbackpropagation of the usual softmax operation in a deep network (up to a negative sign).\nUsing this observation we can calculate the necessary differentials to take the forms\nshown below:\n\u2202L\n\u2202wDet=sd\nndnd/summationdisplay\ni=1/parenleftBig\u2202L\nEt(X(i)\nd=ld)Qt(Yd= 1) + (9)\n/summationdisplay\nl/prime/negationslash=ld\u2202L"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e4a-6db4-11ee-bfc5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-48", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "d=ld)Qt(Yd= 1) + (9)\n/summationdisplay\nl/prime/negationslash=ld\u2202L\n\u2202Et(X(i)\nd=l/prime)Qt(Yd= 1)/parenrightBig\n+\n\u2202L\n\u2202Et(Yd= 0)sd\nndnd/summationdisplay\ni=1Qt(X(i)\nd=ld) +\n\u2202L\n\u2202Et(Yd= 1)sd\nndnd/summationdisplay\ni=1/parenleftBig\n1\u2212Qt(X(i)\nd=ld)/parenrightBig\n\u2202L\n\u2202Qt(X(i)\nd=ld)=wDet\u2202L\n\u2202Et(Yd= 0)\u2212wDet\u2202L\n\u2202Et(Yd= 1)(10)\n\u2202L\n\u2202Qt(Yd= 0)=wDetsd\nndnd/summationdisplay\ni=1/parenleftBigg\n\u2202L\nEt(X(i)\nd=ld)/parenrightBigg\n(11)\n\u2202L\n\u2202Qt(Yd= 1)=wDetsd\nndnd/summationdisplay"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e4b-6db4-11ee-a4bc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-49", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "d=ld)/parenrightBigg\n(11)\n\u2202L\n\u2202Qt(Yd= 1)=wDetsd\nndnd/summationdisplay\ni=1/summationdisplay\nl/negationslash=ld/parenleftBigg\n\u2202L\n\u2202Et(X(i)\nd=l/prime)/parenrightBigg\n(12)\n\u2202L\n\u2202wLow(l)=/summationdisplay\ni\u2208s\uf8eb\n\uf8ed\u2202L\n\u2202Et(X(i)\ns=l)/productdisplay\nj\u2208c,j/negationslash=iQt(Xj=l)\uf8f6\n\uf8f8 (13)\nHigher Order Conditional Random Fields in Deep Neural Networks 19\n\u2202L\n\u2202wHigh=/summationdisplay\ni\u2208s/summationdisplay\nl\u2208L\uf8eb\n\uf8ed\u2202L\n\u2202Et(X(i)\ns=l)\uf8eb\n\uf8ed1\u2212/productdisplay\nj\u2208c,j/negationslash=iQt(Xj=l)\uf8f6\n\uf8f8\uf8f6\n\uf8f8 (14)\nEffect of the superpixel potentials on the derivatives\u2202L"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e4c-6db4-11ee-91c4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-50", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "\uf8f8\uf8f6\n\uf8f8 (14)\nEffect of the superpixel potentials on the derivatives\u2202L\n\u2202Qt(Xi=l)were negligible.\nTherefore, we ignored them in our calculations.\nB Additional Experimental Results\nTable 5 presents more detailed results of our method, and that of other state-of-the-art\ntechniques, on the PASCAL VOC 2012 test set. In particular, we present the accuracy\nfor every class in the VOC test set. Note that our per-class accuracy improves over our\nbaseline, CRF-RNN [10], for all of the 20 classes in PASCAL VOC.\nFigure 6 shows more sample results of our system, compared to our baseline, CRFas-RNN [10]. Figure 7 shows examples of failure cases of our method. Figure 8 examines\nthe effect of each of our potentials. Finally, Figure 9 shows a qualitative comparison\nbetween the output of our system and other current methods on the PASCAL VOC 2012\ntest set.\n20 Arnab et al.Table 5: Comparison of the mean Intersection over Union (IoU) accuracy of our approach and other state-of-the-art methods on the Pascal\nVOC 2012 test set. Scores for other methods were taken from the original authors\u2019 publications.\nMethods trained with\nCOCOMean"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e4d-6db4-11ee-b678-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-51", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "VOC 2012 test set. Scores for other methods were taken from the original authors\u2019 publications.\nMethods trained with\nCOCOMean\nIoU(%)aeroplanebike bird boat bottle bus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv\nOur method 77.9 92.5 59.1 90.3 70.6 74.4 92.4 84.1 88.3 36.8 85.6 67.1 85.1 86.9 88.2 82.6 62.6 85.0 56.2 81.9 72.5\nDPN [9] 77.5 89.0 61.6 87.7 66.8 74.7 91.2 84.3 87.6 36.5 86.3 66.1 84.4 87.8 85.6 85.4 63.6 87.3 61.3 79.4 66.4\nSuper Bound. [45] 75.7 90.3 37.9 89.6 67.8 74.6 89.3 84.1 89.1 35.8 83.6 66.2 82.9 81.7 85.6 84.6 60.3 84.8 60.7 78.3 68.3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e4e-6db4-11ee-9e28-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-52", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "Dilated Conv. [46] 75.3 91.7 39.6 87.8 63.1 71.8 89.7 82.9 89.8 37.2 84.0 63.0 83.3 89.0 83.8 85.1 56.8 87.6 56.0 80.2 64.7\nBoxSup [35] 75.2 89.8 38.0 89.2 68.9 68.0 89.6 83.0 87.7 34.4 83.6 67.1 81.5 83.7 85.2 83.5 58.6 84.9 55.8 81.2 70.7\nAttention [47] 75.1 92.0 41.2 87.8 57.2 72.7 92.8 85.9 90.5 30.5 78.0 62.8 85.8 85.3 87.2 85.6 57.7 85.1 56.5 83.0 65.0\nCRF-as-RNN [10] 74.7 90.4 55.3 88.7 68.4 69.8 88.3 82.4 85.1 32.6 78.5 64.4 79.6 81.9 86.4 81.8 58.6 82.4 53.5 77.4 70.1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e4f-6db4-11ee-96da-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-53", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "WSSL [48] 73.9 89.2 46.7 88.5 63.5 68.4 87.0 81.2 86.3 32.6 80.7 62.4 81.0 81.3 84.3 82.1 56.2 84.6 58.2 76.2 67.2\nDeepLab [12] 72.7 89.1 38.3 88.1 63.3 69.7 87.1 83.1 85.0 29.3 76.5 56.5 79.8 77.9 85.8 82.4 57.4 84.3 54.9 80.5 64.1\nMethods trained\nwithout COCO\nOur method 73.9 89.3 40.0 81.6 65.1 71.7 90.1 81.3 85.7 32.4 82.1 62.2 82.6 83.7 84.5 81.1 60.8 85.2 49.6 80.0 69.9\nDPN [9] 74.1 87.7 59.4 78.4 64.9 70.3 89.3 83.5 86.1 31.7 79.9 62.6 81.9 80.0 83.5 82.3 60.5 83.2 53.4 77.9 65.0"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e50-6db4-11ee-bb0f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-54", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "DeconvNet [51] 72.5 89.9 39.3 79.7 63.9 68.2 87.4 81.2 86.1 28.5 77.0 62.0 79.0 80.3 83.6 80.2 58.8 83.4 54.3 80.7 65.0\nCRF-as-RNN [10] 72.0 87.5 39.0 79.7 64.2 68.3 87.6 80.8 84.4 30.4 78.2 60.4 80.5 77.8 83.1 80.6 59.5 82.8 47.8 78.3 67.1\nDeepLab [12] 71.6 84.4 54.5 81.5 63.6 65.9 85.1 79.1 83.4 30.7 74.1 59.8 79.0 76.1 83.2 80.8 59.7 82.2 50.4 73.1 63.7\nPiecewise [11] 70.7 87.5 37.7 75.8 57.4 72.3 88.4 82.6 80.0 33.4 71.5 55.0 79.3 78.4 81.3 82.7 56.1 79.8 48.6 77.1 66.3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e51-6db4-11ee-a953-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-55", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "Zoomout [52] 69.6 85.6 37.3 83.2 62.5 66.0 85.1 80.7 84.9 27.2 73.2 57.5 78.1 79.2 81.1 77.1 53.6 74.0 49.2 71.7 63.3\nFCN-8s [3] 62.2 76.8 34.2 68.9 49.4 60.3 75.3 74.7 77.6 21.4 62.5 46.8 71.8 63.9 76.5 73.9 45.2 72.4 37.4 70.9 55.1\nCFM [28] 61.8 75.7 26.7 69.5 48.8 65.6 81.0 69.2 73.3 30.0 68.7 51.5 69.1 68.1 71.7 67.5 50.4 66.5 44.4 58.9 53.5\nNUS UDS [53] 50.0 67.0 24.5 47.2 45.0 47.9 65.3 60.6 58.5 15.5 50.8 37.4 45.8 59.9 62.0 52.7 40.8 48.2 36.8 53.1 45.6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e52-6db4-11ee-9cf5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-56", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "O2P [54] 47.8 64.0 27.3 54.1 39.2 48.7 56.6 57.7 52.5 14.2 54.8 29.6 42.2 58.0 54.8 50.2 36.6 58.6 31.6 48.4 38.6\nHigher Order Conditional Random Fields in Deep Neural Networks 21\nInput image CRF-as-RNN [10] Our method Ground truth\nFig. 6: Examples of images where our method has improved over our baseline,\nCRF-as-RNN [10]. The input images have the detection bounding boxes overlaid on\nthem. Note that the method of [10] does not make use of this information. The improvements from our method are due to our detection potentials, as well as our superpixel\nbased potentials. Note that all images are from the reduced validation set of VOC 2012\nand have not been trained on at all. Best viewed in color.\n22 Arnab et al.\nInput image CRF-as-RNN [10] Our method Ground truth\nFig. 7: Examples of failure cases where our method has performed poorly. The \ufb01rst\nrow shows an example of how the detection of the person has now resulted in the sofa\nbeing misclassi\ufb01ed (although our system is able to reject the other false detection). Our"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e53-6db4-11ee-bca4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-57", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "row shows an example of how the detection of the person has now resulted in the sofa\nbeing misclassi\ufb01ed (although our system is able to reject the other false detection). Our\nsuperpixel potentials have a tendency to remove spurious noise by enforcing consistency\nwithin regions. However, as shown in the second row, sometimes the \u201cnoise\u201d being\nremoved is actually the correct label. In the other cases, we are limited by our pixelwise\nclassi\ufb01cation unaries which are poor. Our superpixel and detection potentials are not\nalways able to compensate for this. Note that all images are from the reduced validation\nset of VOC 2012 and have not been trained on at all. The input images have the detection\nbounding boxes overlaid on them. Note that the method of [10] does not make use of\nthis information. Best viewed in color.\nHigher Order Conditional Random Fields in Deep Neural Networks 23\nInput image Pairwise onlySuperpixels\nonlyDetections\nonlyDetections\nand\nSuperpixelsGround truth\nFig. 8: Comparison of pairwise potentials, superpixel and pairwise potentials, detection and pairwise potentials, and a combination of all three (Row 1 and 2) These\nare examples where superpixel potentials help to remove spurious noise in the output"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e54-6db4-11ee-a798-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-58", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "are examples where superpixel potentials help to remove spurious noise in the output\nbut detection potentials do not affect the result. The \ufb01nal result still improves when\nall potentials are combined. (Row 3) Detection potentials greatly improve the result\nby recognising the train correctly (the pixelwise unaries are largest for \u201cbus\u201d). And\nsuperpixels, when combined with detections, slightly improve the output. (Row 4) An\nexample where both superpixel and detection potentials improve the \ufb01nal output. (Row 5)\nA case where the superpixel worsens the result as, although the output is more consistent\namong superpixel regions, some pixels have had their correct labels removed. However,\nthe correct detection improves the result, and the output of combining superpixel and\ndetection potentials is actually better than either potential in isolation. (Row 6) Here, the\ndetection (although correct) worsens the output due to its imprecise foreground mask.\nSuperpixel potentials also exacerbate the result, since the legs of the chair and the chair\u2019s\nshadow are confused to be part of the same superpixel region. However, when the two\npotentials are combined, the result is slightly better than with only detection potentials.\n24 Arnab et al."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e55-6db4-11ee-8678-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1511.08119-59", "title": "Higher Order Conditional Random Fields in Deep Neural Networks", "source": "http://arxiv.org/pdf/1511.08119", "start_index": 0, "chroma:document": "potentials are combined, the result is slightly better than with only detection potentials.\n24 Arnab et al.\nInput image FCN-8s [3] Deeplab [12] CRF-as-RNN [10] Our method Ground truth\nFig. 9: Qualitative comparison with other current methods. Sample results of our\nmethod compared to other current techniques on VOC 2012. We reproduced the segmentation results of Deeplab from their original publication, whilst we reproduced the results\nof FCN-8s and CRF-as-RNN from their publicly-available source code. Best viewed in\ncolour."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e56-6db4-11ee-a37b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-0", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "Deep Speech 2: End-to-End Speech Recognition in\nEnglish and Mandarin\nBaidu Research \u2013 Silicon Valley AI Lab\u0003\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro,\nJingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel,\nLinxi Fan, Christopher Fougner, Tony Han, Awni Hannun, Billy Jun, Patrick LeGresley,\nLibby Lin, Sharan Narang, Andrew Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman,\nSanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang,\nBo Xiao, Dani Yogatama, Jun Zhan, Zhenyao Zhu\nAbstract\nWe show that an end-to-end deep learning approach can be used to recognize\neither English or Mandarin Chinese speech\u2014two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages. Key to our approach is\nour application of HPC techniques, resulting in a 7x speedup over our previous"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e57-6db4-11ee-bf19-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-1", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "our application of HPC techniques, resulting in a 7x speedup over our previous\nsystem [26]. Because of this ef\ufb01ciency, experiments that previously took weeks\nnow run in days. This enables us to iterate more quickly to identify superior architectures and algorithms. As a result, in several cases, our system is competitive\nwith the transcription of human workers when benchmarked on standard datasets.\nFinally, using a technique called Batch Dispatch with GPUs in the data center, we\nshow that our system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.\n1 Introduction\nDecades worth of hand-engineered domain knowledge has gone into current state-of-the-art automatic speech recognition (ASR) pipelines. A simple but powerful alternative solution is to train such\nASR models end-to-end, using deep learning to replace most modules with a single model [26]. We\npresent the second generation of our speech system that exempli\ufb01es the major advantages of endto-end learning. The Deep Speech 2 ASR pipeline approaches or exceeds the accuracy of Amazon\nMechanical Turk human workers on several benchmarks, works in multiple languages with little\nmodi\ufb01cation, and is deployable in a production setting. It thus represents a signi\ufb01cant step towards"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e58-6db4-11ee-a3ed-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-2", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "modi\ufb01cation, and is deployable in a production setting. It thus represents a signi\ufb01cant step towards\na single ASR system that addresses the entire range of speech recognition contexts handled by humans. Since our system is built on end-to-end deep learning, we can employ a spectrum of deep\nlearning techniques: capturing large training sets, training larger models with high performance\ncomputing, and methodically exploring the space of neural network architectures. We show that\nthrough these techniques we are able to reduce error rates of our previous end-to-end system [26] in\nEnglish by up to 43%, and can also recognize Mandarin speech with high accuracy.\nOne of the challenges of speech recognition is the wide range of variability in speech and acoustics.\nAs a result, modern ASR pipelines are made up of numerous components including complex feature\nextraction, acoustic models, language and pronunciation models, speaker adaptation, etc. Building and tuning these individual components makes developing a new speech recognizer very hard,\nespecially for a new language. Indeed, many parts do not generalize well across environments or\nlanguages and it is often necessary to support multiple application-speci\ufb01c systems in order to provide acceptable accuracy. This state of affairs is different from human speech recognition: people\n\u0003Authorship order is alphabetical."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e59-6db4-11ee-a5e7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-3", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "\u0003Authorship order is alphabetical.\n1arXiv:1512.02595v1  [cs.CL]  8 Dec 2015\nhave the innate ability to learn any language during childhood, using general skills to learn language.\nAfter learning to read and write, most humans can transcribe speech with robustness to variation in\nenvironment, speaker accent and noise, without additional training for the transcription task. To\nmeet the expectations of speech recognition users, we believe that a single engine must learn to be\nsimilarly competent; able to handle most applications with only minor modi\ufb01cations and able to\nlearn new languages from scratch without dramatic changes. Our end-to-end system puts this goal\nwithin reach, allowing us to approach or exceed the performance of human workers on several tests\nin two very different languages: Mandarin and English.\nSince Deep Speech 2 (DS2) is an end-to-end deep learning system, we can achieve performance\ngains by focusing on three crucial components: the model architecture, large labeled training\ndatasets, and computational scale. This approach has also yielded great advances in other application areas such as computer vision and natural language. This paper details our contribution to\nthese three areas for speech recognition, including an extensive investigation of model architectures"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e5a-6db4-11ee-b239-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-4", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "these three areas for speech recognition, including an extensive investigation of model architectures\nand the effect of data and model size on recognition performance. In particular, we describe numerous experiments with neural networks trained with the Connectionist Temporal Classi\ufb01cation (CTC)\nloss function [22] to predict speech transcriptions from audio. We consider networks composed of\nmany layers of recurrent connections, convolutional \ufb01lters, and nonlinearities, as well as the impact\nof a speci\ufb01c instance of Batch Normalization [63] (BatchNorm) applied to RNNs. We not only\n\ufb01nd networks that produce much better predictions than those in previous work [26], but also \ufb01nd\ninstances of recurrent models that can be deployed in a production setting with no signi\ufb01cant loss in\naccuracy.\nBeyond the search for better model architecture, deep learning systems bene\ufb01t greatly from large\nquantities of training data. We detail our data capturing pipeline that has enabled us to create larger\ndatasets than what is typically used to train speech recognition systems. Our English speech system\nis trained on 11,940 hours of speech, while the Mandarin system is trained on 9,400 hours. We use\ndata synthesis to further augment the data during training.\nTraining on large quantities of data usually requires the use of larger models. Indeed, our models"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e5b-6db4-11ee-982b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-5", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "data synthesis to further augment the data during training.\nTraining on large quantities of data usually requires the use of larger models. Indeed, our models\nhave many more parameters than those used in our previous system. Training a single model at\nthese scales requires tens of exaFLOPs1that would require 3-6 weeks to execute on a single GPU.\nThis makes model exploration a very time consuming exercise, so we have built a highly optimized\ntraining system that uses 8 or 16 GPUs to train one model. In contrast to previous large-scale training\napproaches that use parameter servers and asynchronous updates [18, 10], we use synchronous SGD,\nwhich is easier to debug while testing new ideas, and also converges faster for the same degree of\ndata parallelism. To make the entire system ef\ufb01cient, we describe optimizations for a single GPU\nas well as improvements to scalability for multiple GPUs. We employ optimization techniques\ntypically found in High Performance Computing to improve scalability. These optimizations include\na fast implementation of the CTC loss function on the GPU, and a custom memory allocator. We\nalso use carefully integrated compute nodes and a custom implementation of all-reduce to accelerate\ninter-GPU communication. Overall the system sustains approximately 50 teraFLOP/second when\ntraining on 16 GPUs. This amounts to 3 teraFLOP/second per GPU which is about 50% of peak"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e5c-6db4-11ee-b344-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-6", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "training on 16 GPUs. This amounts to 3 teraFLOP/second per GPU which is about 50% of peak\ntheoretical performance. This scalability and ef\ufb01ciency cuts training times down to 3 to 5 days,\nallowing us to iterate more quickly on our models and datasets.\nWe benchmark our system on several publicly available test sets and compare the results to our\nprevious end-to-end system [26]. Our goal is to eventually reach human-level performance not only\non speci\ufb01c benchmarks, where it is possible to improve through dataset-speci\ufb01c tuning, but on a\nrange of benchmarks that re\ufb02ects a diverse set of scenarios. To that end, we have also measured\nthe performance of human workers on each benchmark for comparison. We \ufb01nd that our system\noutperforms humans in some commonly-studied benchmarks and has signi\ufb01cantly closed the gap in\nmuch harder cases. In addition to public benchmarks, we show the performance of our Mandarin\nsystem on internal datasets that re\ufb02ect real-world product scenarios.\nDeep learning systems can be challenging to deploy at scale. Large neural networks are computationally expensive to evaluate for each user utterance, and some network architectures are more"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e5d-6db4-11ee-ac2f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-7", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "Deep learning systems can be challenging to deploy at scale. Large neural networks are computationally expensive to evaluate for each user utterance, and some network architectures are more\neasily deployed than others. Through model exploration, we \ufb01nd high-accuracy, deployable network architectures, which we detail here. We also employ a batching scheme suitable for GPU\n11 exaFLOP = 1018FLoating-point OPerations.\n2\nhardware called Batch Dispatch that leads to an ef\ufb01cient, real-time implementation of our Mandarin\nengine on production servers. Our implementation achieves a 98th percentile compute latency of 67\nmilliseconds, while the server is loaded with 10 simultaneous audio streams.\nThe remainder of the paper is as follows. We begin with a review of related work in deep learning,\nend-to-end speech recognition, and scalability in Section 2. Section 3 describes the architectural and\nalgorithmic improvements to the model and Section 4 explains how to ef\ufb01ciently compute them. We\ndiscuss the training data and steps taken to further augment the training set in Section 5. An analysis\nof results for the DS2 system in English and Mandarin is presented in Section 6. We end with a\ndescription of the steps needed to deploy DS2 to real users in Section 7.\n2 Related Work"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e5e-6db4-11ee-8de4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-8", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "description of the steps needed to deploy DS2 to real users in Section 7.\n2 Related Work\nThis work is inspired by previous work in both deep learning and speech recognition. Feed-forward\nneural network acoustic models were explored more than 20 years ago [7, 50, 19]. Recurrent neural networks and networks with convolution were also used in speech recognition around the same\ntime [51, 67]. More recently DNNs have become a \ufb01xture in the ASR pipeline with almost all\nstate of the art speech work containing some form of deep neural network [42, 29, 17, 16, 43, 58].\nConvolutional networks have also been found bene\ufb01cial for acoustic models [1, 53]. Recurrent\nneural networks, typically LSTMs, are just beginning to be deployed in state-of-the art recognizers [24, 25, 55] and work well together with convolutional layers for the feature extraction [52].\nModels with both bidirectional [24] and unidirectional recurrence have been explored as well.\nEnd-to-end speech recognition is an active area of research, showing compelling results when used\nto re-score the outputs of a DNN-HMM [23] and standalone [26]. Two methods are currently used to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e5f-6db4-11ee-a1a0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-9", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "to re-score the outputs of a DNN-HMM [23] and standalone [26]. Two methods are currently used to\nmap variable length audio sequences directly to variable length transcriptions. The RNN encoderdecoder paradigm uses an encoder RNN to map the input to a \ufb01xed length vector and a decoder\nnetwork to expand the \ufb01xed length vector into a sequence of output predictions [11, 62]. Adding an\nattentional mechanism to the decoder greatly improves performance of the system, particularly with\nlong inputs or outputs [2]. In speech, the RNN encoder-decoder with attention performs well both\nin predicting phonemes [12] or graphemes [3, 8].\nThe other commonly used technique for mapping variable length audio input to variable length\noutput is the CTC loss function [22] coupled with an RNN to model temporal information. The CTCRNN model performs well in end-to-end speech recognition with grapheme outputs [23, 27, 26, 40].\nThe CTC-RNN model has also been shown to work well in predicting phonemes [41, 54], though\na lexicon is still needed in this case. Furthermore it has been necessary to pre-train the CTC-RNN\nnetwork with a DNN cross-entropy network that is fed frame-wise alignments from a GMM-HMM"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e60-6db4-11ee-aca5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-10", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "network with a DNN cross-entropy network that is fed frame-wise alignments from a GMM-HMM\nsystem [54]. In contrast, we train the CTC-RNN networks from scratch without the need of framewise alignments for pre-training.\nExploiting scale in deep learning has been central to the success of the \ufb01eld thus far [36, 38]. Training on a single GPU resulted in substantial performance gains [49], which were subsequently scaled\nlinearly to two [36] or more GPUs [15]. We take advantage of work in increasing individual GPU\nef\ufb01ciency for low-level deep learning primitives [9]. We build on the past work in using modelparallelism [15], data-parallelism [18] or a combination of the two [64, 26] to create a fast and\nhighly scalable system for training deep RNNs in speech recognition.\nData has also been central to the success of end-to-end speech recognition, with over 7000 hours\nof labeled speech used in Deep Speech 1 (DS1) [26]. Data augmentation has been highly effective\nin improving the performance of deep learning in computer vision [39, 56, 14]. This has also been\nshown to improve speech systems [21, 26]. Techniques used for data augmentation in speech range"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e61-6db4-11ee-9f6a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-11", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "shown to improve speech systems [21, 26]. Techniques used for data augmentation in speech range\nfrom simple noise addition [26] to complex perturbations such as simulating changes to the vocal\ntract length and rate of speech of the speaker [31, 35].\nExisting speech systems can also be used to bootstrap new data collection. In one approach, the\nauthors use one speech engine to align and \ufb01lter a thousand hours of read speech [46]. In another\napproach, a heavy-weight of\ufb02ine speech recognizer is used to generate transcriptions for tens of\nthousands of hours of speech [33]. This is then passed through a \ufb01lter and used to re-train the recognizer, resulting in signi\ufb01cant performance gains. We draw inspiration from these past approaches in\n3\nbootstrapping larger datasets and data augmentation to increase the effective amount of labeled data\nfor our system.\n3 Model Architecture\nA simple multi-layer model with a single recurrent layer cannot exploit thousands of hours of labelled speech. In order to learn from datasets this large, we increase the model capacity via depth.\nWe explore architectures with up to 11 layers including many bidirectional recurrent layers and convolutional layers. These models have nearly 8 times the amount of computation per data example as\nthe models in Deep Speech 1 making fast optimization and computation critical. In order to optimize"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e62-6db4-11ee-beca-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-12", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "the models in Deep Speech 1 making fast optimization and computation critical. In order to optimize\nthese models successfully, we use Batch Normalization for RNNs and a novel optimization curriculum we call SortaGrad. We also exploit long strides between RNN inputs to reduce computation\nper example by a factor of 3. This is helpful for both training and evaluation, though requires some\nmodi\ufb01cations in order to work well with CTC. Finally, though many of our research results make\nuse of bidirectional recurrent layers, we \ufb01nd that excellent models exist using only unidirectional\nrecurrent layers\u2014a feature that makes such models much easier to deploy. Taken together these\nfeatures allow us to tractably optimize deep RNNs and improve performance by more than 40% in\nboth English and Mandarin error rates over the smaller baseline models.\n3.1 Preliminaries\nFigure 1 shows the architecture of the DS2 system which at its core is similar to the previous DS1\nsystem [26]: a recurrent neural network (RNN) trained to ingest speech spectrograms and generate\ntext transcriptions.\nLet a single utterance x(i)and label y(i)be sampled from a training set X ="}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e63-6db4-11ee-8d5e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-13", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "text transcriptions.\nLet a single utterance x(i)and label y(i)be sampled from a training set X =\nf(x(1),y(1)), (x(2),y(2)),:::g. Each utterance, x(i), is a time-series of length T(i)where every\ntime-slice is a vector of audio features, x(i)\nt,t= 0,:::,T(i)\u00001. We use a spectrogram of power\nnormalized audio clips as the features to the system, so x(i)\nt,pdenotes the power of the p\u2019th frequency\nbin in the audio frame at time t. The goal of the RNN is to convert an input sequence x(i)into a\n\ufb01nal transcription y(i). For notational convenience, we drop the superscripts and use xto denote a\nchosen utterance and ythe corresponding label.\nThe outputs of the network are the graphemes of each language. At each output time-step t, the RNN\nmakes a prediction over characters, p(`tjx), where`tis either a character in the alphabet or the blank\nsymbol. In English we have `t2fa, b, c,:::,z,space ,apostrophe ,blankg, where we have added"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e64-6db4-11ee-9e23-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-14", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "symbol. In English we have `t2fa, b, c,:::,z,space ,apostrophe ,blankg, where we have added\ntheapostrophe as well as a space symbol to denote word boundaries. For the Mandarin system the\nnetwork outputs simpli\ufb01ed Chinese characters. We describe this in more detail in Section 3.9.\nThe RNN model is composed of several layers of hidden units. The architectures we experiment\nwith consist of one or more convolutional layers, followed by one or more recurrent layers, followed\nby one or more fully connected layers.\nThe hidden representation at layer lis given byhlwith the convention that h0represents the input\nx. The bottom of the network is one or more convolutions over the time dimension of the input. For\na context window of size c, thei-th activation at time-step tof the convolutional layer is given by\nhl\nt,i=f(wl\ni\u000ehl\u00001\nt\u0000c:t+c) (1)\nwhere\u000edenotes the element-wise product between the i-th \ufb01lter and the context window of the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e65-6db4-11ee-8800-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-15", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "t\u0000c:t+c) (1)\nwhere\u000edenotes the element-wise product between the i-th \ufb01lter and the context window of the\nprevious layers activations, and fdenotes a unary nonlinear function. We use the clipped recti\ufb01edlinear (ReLU) function \u001b(x) = minfmaxfx, 0g, 20gas our nonlinearity. In some layers, usually\nthe \ufb01rst, we sub-sample by striding the convolution by sframes. The goal is to shorten the number\nof time-steps for the recurrent layers above.\nFollowing the convolutional layers are one or more bidirectional recurrent layers [57]. The forward\nin time\u0000 !hland backward in time \u0000hlrecurrent layer activations are computed as\n\u0000 !hl\nt=g(hl\u00001\nt,\u0000 !hl\nt\u00001)\n \u0000hl\nt=g(hl\u00001\nt, \u0000hl\nt+1)(2)\n4\nCTC\nSpectrogramRecurrentorGRU(Bidirectional)1D or 2DInvariantConvolutionFully Connected\nBatchNormalizationFigure 1: Architecture of the DS2 system used to train on both English and Mandarin speech. We explore"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e66-6db4-11ee-8510-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-16", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "BatchNormalizationFigure 1: Architecture of the DS2 system used to train on both English and Mandarin speech. We explore\nvariants of this architecture by varying the number of convolutional layers from 1 to 3 and the number of\nrecurrent or GRU layers from 1 to 7.\nThe two sets of activations are summed to form the output activations for the layer hl=\u0000 !hl+ \u0000hl.\nThe function g(\u0001)can be the standard recurrent operation\n\u0000 !hl\nt=f(Wlhl\u00001\nt+\u0000 !Ul\u0000 !hl\nt\u00001+bl) (3)\nwhereWlis the input-hidden weight matrix,\u0000 !Ulis the recurrent weight matrix and blis a bias term.\nIn this case the input-hidden weights are shared for both directions of the recurrence. The function\ng(\u0001)can also represent more complex recurrence operations such as the Long Short-Term Memory\n(LSTM) units [30] and the gated recurrent units (GRU) [11].\nAfter the bidirectional recurrent layers we apply one or more fully connected layers with\nhl\nt=f(Wlhl\u00001\nt+bl) (4)\nThe output layer Lis a softmax computing a probability distribution over characters given by\np(`t=kjx) =exp(wL\nk\u0001hL\u00001"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e67-6db4-11ee-a573-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-17", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "The output layer Lis a softmax computing a probability distribution over characters given by\np(`t=kjx) =exp(wL\nk\u0001hL\u00001\nt)P\njexp(wL\nj\u0001hL\u00001\nt)(5)\nThe model is trained using the CTC loss function [22]. Given an input-output pair (x,y)and the\ncurrent parameters of the network \u0012, we compute the loss function L(x,y;\u0012)and its derivative with\nrespect to the parameters of the network r\u0012L(x,y;\u0012). This derivative is then used to update the\nnetwork parameters through the backpropagation through time algorithm.\nIn the following subsections we describe the architectural and algorithmic improvements made relative to DS1 [26]. Unless otherwise stated these improvements are language agnostic. We report\nresults on an English speaker held out development set which is an internal dataset containing 2048\nutterances of primarily read speech. All models are trained on datasets described in Section 5.\nWe report Word Error Rate (WER) for the English system and Character Error Rate (CER) for the\nMandarin system. In both cases we integrate a language model in a beam search decoding step as\ndescribed in Section 3.8.\n5\nArchitecture Hidden Units Train Dev\nBaseline BatchNorm Baseline BatchNorm"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e68-6db4-11ee-9063-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-18", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "described in Section 3.8.\n5\nArchitecture Hidden Units Train Dev\nBaseline BatchNorm Baseline BatchNorm\n1 RNN, 5 total 2400 10.55 11.99 13.55 14.40\n3 RNN, 5 total 1880 9.55 8.29 11.61 10.56\n5 RNN, 7 total 1510 8.59 7.61 10.77 9.78\n7 RNN, 9 total 1280 8.76 7.68 10.83 9.52\nTable 1: Comparison of WER on a training and development set for various depths of RNN, with and without\nBatchNorm. The number of parameters is kept constant as the depth increases, thus the number of hidden units\nper layer decreases. All networks have 38 million parameters. The architecture \u201cM RNN, N total\u201d implies 1\nlayer of 1D convolution at the input, M consecutive bidirectional RNN layers, and the rest as fully-connected\nlayers with N total layers in the network.\n3.2 Batch Normalization for Deep RNNs\nTo ef\ufb01ciently scale our model as we scale the training set, we increase the depth of the networks by\nadding more hidden layers, rather than making each layer larger. Previous work has examined doing\nso by increasing the number of consecutive bidirectional recurrent layers [24]. We explore Batch"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e69-6db4-11ee-b783-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-19", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "adding more hidden layers, rather than making each layer larger. Previous work has examined doing\nso by increasing the number of consecutive bidirectional recurrent layers [24]. We explore Batch\nNormalization (BatchNorm) as a technique to accelerate training for such networks [63] since they\noften suffer from optimization issues.\nRecent research has shown that BatchNorm improves the speed of convergence of recurrent nets,\nwithout showing any improvement in generalization performance [37]. In contrast, we demonstrate\nthat when applied to very deep networks of simple RNNs on large data sets, batch normalization\nsubstantially improves \ufb01nal generalization error while greatly accelerating training.\nIn a typical feed-forward layer containing an af\ufb01ne transformation followed by a non-linearity f(\u0001),\nwe insert a BatchNorm transformation by applying f(B(Wh))instead off(Wh+b), where\nB(x) =\rx\u0000E[x]\n(Var[x] +\u000f)1=2+\f. (6)\nThe terms EandVar are the empirical mean and variance over a minibatch. The bias bof the\nlayer is dropped since its effect is cancelled by mean removal. The learnable parameters \rand\f\nallow the layer to scale and shift each hidden unit as desired. The constant \u000fis small and positive,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e6a-6db4-11ee-bc95-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-20", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "allow the layer to scale and shift each hidden unit as desired. The constant \u000fis small and positive,\nand is included only for numerical stability. In our convolutional layers the mean and variance\nare estimated over all the temporal output units for a given convolutional \ufb01lter on a minibatch.\nThe BatchNorm transformation reduces internal covariate shift by insulating a given layer from\npotentially uninteresting changes in the mean and variance of the layer\u2019s input.\nWe consider two methods of extending BatchNorm to bidirectional RNNs [37]. A natural extension\nis to insert a BatchNorm transformation immediately before every non-linearity. Equation 3 then\nbecomes\u0000 !hl\nt=f(B(Wlhl\u00001\nt+\u0000 !Ul\u0000 !hl\nt\u00001)). (7)\nIn this case the mean and variance statistics are accumulated over a single time-step of the minibatch.\nThe sequential dependence between time-steps prevents averaging over all time-steps. We \ufb01nd that\nthis technique does not lead to improvements in optimization. We also tried accumulating an average\nover successive time-steps, so later time-steps are normalized over all present and previous timesteps. This also proved ineffective and greatly complicated backpropagation."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e6b-6db4-11ee-bb88-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-21", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "over successive time-steps, so later time-steps are normalized over all present and previous timesteps. This also proved ineffective and greatly complicated backpropagation.\nWe \ufb01nd that sequence-wise normalization [37] overcomes these issues. The recurrent computation\nis given by\u0000 !hl\nt=f(B(Wlhl\u00001\nt) +\u0000 !Ul\u0000 !hl\nt\u00001). (8)\nFor each hidden unit, we compute the mean and variance statistics over all items in the minibatch\nover the length of the sequence. Figure 2 shows that deep networks converge faster with sequencewise normalization. Table 1 shows that the performance improvement from sequence-wise normalization increases with the depth of the network, with a 12% performance difference for the deepest\nnetwork. When comparing depth, in order to control for model size we hold constant the total\n6\n50 100 150 200 250 300\nIteration ( \u21e5103)2030405060Cost5-1 BN\n5-1 No BN\n9-7 BN\n9-7 No BNFigure 2: Training curves of two models trained with and without BatchNorm. We start the plot after the \ufb01rst\nepoch of training as the curve is more dif\ufb01cult to interpret due to the SortaGrad curriculum method mentioned\nin Section 3.3\nTrain Dev"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e6c-6db4-11ee-8835-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-22", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "epoch of training as the curve is more dif\ufb01cult to interpret due to the SortaGrad curriculum method mentioned\nin Section 3.3\nTrain Dev\nBaseline BatchNorm Baseline BatchNorm\nNot Sorted 10.71 8.04 11.96 9.78\nSorted 8.76 7.68 10.83 9.52\nTable 2: Comparison of WER on a training and development set with and without SortaGrad, and with and\nwithout batch normalization.\nnumber of parameters and still see strong performance gains. We would expect to see even larger\nimprovements from depth if we held constant the number of activations per layer and added layers. We also \ufb01nd that BatchNorm harms generalization error for the shallowest network just as it\nconverges slower for shallower networks.\nThe BatchNorm approach works well in training, but is dif\ufb01cult to implement for a deployed ASR\nsystem, since it is often necessary to evaluate a single utterance in deployment rather than a batch.\nWe \ufb01nd that normalizing each neuron to its mean and variance over just the sequence degrades\nperformance. Instead, we store a running average of the mean and variance for the neuron collected\nduring training, and use these for evaluation in deployment [63]. Using this technique, we can"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e6d-6db4-11ee-aba0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-23", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "performance. Instead, we store a running average of the mean and variance for the neuron collected\nduring training, and use these for evaluation in deployment [63]. Using this technique, we can\nevaluate a single utterance at a time with better results than evaluating with a large batch.\n3.3 SortaGrad\nTraining on examples of varying length pose some algorithmic challenges. One possible solution is\ntruncating backpropagation through time [68], so that all examples have the same sequence length\nduring training [52]. However, this can inhibit the ability to learn longer term dependencies. Other\nworks have found that presenting examples in order of dif\ufb01culty can accelerate online learning [6,\n70]. A common theme in many sequence learning problems including machine translation and\nspeech recognition is that longer examples tend to be more challenging [11].\nThe CTC cost function that we use implicitly depends on the length of the utterance,\nL(x,y;\u0012) =\u0000logX\n`2Align (x,y)TY\ntpctc(`tjx;\u0012). (9)\nwhere Align (x,y)is the set of all possible alignments of the characters of the transcription yto\nframes of input xunder the CTC operator. In equation 9, the inner term is a product over time-steps"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e6e-6db4-11ee-a150-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-24", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "frames of input xunder the CTC operator. In equation 9, the inner term is a product over time-steps\nof the sequence, which shrinks with the length of the sequence since pctc(`tjx;\u0012)<1. This motivates a curriculum learning strategy we title SortaGrad. SortaGrad uses the length of the utterance\nas a heuristic for dif\ufb01culty, since long utterances have higher cost than short utterances.\n7\nArchitecture Simple RNN GRU\n5 layers, 1 Recurrent 14.40 10.53\n5 layers, 3 Recurrent 10.56 8.00\n7 layers, 5 Recurrent 9.78 7.79\n9 layers, 7 Recurrent 9.52 8.19\nTable 3: Comparison of development set WER for networks with either simple RNN or GRU, for various\ndepths. All models have batch normalization, one layer of 1D-invariant convolution, and approximately 38\nmillion parameters.\nIn the \ufb01rst training epoch, we iterate through the training set in increasing order of the length of\nthe longest utterance in the minibatch. After the \ufb01rst epoch, training reverts back to a random order\nover minibatches. Table 2 shows a comparison of training cost with and without SortaGrad on the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e6f-6db4-11ee-b0a9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-25", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "over minibatches. Table 2 shows a comparison of training cost with and without SortaGrad on the\n9 layer model with 7 recurrent layers. This effect is particularly pronounced for networks without\nBatchNorm, since they are numerically less stable. In some sense the two techniques substitute for\none another, though we still \ufb01nd gains when applying SortaGrad and BatchNorm together. Even\nwith BatchNorm we \ufb01nd that this curriculum improves numerical stability and sensitivity to small\nchanges in training. Numerical instability can arise from different transcendental function implementations in the CPU and the GPU, especially when computing the CTC cost. This curriculum\ngives comparable results for both implementations.\nWe suspect that these bene\ufb01ts occur primarily because long utterances tend to have larger gradients,\nyet we use a \ufb01xed learning rate independent of utterance length. Furthermore, longer utterances are\nmore likely to cause the internal state of the RNNs to explode at an early stage in training.\n3.4 Comparison of simple RNNs and GRUs\nThe models we have shown so far are simple RNNs that have bidirectional recurrent layers with the\nrecurrence for both the forward in time and backward in time directions modeled by Equation 3."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e70-6db4-11ee-b862-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-26", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "recurrence for both the forward in time and backward in time directions modeled by Equation 3.\nCurrent research in speech and language processing has shown that having a more complex recurrence can allow the network to remember state over more time-steps while making them more\ncomputationally expensive to train [52, 8, 62, 2]. Two commonly used recurrent architectures are the\nLong Short-Term Memory (LSTM) units [30] and the Gated Recurrent Units (GRU) [11], though\nmany other variations exist. A recent comprehensive study of thousands of variations of LSTM and\nGRU architectures showed that a GRU is comparable to an LSTM with a properly initialized forget\ngate bias, and their best variants are competitive with each other [32]. We decided to examine GRUs\nbecause experiments on smaller data sets showed the GRU and LSTM reach similar accuracy for\nthe same number of parameters, but the GRUs were faster to train and less likely to diverge.\nThe GRUs we use are computed by\nzt=\u001b(Wzxt+Uzht\u00001+bz)\nrt=\u001b(Wrxt+Urht\u00001+br)\n~ht=f(Whxt+rt\u000eUhht\u00001+bh)\nht= (1\u0000zt)ht\u00001+zt~ht(10)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e71-6db4-11ee-a15f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-27", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "~ht=f(Whxt+rt\u000eUhht\u00001+bh)\nht= (1\u0000zt)ht\u00001+zt~ht(10)\nwhere\u001b(\u0001)is the sigmoid function, zandrrepresent the update andreset gates respectively, and\nwe drop the layer superscripts for simplicity. We differ slightly from the standard GRU in that we\nmultiply the hidden state ht\u00001byUhprior to scaling by the reset gate. This allows for all operations\nonht\u00001to be computed in a single matrix multiplication. The output nonlinearity f(\u0001)is typically\nthe hyperbolic tangent function tanh . However, we \ufb01nd similar performance for tanh and clippedReLU nonlinearities and choose to use the clipped-ReLU for simplicity and uniformity with the rest\nof the network.\nBoth GRU and simple RNN architectures bene\ufb01t from batch normalization and show strong results with deep networks. However, Table 3 shows that for a \ufb01xed number of parameters, the GRU\narchitectures achieve better WER for all network depths. This is clear evidence of the long term\ndependencies inherent in the speech recognition task present both within individual words and be8\nArchitecture Channels Filter dimension Stride Regular Dev Noisy Dev"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e72-6db4-11ee-8731-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-28", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "dependencies inherent in the speech recognition task present both within individual words and be8\nArchitecture Channels Filter dimension Stride Regular Dev Noisy Dev\n1-layer 1D 1280 11 2 9.52 19.36\n2-layer 1D 640, 640 5, 5 1, 2 9.67 19.21\n3-layer 1D 512, 512, 512 5, 5, 5 1, 1, 2 9.20 20.22\n1-layer 2D 32 41x11 2x2 8.94 16.22\n2-layer 2D 32, 32 41x11, 21x11 2x2, 2x1 9.06 15.71\n3-layer 2D 32, 32, 96 41x11, 21x11, 21x11 2x2, 2x1, 2x1 8.61 14.74\nTable 4: Comparison of WER for various arrangements of convolutional layers. In all cases, the convolutions\nare followed by 7 recurrent layers and 1 fully connected layer. For 2D-invariant convolutions the \ufb01rst dimension is frequency and the second dimension is time. All models have BatchNorm, SortaGrad, and 35 million\nparameters.\ntween words. As we discuss in Section 3.8, even simple RNNs are able to implicitly learn a language"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e73-6db4-11ee-a493-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-29", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "parameters.\ntween words. As we discuss in Section 3.8, even simple RNNs are able to implicitly learn a language\nmodel due to the large amount of training data. Interestingly, the GRU networks with 5 or more recurrent layers do not signi\ufb01cantly improve performance. We attribute this to the thinning from 1728\nhidden units per layer for 1 recurrent layer to 768 hidden units per layer for 7 recurrent layers, to\nkeep the total number of parameters constant.\nThe GRU networks outperform the simple RNNs in Table 3. However, in later results (Section 6) we\n\ufb01nd that as we scale up the model size, for a \ufb01xed computational budget the simple RNN networks\nperform slightly better. Given this, most of the remaining experiments use the simple RNN layers\nrather than the GRUs.\n3.5 Frequency Convolutions\nTemporal convolution is commonly used in speech recognition to ef\ufb01ciently model temporal translation invariance for variable length utterances. This type of convolution was \ufb01rst proposed for\nneural networks in speech more than 25 years ago [67]. Many neural network speech models have a\n\ufb01rst layer that processes input frames with some context window [16, 66]. This can be viewed as a\ntemporal convolution with a stride of one."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e74-6db4-11ee-8abf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-30", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "\ufb01rst layer that processes input frames with some context window [16, 66]. This can be viewed as a\ntemporal convolution with a stride of one.\nAdditionally, sub-sampling is essential to make recurrent neural networks computationally tractable\nwith high sample-rate audio. The DS1 system accomplished this through the use of a spectrogram\nas input and temporal convolution in the \ufb01rst layer with a stride parameter to reduce the number of\ntime-steps [26].\nConvolutions in frequency and time domains, when applied to the spectral input features prior to\nany other processing, can slightly improve ASR performance [1, 53, 60]. Convolution in frequency\nattempts to model spectral variance due to speaker variability more concisely than what is possible with large fully connected networks. Since spectral ordering of features is removed by fullyconnected and recurrent layers, frequency convolutions work better as the \ufb01rst layers of the network.\nWe experiment with adding between one and three layers of convolution. These are both in the timeand-frequency domain (2D invariance) and in the time-only domain (1D invariance). In all cases we\nuse a same convolution, preserving the number of input features in both frequency and time. In\nsome cases, we specify a stride across either dimension which reduces the size of the output. We do"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e75-6db4-11ee-9927-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-31", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "use a same convolution, preserving the number of input features in both frequency and time. In\nsome cases, we specify a stride across either dimension which reduces the size of the output. We do\nnot explicitly control for the number of parameters, since convolutional layers add a small fraction\nof parameters to our networks. All networks shown in Table 4 have about 35 million parameters.\nWe report results on two datasets\u2014a development set of 2048 utterances (\u201cRegular Dev\u201d) and a\nmuch noisier dataset of 2048 utterances (\u201cNoisy Dev\u201d) randomly sampled from the CHiME 2015\ndevelopment datasets [4]. We \ufb01nd that multiple layers of 1D-invariant convolutions provides a very\nsmall bene\ufb01t. The 2D-invariant convolutions improve results substantially on noisy data, while\nproviding a small bene\ufb01t on clean data. The change from one layer of 1D-invariant convolution to\nthree layers of 2D-invariant convolution improves WER by 23.9% on the noisy development set.\n9\nDev no LM Dev LM\nStride Unigrams Bigrams Unigrams Bigrams\n2 14.93 14.56 9.52 9.66\n3 15.01 15.60 9.65 10.06\n4 18.86 14.84 11.92 9.93"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e76-6db4-11ee-815f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-32", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "3 15.01 15.60 9.65 10.06\n4 18.86 14.84 11.92 9.93\nTable 5: Comparison of WER with different amounts of striding for unigram and bigram outputs on a model\nwith 1 layer of 1D-invariant convolution, 7 recurrent layers, and 1 fully connected layer. All models have\nBatchNorm, SortaGrad, and 35 million parameters. The models are compared on a development set with and\nwithout the use of a 5-gram language model.\n3.6 Striding\nIn the convolutional layers, we apply a longer stride and wider context to speed up training as fewer\ntime-steps are required to model a given utterance. Downsampling the input sound (through FFT and\nconvolutional striding) reduces the number of time-steps and computation required in the following\nlayers, but at the expense of reduced performance.\nIn our Mandarin models, we employ striding in the straightforward way. However, in English,\nstriding can reduce accuracy simply because the output of our network requires at least one timestep per output character, and the number of characters in English speech per time-step is high\nenough to cause problems when striding2. To overcome this, we can enrich the English alphabet"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e77-6db4-11ee-adfa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-33", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "enough to cause problems when striding2. To overcome this, we can enrich the English alphabet\nwith symbols representing alternate labellings like whole words, syllables or non-overlapping ngrams. In practice, we use non-overlapping bi-graphemes or bigrams, since these are simple to\nconstruct, unlike syllables, and there are few of them compared to alternatives such as whole words.\nWe transform unigram labels into bigram labels through a simple isomorphism.\nNon-overlapping bigrams shorten the length of the output transcription and thus allow for a decrease\nin the length of the unrolled RNN. The sentence the cat sat with non-overlapping bigrams is segmented as [th,e,space ,ca,t,space ,sa,t]. Notice that for words with odd number of characters, the\nlast character becomes an unigram and space is treated as an unigram as well. This isomorphism\nensures that the same words are always composed of the same bigram and unigram tokens. The\noutput set of bigrams consists of all bigrams that occur in the training set.\nIn Table 5 we show results for both the bigram and unigram systems for various levels of striding,\nwith or without a language model. We observe that bigrams allow for larger strides without any"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e78-6db4-11ee-9b6f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-34", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "with or without a language model. We observe that bigrams allow for larger strides without any\nsacri\ufb01ce in in the word error rate. This allows us to reduce the number of time-steps of the unrolled\nRNN bene\ufb01ting both computation and memory usage.\n3.7 Row Convolution and Unidirectional Models\nBidirectional RNN models are challenging to deploy in an online, low-latency setting, because they\nare built to operate on an entire sample, and so it is not possible to perform the transcription process\nas the utterance streams from the user. We have found an unidirectional architecture that performs\nas well as our bidirectional models. This allows us to use unidirectional, forward-only RNN layers\nin our deployment system.\nTo accomplish this, we employ a special layer that we call row convolution, shown in Figure 3. The\nintuition behind this layer is that we only need a small portion of future information to make an\naccurate prediction at the current time-step. Suppose at time-step t, we use a future context of \u001c\nsteps. We now have a feature matrix ht:t+\u001c= [ht,ht+1, ...,ht+\u001c]of sized\u0002(\u001c+ 1) . We de\ufb01ne a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e79-6db4-11ee-8701-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-35", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "parameter matrix Wof the same size as ht:t+\u001c. The activations rtfor the new layer at time-step t\nare\n2Chinese characters are more similar to English syllables than English characters. This is re\ufb02ected in our\ntraining data, where there are on average 14.1 characters/s in English, while only 3.3 characters/s in Mandarin.\nConversely, the Shannon entropy per character as calculated from occurrence in the training set, is less in\nEnglish due to the smaller character set\u20144.9 bits/char compared to 12.6 bits/char in Mandarin. This implies\nthat spoken Mandarin has a lower temporal entropy density, \u001841 bits/s compared to \u001858 bits/s, and can thus\nmore easily be temporally compressed without losing character information.\n10\nRecurrent layerRow conv layerhtht+1ht+2ht+3rt+3rt+2rt+1rtFigure 3: Row convolution architecture with future context size of 2\nrt,i=\u001c+1X\nj=1Wi,jht+j\u00001,i, for 1\u0014i\u0014d. (11)\nSince the convolution-like operation in Eq. 11 is row oriented for both Wandht:t+\u001c, we call this\nlayer row convolution."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e7a-6db4-11ee-9e4d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-36", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "Since the convolution-like operation in Eq. 11 is row oriented for both Wandht:t+\u001c, we call this\nlayer row convolution.\nWe place the row convolution layer above all recurrent layers. This has two advantages. First, this\nallows us to stream all computation below the row convolution layer on a \ufb01ner granularity given little\nfuture context is needed. Second, this results in better CER compared to the best bidirectional model\nfor Mandarin. We conjecture that the recurrent layers have learned good feature representations,\nso the row convolution layer simply gathers the appropriate information to feed to the classi\ufb01er.\nResults for a unidirectional Mandarin speech system with row convolution and a comparison to a\nbidirectional model are given in Section 7 on deployment.\n3.8 Language Model\nWe train our RNN Models over millions of unique utterances, which enables the network to learn a\npowerful implicit language model. Our best models are quite adept at spelling, without any external\nlanguage constraints. Further, in our development datasets we \ufb01nd many cases where our models can\nimplicitly disambiguate homophones\u2014for example, \u201che expects the Japanese agent tosell it for two\nhundred seventy \ufb01ve thousand dollars\u201d. Nevertheless, the labeled training data is small compared"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e7b-6db4-11ee-a363-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-37", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "hundred seventy \ufb01ve thousand dollars\u201d. Nevertheless, the labeled training data is small compared\nto the size of unlabeled text corpora that are available. Thus we \ufb01nd that WER improves when we\nsupplement our system with a language model trained from external text.\nWe use an n-gram language model since they scale well to large amounts of unlabeled text [26].\nFor English, our language model is a Kneser-Ney smoothed 5-gram model with pruning that is\ntrained using the KenLM toolkit [28] on cleaned text from the Common Crawl Repository3. The\nvocabulary is the most frequently used 400,000 words from 250 million lines of text, which produces\na language model with about 850 million n-grams. For Mandarin, the language model is a KneserNey smoothed character level 5-gram model with pruning that is trained on an internal text corpus\nof 8 billion lines of text. This produces a language model with about 2 billion n-grams.\nDuring inference we search for the transcription ythat maximizes Q(y)shown in Equation 12. This\nis a linear combination of logprobabilities from the CTC trained network and language model, along\nwith a word insertion term [26]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e7c-6db4-11ee-8421-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-38", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "is a linear combination of logprobabilities from the CTC trained network and language model, along\nwith a word insertion term [26].\nQ(y) = log(pctc(yjx)) +\u000blog(plm(y)) +\fword_count (y) (12)\nThe weight\u000bcontrols the relative contributions of the language model and the CTC network. The\nweight\fencourages more words in the transcription. These parameters are tuned on a development\nset. We use a beam search to \ufb01nd the optimal transcription [27].\n3http://commoncrawl.org\n11\nLanguage Architecture Dev no LM Dev LM\nEnglish 5-layer, 1 RNN 27.79 14.39\nEnglish 9-layer, 7 RNN 14.93 9.52\nMandarin 5-layer, 1 RNN 9.80 7.13\nMandarin 9-layer, 7 RNN 7.55 5.81\nTable 6: Comparison of WER for English and CER for Mandarin with and without a language model. These\nare simple RNN models with only one layer of 1D invariant convolution.\nTable 6 shows that an external language model helps both English and Mandarin speech systems.\nThe relative improvement given by the language model drops from 48% to 36% in English and 27%"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e7d-6db4-11ee-9a98-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-39", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "Table 6 shows that an external language model helps both English and Mandarin speech systems.\nThe relative improvement given by the language model drops from 48% to 36% in English and 27%\nto 23% in Mandarin, as we go from a model with 5 layers and 1 recurrent layer to a model with 9\nlayers and 7 recurrent layers. We hypothesize that the network builds a stronger implicit language\nmodel with more recurrent layers.\nThe relative performance improvement from a language model is higher in English than in Mandarin.\nWe attribute this to the fact that a Chinese character represents a larger block of information than\nan English character. For example, if we output directly to syllables or words in English, the model\nwould make fewer spelling mistakes and the language model would likely help less.\n3.9 Adaptation to Mandarin\nThe techniques that we have described so far can be used to build an end-to-end Mandarin speech\nrecognition system that outputs Chinese characters directly. This precludes the need to construct a\npronunciation model, which is often a fairly involved component for porting speech systems to other\nlanguages [59]. Direct output to characters also precludes the need to explicitly model language\nspeci\ufb01c pronunciation features. For example we do not need to model Mandarin tones explicitly, as\nsome speech systems must do [59, 45].\nThe only architectural changes we make to our networks are due to the characteristics of the Chinese"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e7e-6db4-11ee-a1d1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-40", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "some speech systems must do [59, 45].\nThe only architectural changes we make to our networks are due to the characteristics of the Chinese\ncharacter set. Firstly, the output layer of the network outputs about 6000 characters, which includes\nthe Roman alphabet, since hybrid Chinese-English transcripts are common. We incur an out of\nvocabulary error at evaluation time if a character is not contained in this set. This is not a major\nconcern, as our test set has only 0.74% out of vocab characters.\nWe use a character level language model in Mandarin as words are not usually segmented in text.\nThe word insertion term of Equation 12 becomes a character insertion term. In addition, we \ufb01nd that\nthe performance of the beam search during decoding levels off at a smaller beam size. This allows\nus to use a beam size of 200 with a negligible degradation in CER. In Section 6.2, we show that\nour Mandarin speech models show roughly the same improvements to architectural changes as our\nEnglish speech models.\n4 System Optimizations\nOur networks have tens of millions of parameters, and the training algorithm takes tens of singleprecision exaFLOPs to converge. Since our ability to evaluate hypotheses about our data and models depends on the ability to train models quickly, we built a highly optimized training system."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e7f-6db4-11ee-bc89-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-41", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "This system has two main components\u2014a deep learning library written in C ++, along with a highperformance linear algebra library written in both CUDA and C ++. Our optimized software, running\non dense compute nodes with 8 Titan X GPUs per node, allows us to sustain 24 single-precision\nteraFLOP/second when training a single model on one node. This is 45% of the theoretical peak\ncomputational throughput of each node. We also can scale to multiple nodes, as outlined in the next\nsubsection.\n4.1 Scalability and Data-Parallelism\nWe use the standard technique of data-parallelism to train on multiple GPUs using synchronous\nSGD. Our most common con\ufb01guration uses a minibatch of 512on8GPUs. Our training pipeline\n12\nbinds one process to each GPU. These processes then exchange gradient matrices during the backpropagation by using all-reduce, which exchanges a matrix between multiple processes and sums\nthe result so that at the end, each process has a copy of the sum of all matrices from all processes.\nWe \ufb01nd synchronous SGD useful because it is reproducible and deterministic. We have found\nthat the appearance of non-determinism in our system often signals a serious bug, and so having\nreproducibility as a goal has greatly facilitates debugging. In contrast, asynchronous methods such"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1a8e80-6db4-11ee-86ee-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-42", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "that the appearance of non-determinism in our system often signals a serious bug, and so having\nreproducibility as a goal has greatly facilitates debugging. In contrast, asynchronous methods such\nas asynchronous SGD with parameter servers as found in Dean et al. [18] typically do not provide\nreproducibility and are therefore more dif\ufb01cult to debug. Synchronous SGD is simple to understand\nand implement. It scales well as we add multiple nodes to the training process.\n2021222324252627\nGPUs211212213214215216217218219Time (seconds)5-3 (2560)\n9-7 (1760)\nFigure 4: Scaling comparison of two networks\u2014a 5 layer model with 3 recurrent layers containing 2560\nhidden units in each layer and a 9 layer model with 7 recurrent layers containing 1760 hidden units in each\nlayer. The times shown are to train 1 epoch. The 5 layer model trains faster because it uses larger matrices and\nis more computationally ef\ufb01cient.\nFigure 4 shows that time taken to train one epoch halves as we double the number of GPUs that\nwe train on, thus achieving near-linear weak scaling. We keep the minibatch per GPU constant at\n64 during this experiment, effectively doubling the minibatch as we double the number of GPUs."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab49a-6db4-11ee-8688-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-43", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "64 during this experiment, effectively doubling the minibatch as we double the number of GPUs.\nAlthough we have the ability to scale to large minibatches, we typically use either 8 or 16 GPUs\nduring training with a minibatch of 512 or 1024, in order to converge to the best result.\nSince all-reduce is critical to the scalability of our training, we wrote our own implementation of\nthe ring algorithm [48, 65] for higher performance and better stability. Our implementation avoids\nextraneous copies between CPU and GPU, and is fundamental to our scalability. We con\ufb01gure\nOpenMPI with the smcuda transport that can send and receive buffers residing in the memory of\ntwo different GPUs by using GPUDirect. When two GPUs are in the same PCI root complex,\nthis avoids any unnecessary copies to CPU memory. This also takes advantage of tree-structured\ninterconnects by running multiple segments of the ring concurrently between neighboring devices.\nWe built our implementation using MPI send and receive, along with CUDA kernels for the elementwise operations.\nTable 7 compares the performance of our all-reduce implementation with that provided by OpenMPI\nversion 1.8.5. We report the time spent in all-reduce for a full training run that ran for one epoch\non our English dataset using a 5 layer, 3 recurrent layer architecture with 2560 hidden units for all"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab49b-6db4-11ee-b88d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-44", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "on our English dataset using a 5 layer, 3 recurrent layer architecture with 2560 hidden units for all\nlayers. In this table, we use a minibatch of 64 per GPU, expanding the algorithmic minibatch as we\nscale to more GPUs. We see that our implementation is considerably faster than OpenMPI\u2019s when\nthe communication is within a node (8 GPUs or less). As we increase the number of GPUs and\nincrease the amount of inter-node communication, the gap shrinks, although our implementation is\nstill 2-4X faster.\nAll of our training runs use either 8 or 16 GPUs, and in this regime, our all-reduce implementation\nresults in 2.5\u0002faster training for the full training run, compared to using OpenMPI directly. Optimizing all-reduce has thus resulted in important productivity bene\ufb01ts for our experiments, and has\nmade our simple synchronous SGD approach scalable.\n13\nGPU OpenMPI Our Performance\nall-reduce all-reduce Gain\n4 55359.1 2587.4 21.4\n8 48881.6 2470.9 19.8\n16 21562.6 1393.7 15.5\n32 8191.8 1339.6 6.1\n64 1395.2 611.0 2.3\n128 1602.1 422.6 3.8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab49c-6db4-11ee-bdcc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-45", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "32 8191.8 1339.6 6.1\n64 1395.2 611.0 2.3\n128 1602.1 422.6 3.8\nTable 7: Comparison of two different all-reduce implementations. All times are in seconds. Performance gain\nis the ratio of OpenMPI all-reduce time to our all-reduce time.\nLanguage Architecture CPU CTC Time GPU CTC Time Speedup\nEnglish 5-layer, 3 RNN 5888.12 203.56 28.9\nMandarin 5-layer, 3 RNN 1688.01 135.05 12.5\nTable 8: Comparison of time spent in seconds in computing the CTC loss function and gradient in one epoch\nfor two different implementations. Speedup is the ratio of CPU CTC time to GPU CTC time.\n4.2 GPU implementation of CTC loss function\nCalculating the CTC loss function is more complicated than performing forward and back propagation on our RNN architectures. Originally, we transferred activations from the GPUs to the\nCPU, where we calculated the loss function using an OpenMP parallelized implementation of CTC.\nHowever, this implementation limited our scalability rather signi\ufb01cantly, for two reasons. Firstly,\nit became computationally more signi\ufb01cant as we improved ef\ufb01ciency and scalability of the RNN"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab49d-6db4-11ee-9347-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-46", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "it became computationally more signi\ufb01cant as we improved ef\ufb01ciency and scalability of the RNN\nitself. Secondly, transferring large activation matrices between CPU and GPU required us to spend\ninterconnect bandwidth for CTC, rather than on transferring gradient matrices to allow us to scale\nusing data parallelism to more processors.\nTo overcome this, we wrote a GPU implementation of the CTC loss function. Our parallel implementation relies on a slight refactoring to simplify the dependences in the CTC calculation, as well\nas the use of optimized parallel sort implementations from ModernGPU [5]. We give more details\nof this parallelization in the Appendix.\nTable 8 compares the performance of two CTC implementations. The GPU implementation saves\nus 95 minutes per epoch in English, and 25 minutes in Mandarin. This reduces overall training time\nby 10-20%, which is also an important productivity bene\ufb01t for our experiments.\n4.3 Memory allocation\nOur system makes frequent use of dynamic memory allocations to GPU and CPU memory, mainly\nto store activation data for variable length utterances, and for intermediate results. Individual allocations can be very large; over 1 GB for the longest utterances. For these very large allocations\nwe found that CUDA\u2019s memory allocator and even std::malloc introduced signi\ufb01cant overhead"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab49e-6db4-11ee-bea9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-47", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "we found that CUDA\u2019s memory allocator and even std::malloc introduced signi\ufb01cant overhead\ninto our application\u2014over a 2x slowdown from using std::malloc in some cases. This is because\nboth cudaMalloc andstd::malloc forward very large allocations to the operating system or GPU\ndriver to update the system page tables. This is a good optimization for systems running multiple\napplications, all sharing memory resources, but editing page tables is pure overhead for our system\nwhere nodes are dedicated entirely to running a single model. To get around this limitation, we\nwrote our own memory allocator for both CPU and GPU allocations. Our implementation follows\nthe approach of the last level shared allocator in jemalloc: all allocations are carved out of contiguous memory blocks using the buddy algorithm [34]. To avoid fragmentation, we preallocate all of\nGPU memory at the start of training and subdivide individual allocations from this block. Similarly, we set the CPU memory block size that we forward to mmap to be substantially larger than\nstd::malloc , at 12GB.\n14\nDataset Speech Type Hours\nWSJ read 80\nSwitchboard conversational 300\nFisher conversational 2000\nLibriSpeech read 960\nBaidu read 5000\nBaidu mixed 3600\nTotal 11940"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab49f-6db4-11ee-81b2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-48", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "WSJ read 80\nSwitchboard conversational 300\nFisher conversational 2000\nLibriSpeech read 960\nBaidu read 5000\nBaidu mixed 3600\nTotal 11940\nTable 9: Summary of the datasets used to train DS2 in English. The Wall Street Journal (WSJ), Switchboard\nand Fisher [13] corpora are all published by the Linguistic Data Consortium. The LibriSpeech dataset [46] is\navailable free on-line. The other datasets are internal Baidu corpora.\nMost of the memory required for training deep recurrent networks is used to store activations through\neach layer for use by back propagation, not to store the parameters of the network. For example,\nstoring the weights for a 70M parameter network with 9 layers requires approximately 280 MB of\nmemory, but storing the activations for a batch of 64, seven-second utterances requires 1.5 GB of\nmemory. TitanX GPUs include 12GB of GDDR5 RAM, and sometimes very deep networks can\nexceed the GPU memory capacity when processing long utterances. This can happen unpredictably,\nespecially when the distribution of utterance lengths includes outliers, and it is desirable to avoid a\ncatastrophic failure when this occurs. When a requested memory allocation exceeds available GPU"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4a0-6db4-11ee-ae57-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-49", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "especially when the distribution of utterance lengths includes outliers, and it is desirable to avoid a\ncatastrophic failure when this occurs. When a requested memory allocation exceeds available GPU\nmemory, we allocate page-locked GPU-memory-mapped CPU memory using cudaMallocHost instead. This memory can be accessed directly by the GPU by forwarding individual memory transactions over PCIe at reduced bandwidth, and it allows a model to continue to make progress even\nafter encountering an outlier.\nThe combination of fast memory allocation with a fallback mechanism that allows us to slightly\nover\ufb02ow available GPU memory in exceptional cases makes the system signi\ufb01cantly simpler, more\nrobust, and more ef\ufb01cient.\n5 Training Data\nLarge-scale deep learning systems require an abundance of labeled training data. We have collected\nan extensive training dataset for both English and Mandarin speech models, in addition to augmenting our training with publicly available datasets. In English we use 11,940 hours of labeled speech\ndata containing 8 million utterances summarized in Table 9. For the Mandarin system we use 9,400\nhours of labeled audio containing 11 million utterances. The Mandarin speech data consists of internal Baidu corpora, representing a mix of read speech and spontaneous speech, in both standard\nMandarin and accented Mandarin.\n5.1 Dataset Construction"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4a1-6db4-11ee-8b9f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-50", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "Mandarin and accented Mandarin.\n5.1 Dataset Construction\nSome of the internal English (3,600 hours) and Mandarin (1,400 hours) datasets were created from\nraw data captured as long audio clips with noisy transcriptions. The length of these clips ranged from\nseveral minutes to more than hour, making it impractical to unroll them in time in the RNN during\ntraining. To solve this problem, we developed an alignment, segmentation and \ufb01ltering pipeline that\ncan generate a training set with shorter utterances and few erroneous transcriptions.\nThe \ufb01rst step in the pipeline is to use an existing bidirectional RNN model trained with CTC to\nalign the transcription to the frames of audio. For a given audio-transcript pair, (x,y), we \ufb01nd the\nalignment that maximizes\n`\u0003= arg max\n`2Align (x,y)TY\ntpctc(`tjx;\u0012). (13)\nThis is essentially a Viterbi alignment found using a RNN model trained with CTC. Since Equation 9\nintegrates over the alignment, the CTC loss function is never explicitly asked to produce an accurate\nalignment. In principle, CTC could choose to emit all the characters of the transcription after some\n15"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4a2-6db4-11ee-85bb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-51", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "alignment. In principle, CTC could choose to emit all the characters of the transcription after some\n15\n\ufb01xed delay and this can happen with unidirectional RNNs [54]. However, we found that CTC\nproduces an accurate alignment when trained with a bidirectional RNN.\nFollowing the alignment is a segmentation step that splices the audio and the corresponding aligned\ntranscription whenever it encounters a long series of consecutive blank labels occurs, since this\nusually denotes a stretch of silence. By tuning the number of consecutive blank s, we can tune the\nlength of the utterances generated. For the English speech data, we also require a space token to be\nwithin the stretch of blank s in order to segment only on word boundaries. We tune the segmentation\nto generate utterances that are on average 7 seconds long.\nThe \ufb01nal step in the pipeline removes erroneous examples that arise from a failed alignment. We\ncrowd source the ground truth transcriptions for several thousand examples. The word level edit\ndistance between the ground truth and the aligned transcription is used to produce a good orbad\nlabel. The threshold for the word level edit distance is chosen such that the resulting WER of the\ngood portion of the development set is less than 5%. We then train a linear classi\ufb01er to accurately"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4a3-6db4-11ee-a036-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-52", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "good portion of the development set is less than 5%. We then train a linear classi\ufb01er to accurately\npredict bad examples given the input features generated from the speech recognizer. We \ufb01nd the\nfollowing features useful: the raw CTC cost, the CTC cost normalized by the sequence length,\nthe CTC cost normalized by the transcript length, the ratio of the sequence length to the transcript\nlength, the number of words in the transcription and the number of characters in the transcription.\nFor the English dataset, we \ufb01nd that the \ufb01ltering pipeline reduces the WER from 17% to 5% while\nretaining more than 50% of the examples.\n5.2 Data Augmentation\nWe augment our training data by adding noise to increase the effective size of our training data\nand to improve our robustness to noisy speech [26]. Although the training data contains some\nintrinsic noise, we can increase the quantity and variety of noise through augmentation. Too much\nnoise augmentation tends to make optimization dif\ufb01cult and can lead to worse results, and too little\nnoise augmentation makes the system less robust to low signal-to-noise speech. We \ufb01nd that a good\nbalance is to add noise to 40% of the utterances that are chosen at random. The noise source consists"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4a4-6db4-11ee-a680-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-53", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "balance is to add noise to 40% of the utterances that are chosen at random. The noise source consists\nof several thousand hours of randomly selected audio clips combined to produce hundreds of hours\nof noise.\n5.3 Scaling Data\nOur English and Mandarin corpora are substantially larger than those commonly reported in speech\nrecognition literature. In Table 10, we show the effect of increasing the amount of labeled training\ndata on WER. This is done by randomly sampling the full dataset before training. For each dataset,\nthe model was trained for up to 20 epochs though usually early-stopped based on the error on a held\nout development set. We note that the WER decreases with a power law for both the regular and\nnoisy development sets. The WER decreases by \u001840% relative for each factor of 10 increase in\ntraining set size. We also observe a consistent gap in WER ( \u001860% relative) between the regular and\nnoisy datasets, implying that more data bene\ufb01ts both cases equally.\nThis implies that a speech system will continue to improve with more labeled training data. We\nhypothesize that equally as important as increasing raw number of hours is increasing the number\nof speech contexts that are captured in the dataset. A context can be any property that makes speech\nunique including different speakers, background noise, environment, and microphone hardware."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4a5-6db4-11ee-ac84-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-54", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "of speech contexts that are captured in the dataset. A context can be any property that makes speech\nunique including different speakers, background noise, environment, and microphone hardware.\nWhile we do not have the labels needed to validate this claim, we suspect that measuring WER as\na function of speakers in the dataset would lead to much larger relative gains than simple random\nsampling.\n6 Results\nTo better assess the real-world applicability of our speech system, we evaluate on a wide range of\ntest sets. We use several publicly available benchmarks and several test sets collected internally.\nTogether these test sets represent a wide range of challenging speech environments including low\nsignal-to-noise ratios (noisy and far-\ufb01eld), accented, read, spontaneous and conversational speech.\n16\nFraction of Data Hours Regular Dev Noisy Dev\n1% 120 29.23 50.97\n10% 1200 13.80 22.99\n20% 2400 11.65 20.41\n50% 6000 9.51 15.90\n100% 12000 8.46 13.59\nTable 10: Comparison of English WER for Regular and Noisy development sets on increasing training dataset\nsize. The architecture is a 9-layer model with 2 layers of 2D-invariant convolution and 7 recurrent layers with\n68M parameters."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4a6-6db4-11ee-8f4b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-55", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "size. The architecture is a 9-layer model with 2 layers of 2D-invariant convolution and 7 recurrent layers with\n68M parameters.\nAll models are trained for 20 epochs on either the full English dataset, described in Table 9, or\nthe full Mandarin dataset described in Section 5. We use stochastic gradient descent with Nesterov\nmomentum [61] along with a minibatch of 512 utterances. If the norm of the gradient exceeds\na threshold of 400, it is rescaled to 400 [47]. The model which performs the best on a held-out\ndevelopment set during training is chosen for evaluation. The learning rate is chosen from [1\u0002\n10\u00004, 6\u000210\u00004]to yield fastest convergence and annealed by a constant factor of 1.2 after each\nepoch. We use a momentum of 0.99 for all models.\nThe language models used are those described in Section 3.8. The decoding parameters from Equation 12 are tuned on a held-out development set. We use a beam size of 500 for the English decoder\nand a beam size of 200 for the Mandarin decoder.\n6.1 English\nThe best DS2 model has 11 layers with 3 layers of 2D convolution, 7 bidirectional recurrent layers,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4a7-6db4-11ee-ad7f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-56", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "6.1 English\nThe best DS2 model has 11 layers with 3 layers of 2D convolution, 7 bidirectional recurrent layers,\na fully-connected output layer along with Batch Normalization. The \ufb01rst layer outputs to bigrams\nwith a temporal stride of 3. By comparison the DS1 model has 5 layers with a single bidirectional\nrecurrent layer and it outputs to unigrams with a temporal stride of 2 in the \ufb01rst layer. We report\nresults on several test sets for both the DS2 and DS1 model. We do not tune or adapt either model\nto any of the speech conditions in the test sets. Language model decoding parameters are set once\non a held-out development set.\nTo put the performance of our system in context, we benchmark most of our results against human\nworkers, since speech recognition is an audio perception and language understanding problem that\nhumans excel at. We obtain a measure of human level performance by paying workers from Amazon\nMechanical Turk to hand-transcribe all of our test sets. Two workers transcribe the same audio clip,\nthat is typically about 5 seconds long, and we use the better of the two transcriptions for the \ufb01nal\nWER calculation. They are free to listen to the audio clip as many times as they like. These workers"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4a8-6db4-11ee-9c41-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-57", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "WER calculation. They are free to listen to the audio clip as many times as they like. These workers\nare mostly based in the United States, and on average spend about 27 seconds per transcription.\nThe hand-transcribed results are compared to the existing ground truth to produce a WER. While\nthe existing ground truth transcriptions do have some label error, this is rarely more than 1%. This\nimplies that disagreement between the ground truth transcripts and the human level transcripts is a\ngood heuristic for human level performance.\n6.1.1 Model Size\nOur English speech training set is substantially larger than the size of commonly used speech\ndatasets. Furthermore, the data is augmented with noise synthesis. To get the best generalization\nerror, we expect that the model size must increase to fully exploit the patterns in the data. In Section 3.2 we explored the effect of model depth while \ufb01xing the number of parameters. In contrast,\nhere we show the effect of varying model size on the performance of the speech system. We only\nvary the size of each layer, while keeping the depth and other architectural parameters constant. We\nevaluate the models on the same Regular and Noisy development sets that we use in Section 3.5.\nThe models in Table 11 differ from those in Table 3 in that we increase the the stride to 3 and output"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4a9-6db4-11ee-9215-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-58", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "The models in Table 11 differ from those in Table 3 in that we increase the the stride to 3 and output\nto bigrams. Because we increase the model size to as many as 100 million parameters, we \ufb01nd that\nan increase in stride is necessary for fast computation and memory constraints. However, in this\nregime we note that the performance advantage of the GRU networks appears to diminish over the\n17\nModel size Model type Regular Dev Noisy Dev\n18\u0002106GRU 10.59 21.38\n38\u0002106GRU 9.06 17.07\n70\u0002106GRU 8.54 15.98\n70\u0002106RNN 8.44 15.09\n100\u0002106GRU 7.78 14.17\n100\u0002106RNN 7.73 13.06\nTable 11: Comparing the effect of model size on the WER of the English speech system on both the regular and\nnoisy development sets. We vary the number of hidden units in all but the convolutional layers. The GRU model\nhas 3 layers of bidirectional GRUs with 1 layer of 2D-invariant convolution. The RNN model has 7 layers of\nbidirectional simple recurrence with 3 layers of 2D-invariant convolution. Both models output bigrams with a\ntemporal stride of 3. All models contain approximately 35 million parameters and are trained with BatchNorm"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4aa-6db4-11ee-acd7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-59", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "temporal stride of 3. All models contain approximately 35 million parameters and are trained with BatchNorm\nand SortaGrad.\nTest set DS1 DS2\nBaidu Test 24.01 13.59\nTable 12: Comparison of DS1 and DS2 WER on an internal test set of 3,300 examples. The test set contains a\nwide variety of speech including accents, low signal-to-noise speech, spontaneous and conversational speech.\nsimple RNN. In fact, for the 100 million parameter networks the simple RNN performs better than\nthe GRU network and is faster to train despite the 2 extra layers of convolution.\nTable 11 shows that the performance of the system improves consistently up to 100 million parameters. All further English DS2 results are reported with this same 100 million parameter RNN model\nsince it achieves the lowest generalization errors.\nTable 12 shows that the 100 million parameter RNN model (DS2) gives a 43.4% relative improvement over the 5-layer model with 1 recurrent layer (DS1) on an internal Baidu dataset of 3,300\nutterances that contains a wide variety of speech including challenging accents, low signal-to-noise\nratios from far-\ufb01eld or background noise, spontaneous and conversational speech.\n6.1.2 Read Speech"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ab-6db4-11ee-ba37-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-60", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "ratios from far-\ufb01eld or background noise, spontaneous and conversational speech.\n6.1.2 Read Speech\nRead speech with high signal-to-noise ratio is arguably the easiest large vocabulary for a continuous\nspeech recognition task. We benchmark our system on two test sets from the Wall Street Journal\n(WSJ) corpus of read news articles. These are available in the LDC catalog as LDC94S13B and\nLDC93S6B. We also take advantage of the recently developed LibriSpeech corpus constructed using\naudio books from the LibriV ox project [46].\nTable 13 shows that the DS2 system outperforms humans in 3 out of the 4 test sets and is competitive\non the fourth. Given this result, we suspect that there is little room for a generic speech system to\nfurther improve on clean read speech without further domain adaptation.\nRead Speech\nTest set DS1 DS2 Human\nWSJ eval\u201992 4.94 3.60 5.03\nWSJ eval\u201993 6.94 4.98 8.08\nLibriSpeech test-clean 7.89 5.33 5.83\nLibriSpeech test-other 21.74 13.25 12.69\nTable 13: Comparison of WER for two speech systems and human level performance on read speech.\n18\nAccented Speech\nTest set DS1 DS2 Human"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ac-6db4-11ee-8e26-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-61", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "Table 13: Comparison of WER for two speech systems and human level performance on read speech.\n18\nAccented Speech\nTest set DS1 DS2 Human\nV oxForge American-Canadian 15.01 7.55 4.85\nV oxForge Commonwealth 28.46 13.56 8.15\nV oxForge European 31.20 17.55 12.76\nV oxForge Indian 45.35 22.44 22.15\nTable 14: Comparing WER of the DS1 system to the DS2 system on accented speech.\nNoisy Speech\nTest set DS1 DS2 Human\nCHiME eval clean 6.30 3.34 3.46\nCHiME eval real 67.94 21.79 11.84\nCHiME eval sim 80.27 45.05 31.33\nTable 15: Comparison of DS1 and DS2 system on noisy speech. \u201cCHiME eval clean\u201d is a noise-free baseline.\nThe \u201cCHiME eval real\u201d dataset is collected in real noisy environments and the \u201cCHiME eval sim\u201d dataset has\nsimilar noise synthetically added to clean speech. Note that we use only one of the six channels to test each\nutterance.\n6.1.3 Accented Speech\nOur source for accented speech is the publicly available V oxForge ( http://www.voxforge.org )"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ad-6db4-11ee-a56f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-62", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "utterance.\n6.1.3 Accented Speech\nOur source for accented speech is the publicly available V oxForge ( http://www.voxforge.org )\ndataset, which has clean speech read from speakers with many different accents. We group these\naccents into four categories. The American-Canadian and Indian groups are self-explanatory. The\nCommonwealth accent denotes speakers with British, Irish, South African, Australian and New\nZealand accents. The European group contains speakers with accents from countries in Europe that\ndo not have English as a \ufb01rst language. We construct a test set from the V oxForge data with 1024\nexamples from each accent group for a total of 4096 examples.\nPerformance on these test sets is to some extent a measure of the breadth and quality of our training\ndata. Table 14 shows that our performance improved on all the accents when we include more\naccented training data and use an architecture that can effectively train on that data. However human\nlevel performance is still notably better than that of DS2 for all but the Indian accent.\n6.1.4 Noisy Speech\nWe test our performance on noisy speech using the publicly available test sets from the recently\ncompleted third CHiME challenge [4]. This dataset has 1320 utterances from the WSJ test set"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4ae-6db4-11ee-92a1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-63", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "We test our performance on noisy speech using the publicly available test sets from the recently\ncompleted third CHiME challenge [4]. This dataset has 1320 utterances from the WSJ test set\nread in various noisy environments, including a bus, a cafe, a street and a pedestrian area. The\nCHiME set also includes 1320 utterances with simulated noise from the same environments as well\nas the control set containing the same utterances delivered by the same speakers in a noise-free\nenvironment. Differences between results on the control set and the noisy sets provide a measure of\nthe network\u2019s ability to handle a variety of real and synthetic noise conditions. The CHiME audio\nhas 6 channels and using all of them can provide substantial performance improvements [69]. We\nuse a single channel for all our results, since multi-channel audio is not pervasive on most devices.\nTable 15 shows that DS2 substantially improves upon DS1, however DS2 is worse than human level\nperformance on noisy data. The relative gap between DS2 and human level performance is larger\nwhen the data comes from a real noisy environment instead of synthetically adding noise to clean\nspeech.\n6.2 Mandarin\nIn Table 16 we compare several architectures trained on the Mandarin Chinese speech, on a development set of 2000 utterances as well as a test set of 1882 examples of noisy speech. This development"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4af-6db4-11ee-a093-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-64", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "In Table 16 we compare several architectures trained on the Mandarin Chinese speech, on a development set of 2000 utterances as well as a test set of 1882 examples of noisy speech. This development\nset was also used to tune the decoding parameters We see that the deepest model with 2D-invariant\n19\nconvolution and BatchNorm outperforms the shallow RNN by 48% relative, thus continuing the\ntrend that we saw with the English system\u2014multiple layers of bidirectional recurrence improves\nperformance substantially.\nArchitecture Dev Test\n5-layer, 1 RNN 7.13 15.41\n5-layer, 3 RNN 6.49 11.85\n5-layer, 3 RNN + BatchNorm 6.22 9.39\n9-layer, 7 RNN + BatchNorm + 2D conv 5.81 7.93\nTable 16: Comparison of the improvements in DeepSpeech with architectural improvements. The development\nand test sets are Baidu internal corpora. All the models in the table have about 80 million parameters each\nWe \ufb01nd that our best Mandarin Chinese speech system transcribes short voice-query like utterances\nbetter than a typical Mandarin Chinese speaker. To benchmark against humans we ran a test with\n100 randomly selected utterances and had a group of 5 humans label all of them together. The group"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4b0-6db4-11ee-b4bf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-65", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "better than a typical Mandarin Chinese speaker. To benchmark against humans we ran a test with\n100 randomly selected utterances and had a group of 5 humans label all of them together. The group\nof humans had an error rate of 4.0% as compared to the speech systems performance of 3.7%. We\nalso compared a single human transcriber to the speech system on 250 randomly selected utterances.\nIn this case the speech system performs much better: 9.7% for the human compared to 5.7% for the\nspeech model.\n7 Deployment\nReal-world applications usually require a speech system to transcribe in real time or with relatively\nlow latency. The system used in Section 6.1 is not well-designed for this task, for several reasons.\nFirst, since the RNN has several bidirectional layers, transcribing the \ufb01rst part of an utterance requires the entire utterance to be presented to the RNN. Second, since we use a wide beam when\ndecoding with a language model, beam search can be expensive, particularly in Mandarin where the\nnumber of possible next characters is very large (around 6000). Third, as described in Section 3, we\nnormalize power across an entire utterance, which again requires the entire utterance to be available\nin advance.\nWe solve the power normalization problem by using some statistics from our training set to perform"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ab4b1-6db4-11ee-a5d6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1512.02595-66", "title": "Deep Speech 2: End-to-End Speech Recognition in English and Mandarin", "source": "http://arxiv.org/pdf/1512.02595", "start_index": 0, "chroma:document": "in advance.\nWe solve the power normalization problem by using some statistics from our training set to perform\nan adaptive normalization of speech inputs during online transcription. We can solve the other\nproblems by modifying our network and decoding procedure to produce a model that performs\nalmost as well while having much lower latency. We focus on our Mandarin system since some\naspects of that system are more challenging to deploy (e.g. the large character set), but the same\ntechniques could also be applied in English.\nIn this section, latency refers to the computational latency of our speech system as measured from\nthe end of an utterance until the transcription is produced. This latency does not include data transmission over the internet, and does not measure latency from the beginning of an utterance until the\n\ufb01rst transcription is produced. We focus on latency from end of utterance to transcription because it\nis important to applications using speech recognition.\n7.1 Batch Dispatch\nIn order to deploy our relatively large deep neural networks at low latency, we have paid special attention to ef\ufb01ciency during deployment. Most internet applications process requests individually as\nthey arrive in the data center. This makes for a straightforward implementation where each request\ncan be managed by one thread. However, processing requests individually is inef\ufb01cient computationally, for two main reasons. Firstly, when processing requests individually, the processor must"}


