Despite much research progress in image semantic segmentation, it remains challenging under adverse environmental conditions caused by imaging limitations of visible spectrum. While thermal infrared cameras have several advantages over cameras for the visible spectrum, such as operating in total darkness, insensitive to illumination variations, robust to shadow effects and strong ability to penetrate haze and smog. These advantages of thermal infrared cameras make the segmentation of semantic objects in day and night. In this paper, we propose a novel network architecture, called edge-conditioned convolutional neural network (EC-CNN), for thermal image semantic segmentation. Particularly, we elaborately design a gated feature-wise transform layer in EC-CNN to adaptively incorporate edge prior knowledge. The whole EC-CNN is end-to-end trained, and can generate high-quality segmentation results with the edge guidance. Meanwhile, we also introduce a new benchmark dataset named "Segment Objects in Day And night"(SODA) for comprehensive evaluations in thermal image semantic segmentation. SODA contains over 7,168 manually annotated and synthetically generated thermal images with 20 semantic region labels and from a broad range of viewpoints and scene complexities. Extensive experiments on SODA demonstrate the effectiveness of the proposed EC-CNN against the state-of-the-art methods.
@cite_36: Sea–land segmentation and ship detection are two prevalent research domains for optical remote sensing harbor images and can find many applications in harbor supervision and management. As the spatial resolution of imaging technology improves, traditional methods struggle to perform well due to the complicated appearance and background distributions. In this paper, we unify the above two tasks into a single framework and apply the deep convolutional neural networks to predict pixelwise label for an input. Specifically, an edge aware convolutional network is proposed to parse a remote sensing harbor image into three typical objects, e.g., sea, land, and ship. Two innovations are made on top of the deep structure. First, we design a multitask model by simultaneously training the segmentation and edge detection networks. Hierarchical semantic features from the segmentation network are extracted to learn the edge network. Second, the outputs of edge pipeline are further employed to refine entire model by adding an edge aware regularization, which helps our method to yield very desirable results that are spatially consistent and well boundary located. It also benefits the segmentation of docked ships that are quite challenging for many previous methods. Experimental results on two datasets collected from Google Earth have demonstrated the effectiveness of our approach both in quantitative and qualitative performance compared with state-of-the-art methods.
@cite_22: Deep convolutional neural networks (CNNs) are the backbone of state-of-art semantic image segmentation systems. Recent work has shown that complementing CNNs with fully-connected conditional random fields (CRFs) can significantly enhance their object localization accuracy, yet dense CRF inference is computationally expensive. We propose replacing the fully-connected CRF with domain transform (DT), a modern edge-preserving filtering method in which the amount of smoothing is controlled by a reference edge map. Domain transform filtering is several times faster than dense CRF inference and we show that it yields comparable semantic segmentation results, accurately capturing object boundaries. Importantly, our formulation allows learning the reference edge map from intermediate CNN features instead of using the image gradient magnitude as in standard DT filtering. This produces task-specific edges in an end-to-end trainable system optimizing the target semantic segmentation quality.
@cite_23: In this work we show that adapting Deep Convolutional Neural Network training to the task of boundary detection can result in substantial improvements over the current state-of-the-art in boundary detection. Our contributions consist firstly in combining a careful design of the loss for boundary detection training, a multi-resolution architecture and training with external data to improve the detection accuracy of the current state of the art. When measured on the standard Berkeley Segmentation Dataset, we improve theoptimal dataset scale F-measure from 0.780 to 0.808 - while human performance is at 0.803. We further improve performance to 0.813 by combining deep learning with grouping, integrating the Normalized Cuts technique within a deep network. We also examine the potential of our boundary detector in conjunction with the task of semantic segmentation and demonstrate clear improvements over state-of-the-art systems. Our detector is fully integrated in the popular Caffe framework and processes a 320x420 image in less than a second.
Besides, there are some works to use edge cues for image semantic segmentation @cite_22 @cite_23 @cite_36 . Chen @cite_22 propose to replace the fully-connected CRF with domain transform which is a modern edge-preserving filtering method in which the amount of smoothing is controlled by a reference edge map. Kokkinos @cite_23 designs CRF-based method, which intergrates CNN score-maps with pairwise features derived from advanced boundary detection, to more precisely localize semantic segmentation boundaries. The outputs of edge network are employed to refine semantic segmentation network by adding an edge aware regularization so as to yield spatially consistent and well boundary located results @cite_36 .