Fast Crack Segmentation with Depth-to-Space Operator for Pavement Maintenance
Keywords:
Convolutional neural network, Depth-to-space operator, Pavement crack segmentationAbstract
The quality of a city's infrastructure drives socioeconomic development. Specifically, it is important to streamline pavement quality monitoring to improve transportation. However, crack segmentation is a computational challenging problem that requires a fast response. In this paper, we propose a Fully Convolutional Network (FCN) for pavement crack segmentation using depth-to-space operation in the decoder and direct connections between the encoder and decoder layers to improve segmentation performance. This approach reduces the number of layers in the decoder. Consequently, training and inference computational costs are reduced. We tested our model on public datasets for comparison with fast state-of-the-art methods. Our model yielded better performance with lower computational costs, reaching real-time segmentation at the rate of 11 frames-per-second. Besides, we introduce a new dataset called CrackIPN as a benchmark that has four times more images and greater image diversity than commonly used datasets.
Downloads
References
H. Cheng, J.-R. Chen, C. Glazier, and Y. Hu, “Novel approach to pavement cracking detection based on fuzzy set theory,” Journal of Computing in Civil Engineering, vol. 13, no. 4, pp. 270–280, 1999.
T. S. Nguyen, M. Avila, and S. Begot, “Automatic detection and classification of defect on road pavement using anisotropy measure,” in 2009 17th European Signal Processing Conference, pp. 617–621, Aug 2009.
T. S. Nguyen, S. Begot, F. Duculty, and M. Avila, “Free-form anisotropy: A new method for crack detection on pavement surface images,” in 2011 18th IEEE International Conference on Image Processing, pp. 1069–1072, Sept 2011.
Q. Li and X. Liu, “Novel approach to pavement image segmentation based on neighboring difference histogram method,” in 2008 Congress on Image and Signal Processing, vol. 2, pp. 792–796, May 2008.
M. S. Kaseko and S. G. Ritchie, “A neural network-based methodology for pavement crack detection and classification,” Transportation Research Part C: Emerging Technologies, vol. 1, no. 4, pp. 275 – 291,
H. Zhao, G. Qin, and X. Wang, “Improvement of canny algorithm based on pavement edge detection,” in 2010 3rd International Congress on Image and Signal Processing, pp. 964–967, IEEE, oct 2010.
H. Oliveira and P. L. Correia, “Crackit - an image processing toolbox for crack detection and characterization,” in 2014 IEEE International Conference on Image Processing (ICIP), IEEE, oct 2014.
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 (N. Navab, J. Horneg-ger, W. M. Wells, and A. F. Frangi, eds.), (Cham), pp. 234–241, Springer International Publishing, 2015.
V. Badrinarayanan, A. Kendall, and R. Cipolla, “SegNet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, pp. 2481–2495, dec 2017.
A. Urbonas, V. Raudonis, R. Maskeliūnas, and R. Damaševičius, “Automated identification of wood veneer surface defects using faster region-based convolutional neural network with data augmentation and transfer learning,” Applied Sciences, vol. 9, p. 4898, nov 2019.
U. Budak, Y. Guo, E. Tanyildizi, and A. Sengur, “Cascaded deep convolutional encoder-decoder neural networks for efficient liver tumor segmentation,” Medical Hypotheses, vol. 134, p. 109431, jan 2020.
F. Yang, L. Zhang, S. Yu, D. Prokhorov, X. Mei, and H. Ling, “Feature pyramid and hierarchical boosting network for pavement crack detection,” IEEE Transactions on Intelligent Transportation Systems, vol. 21, pp. 1525–1535, apr 2020.
C. Shao, Y. Chen, F. Xu, and S. Wang, “A kind of pavement crack detection method based on digital image processing,” in 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), IEEE, dec 2019.
J. Shi, Z. Li, T. Zhu, D. Wang, and C. Ni, “Defect detection of industry wood veneer based on NAS and multi-channel mask r-CNN,” Sensors, vol. 20, p. 4398, aug 2020.
Y. Shi, L. Cui, Z. Qi, F. Meng, and Z. Chen, “Automatic road crack detection using random structured forests,” IEEE Transactions on Intelligent Transportation Systems, vol. 17, pp. 3434–3445, Dec 2016.
Z. Fan, Y. Wu, J. Lu, and W. Li, “Automatic pavement crack detection based on structured prediction with the convolutional neural network,” arXiv preprint arXiv:1802.02208, 2018.
H. Majidifard, Y. Adu-Gyamfi, and W. G. Buttlar, “Deep machine learning approach to develop a new asphalt pavement condition index,” Construction and Building Materials, vol. 247, p. 118513, jun 2020.
M. Rajagopal, M. Balasubramanian, and S. Palanivel, “An efficient framework to detect cracks in rail tracks using neural network classifier,” Computación y Sistemas, vol. 22, sep 2018.
Y.-J. Cha, W. Choi, and O. Büyüköztürk, “Deep learning-based crack damage detection using convolutional neural networks,” Comput.-Aided Civ. Infrastruct. Eng., vol. 32, pp. 361–378, May 2017.
A. Zhang, K. C. P. Wang, B. Li, E. Yang, X. Dai, Y. Peng, Y. Fei, Y. Liu, J. Q. Li, and C. Chen, “Automated pixel level pavement crack detection on 3d asphalt surfaces using a deep learning network,” Computer-Aided Civil and Infrastructure Engineering, vol. 32, no. 10, pp. 805–819, 2017.
R. Amhaz, S. Chambon, J. Idier, and V. Baltazart, “Automatic crack detection on two-dimensional pavement images: An algorithm based on minimal path selection,” IEEE Transactions on Intelligent Transportation Systems, vol. 17, pp. 2718–2729, Oct 2016.
Y. Ren, J. Huang, Z. Hong, W. Lu, J. Yin, L. Zou, and X. Shen, “Image-based concrete crack detection in tunnels using deep fully convolutional networks,” Construction and Building Materials, vol. 234, p. 117367, feb 2020.
K. Gopalakrishnan, S. K. Khaitan, A. Choudhary, and A. Agrawal, “Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection,” Construction and Building Materials, vol. 157, pp. 322 – 330, 2017.
Y. Liu, J. Yao, X. Lu, R. Xie, and L. Li, “DeepCrack: A deep hierarchical feature learning architecture for crack segmentation,” Neurocomputing, vol. 338, pp. 139–153, apr 2019.
N. A. M. Yusof, M. K. Osman, Z. Hussain, M. H. M. Noor, A. Ibrahim, N. M. Tahir, and N. Z. Abidin, “Automated asphalt pavement crack detection and classification using deep convolution neural network,” in 2019 9th IEEE International Conference on Control System, Computing and Engineering (ICCSCE), IEEE, nov 2019.
Q. Mei and M. Gül, “A cost effective solution for pavement crack inspection using cameras and deep neural networks,” Construction and Building Materials, vol. 256, p. 119397, sep 2020.
D. Mazzini, P. Napoletano, F. Piccoli, and R. Schettini, “A novel approach to data augmentation for pavement distress segmentation,” Computers in Industry, vol. 121, p. 103225, oct 2020.
N. Sholevar, A. Golroo, and S. R. Esfahani, “Machine learning techniques for pavement condition evaluation,” Automation in Construction, vol. 136, p. 104190, 2022.
U. Escalona, F. Arce, E. Zamora, and J. H. S. Azuela, “Fully convolutional networks for automatic pavement crack segmentation,” Computacion y Sistemas, vol. 23, jun 2019.
J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440, June 2015.
S. Zhou and W. Song, “Concrete roadway crack segmentation using encoder-decoder networks with range images,” Automation in Construction, vol. 120, p. 103403, dec 2020.
K. Zhang, Y. Zhang, and H.-D. Cheng, “Crackgan: Pavement crack detection using partially accurate ground truths based on generative adversarial learning,” IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 2, pp. 1306–1319, 2020.
M. Sun, R. Guo, J. Zhu, and W. Fan, “Roadway crack segmentation based on an encoder-decoder deep network with multi-scale convolutional blocks,” in 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), IEEE, jan 2020.
A. Canziani, A. Paszke, and E. Culurciello, “An analysis of deep neural network models for practical applications,” arXiv preprint arXiv:1605.07678, 2016.
W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video superresolution using an efficient sub-pixel convolutional neural network,” in
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, jun 2016.
A. Chatterjee and Y.-C. Tsai, “A fast and accurate automated pavement crack detection algorithm,” in 2018 26th European Signal Processing Conference (EUSIPCO), IEEE, sep 2018.
Q. Zou, Z. Zhang, Q. Li, X. Qi, Q. Wang, and S. Wang, “DeepCrack: Learning hierarchical convolutional features for crack detection,” IEEE Transactions on Image Processing, vol. 28, pp. 1498–1512, mar 2019.
D. Kang, S. S. Benipal, D. L. Gopal, and Y.-J. Cha, “Hybrid pixel-level concrete crack segmentation and quantification across complex backgrounds using deep learning,” Automation in Construction, vol. 118, p. 103291, oct 2020.
C. Peng, M. Yang, Q. Zheng, J. Zhang, D. Wang, R. Yan, J. Wang, and B. Li, “A triple-thresholds pavement crack detection method leveraging random structured forest,” Construction and Building Materials, vol. 263, p. 120080, dec 2020.
R. Kalfarisi, Z. Y. Wu, and K. Soh, “Crack detection and segmentation using deep learning with 3d reality mesh model for quantitative assessment and integrated visualization,” Journal of Computing in Civil Engineering, vol. 34, p. 04020010, may 2020.
H. Li, J. Zong, J. Nie, Z. Wu, and H. Han, “Pavement crack detection algorithm based on densely connected and deeply supervised network,” IEEE Access, vol. 9, pp. 11835–11842, 2021.
S. F. Mahenge, S. Wambura, and L. Jiao, “Robust deep representation learning for road crack detection,” in 2021 The 5th International Conference on Video and Image Processing, ICVIP 2021, (New York, NY, USA), p. 117–125, Association for Computing Machinery, 2021.
S. Shim, J. Kim, S.-W. Lee, and G.-C. Cho, “Road damage detection using super-resolution and semi-supervised learning with generative adversarial network,” Automation in Construction, vol. 135, p. 104139, 2022.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” Computer Vision and Pattern Recognition, 2014.
Z. Wojna, V. Ferrari, S. Guadarrama, N. Silberman, L.-C. Chen, A. Fathi, and J. Uijlings, “The devil is in the decoder,” in British Machine Vision Conference 2017, BMVC 2017, pp. 1–13, BMVA Press, 2017.
S. Aich, W. van der Kamp, and I. Stavness, “Semantic binary segmentation using convolutional networks without decoders,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, jun 2018.
L. Zhang, F. Yang, Y. D. Zhang, and Y. J. Zhu, “Road crack detection using deep convolutional neural network,” in 2016 IEEE International Conference on Image Processing (ICIP), IEEE, sep 2016.
S. Chambon and J.-M. Moliard, “Automatic road pavement assessment with image processing: Review and comparison,” International Journal of Geophysics, vol. 2011, pp. 1–20, 2011.
Chollet and Francois, “Keras,” https://keras.io, 2015.
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al., “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” arXiv preprint arXiv:1603.04467, 2016.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 3rd International Conference for Learning Representations, 2014. [52] U. Escalona, E. Zamora, and H. Sossa, “Fast crack segmenta-
tion with depth-to-space operator.” https://github.com/UrielEscalona/CrackIPN, 2022.
U. Escalona, E. Zamora, and H. Sossa, “Video: Fast crack segmentation with depth-to-space operator.” https://youtu.be/tX3QPYGtNyM, 2022.
