Evaluation of Change Detection Algorithms using Difficulty Maps
Keywords:
Algorithm evaluation, change detection, difficulty map, video segmentationAbstract
The evaluation of a change detection algorithm should show its superiority over state-of-the-art algorithms' performances. Evaluating an algorithm involves executing it to segment a set of videos and comparing the results with the ground truth. Here, we used the difficulty level to classify each pixel of each frame of the videos of a dataset as an algorithm performance measure. A structure called "difficulty map" stores information about the difficulty of classifying each pixel in a frame. Based on these maps, we developed a metric that aims to evaluate the performance of algorithms on the difficulty map. The results showed that there are algorithms with the characteristic of classifying pixels that most state-of-the-art algorithms cannot classify (promising algorithms). Identifying such algorithms is essential since improving their performance means facing challenges already overcome by existing approaches.
Downloads
References
S. R. R. Sanches, C. Oliveira, A. C. Sementille, and V. Freire, “Challenging situations for background subtraction algorithms,” Applied Intelligence, vol. 49, pp. 1771–1784, May 2019.
S. R. R. Sanches, R. Nakamura, V. Freire Silva, and R. Tori, “Bilayer segmentation of live video in uncontrolled environments for background substitution: An overview and main challenges,” IEEE Latin America Transactions, vol. 10, no. 5, pp. 2138–2149, 2012.
V. B. d. O. Barth, R. de Oliveira, M. A. de Oliveira, and V. E. do Nascimento, “Vehicle speed monitoring using convolutional neural networks,” IEEE Latin America Transactions, vol. 17, p. 1000–1008, Nov. 2019.
M. Valdeos, A. S. Vadillo Velazco, M. G. Pérez Paredes, and R. M. Arias Velásquez, “Methodology for an automatic license plate recognition system using convolutional neural networks for a peruvian case study,” IEEE Latin America Transactions, vol. 20, p. 1032–1039, Mar. 2022.
N. Goyette, P. M. Jodoin, F. Porikli, J. Konrad, and P. Ishwar, “Changedetection.net: A new change detection benchmark dataset,” in 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–8, June 2012.
A. Sobral and A. Vacavant, “A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos,” Computer Vision and Image Understanding, vol. 122, pp. 4–21, 2014.
S. R. R. Sanches, A. C. Sementille, I. A. Aguilar, and V. Freire, “Recommendations for evaluating the performance of background subtraction algorithms for surveillance systems,” Multimedia Tools and Applications, vol. 80, no. 3, p. 4421–4454, 2021.
UNIVERSITÉ DE SHERBROOKE, “ChangeDetection.NET – a video database for testing change detection algorithms,” 2019. http://www.changedetection.net. Accessed 22 July 2018.
C. M. Silva, K. A. I. Rosa, P. H. Bugatti, P. T. M. Saito, C. G. Corrêa, R. S. Yokoyama, and S. R. R. Sanches, “Method for selecting representative videos for change detection datasets,” Multimed Tools and Applications, vol. 81, no. 3, pp. 3773–3791, 2022.
Université de Sherbrooke, “ChangeDetection.NET – a video database for testing change detection algorithms,” 2022. http://www.changedetection.net. Accessed 5 Jan 2022.
UNIVERSITÉ DE SHERBROOKE, “Results for CD.net 2014,” 2019. http://jacarini.dinf.usherbrooke.ca/results2014/. Accessed 19 Mar 2020.
K. Wang, C. Gou, and F.-Y. Wang, “M4cd: A robust change detection method for intelligent visual surveillance,” 2018. https://arxiv.org/abs/1802.04979. Cornell University. Accessed 12 Nov 2019.
S. Isik, K. Özkan, S. Günal, and O. N. Gerek, “Swcd: a sliding window and self-regulated learning-based background updating method for change detection in videos,” Journal of Electronic Imaging, vol. 27, no. 2, pp. 1–11, 2018.
S. Bianco, G. Ciocca, and R. Schettini, “How far can you get by combining change detection algorithms?,” in Image Analysis and Processing - ICIAP 2017 (S. Battiato, G. Gallo, R. Schettini, and F. Stanco, eds.), (Cham), pp. 96–107, Springer International Publishing, 2017.
B. Wang and P. Dudek, “A fast self-tuning background subtraction algorithm,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 401–404, June 2014.
Y. Wang, Z. Luo, and P.-M. Jodoin, “Interactive deep learning method for segmenting moving objects,” Pattern Recognition Letters, vol. 96, pp. 66 – 75, 2017. Scene Background Modeling and Initialization.
M. Babaee, D. T. Dinh, and G. Rigoll, “A deep convolutional neural network for video sequence background subtraction,” Pattern Recogni tion, vol. 76, pp. 635 – 649, 2018.
L. A. Lim and H. Y. Keles, “Learning multi-scale features for foreground segmentation,” Pattern Analysis and Applications, Aug 2019.
X. Lu, “A multiscale spatio-temporal background model for motion detection,” in 2014 IEEE International Conference on Image Processing (ICIP), pp. 3268–3271, Oct 2014.
M. Braham, S. Piérard, and M. V. Droogenbroeck, “Semantic background subtraction,” in 2017 IEEE International Conference on Image Processing (ICIP), pp. 4552–4556, 2017.
