A Convolutional Neural Network for Learning Local Feature Descriptors on Multispectral Images
Keywords:Local feature descriptor, Multispectral images, Log-Gabor filters, Convolutional neural networks
This work presents a novel convolutional neural network, termed multispectral features network (MF-Net), for learning local feature descriptors in multispectral images. Unlike most existing solutions, which primarily handle images from the visible light spectrum, we propose a learning-based method that deals with image data acquired from different spectrum bands. To design our convolutional neural network, we introduce a new layer that incorporates Log-Gabor filters to enhance the network capability to work with nonlinear intensity variations in images captured from different electromagnetic frequencies spectrum. This layer, entitled mapping layer, can be easily integrated into different network architectures. To demonstrate the efficacy and limitations of our method, we went on experiments with two distinct datasets extensively used in previous works composed of image pairs from the visible spectrum and the infrared spectrum. Experimental results with datasets containing images obtained from visible light and infrared spectrum show that our method can accurately match features, outperforming some state-of-the-art learning-based algorithms.
H. Le, C. Smailis, L. Shi, and I. Kakadiaris, "EDGE20: A cross spectral evaluation dataset for multiple surveillance problems," in 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, mar 2020, DOI: 10.1109/wacv45572.2020.9093573.
Y. Quan, X. Zhong, W. Feng, G. Dauphin, L. Gao, and M. Xing, "A novel feature extension method for the forest disaster monitoring using multispectral data," Remote Sensing, vol. 12, no. 14, p. 2261, jul 2020, DOI: 10.3390/rs12142261.
R. He, J. Cao, L. Song, Z. Sun, and T. Tan, "Adversarial cross-spectral face completion for NIR-VIS face recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 5, pp. 1025–1037, may 2020, DOI: 10.1109/tpami.2019.2961900.
T. Tuytelaars and K. Mikolajczyk, "Local invariant feature detectors: A survey," Foundations and Trends® in Computer Graphics and Vision, vol. 3, no. 3, pp. 177–280, 2007, DOI: 10.1561/0600000017.
C. Fan, H. Jin, F. Wang, G. Zhang, and Y. Li, "Combining and matching keypoints and lines on multispectral images," Infrared Physics & Technology, vol. 96, pp. 316–324, jan 2019, DOI: 10.1016/j.infrared.2018.12.004.
C. Leng, H. Zhang, B. Li, G. Cai, Z. Pei, and L. He, "Local feature descriptor for image matching: A survey," IEEE Access, vol. 7, pp. 6424–6434, 2019, DOI: 10.1109/access.2018.2888856.
C. F. G. Nunes and F. L. C. Padua, "A local feature descriptor based on log-gabor filters for keypoint matching in multispectral images," IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 10, pp. 1850–1854, oct 2017, DOI: 10.1109/lgrs.2017.2738632.
T. Ma, J. Ma, and K. Yu, "A local feature descriptor based on oriented structure maps with guided filtering for multispectral remote sensing image matching," Remote Sensing, vol. 11, no. 8, p. 951, apr 2019, DOI: 10.3390/rs11080951.
M. A. Dede, E. Aptoula, and Y. Genc, "Deep network ensembles for aerial scene classification," IEEE Geoscience and Remote Sensing Letters, vol. 16, no. 5, pp. 732–735, may 2019, DOI: 10.1109/lgrs.2018.2880136.
L. Ren, J. Lu, J. Feng, and J. Zhou, "Uniform and variational deep learning for RGB-d object recognition and person re-identification," IEEE Transactions on Image Processing, vol. 28, no. 10, pp. 4970–4983, oct 2019, DOI: 10.1109/tip.2019.2915655.
L. Ngo, J. Cha, and J.-H. Han, "Deep neural network regression for automated retinal layer segmentation in optical coherence tomography images," IEEE Transactions on Image Processing, vol. 29, pp. 303–312, 2020, DOI: 10.1109/tip.2019.2931461.
Q. Qi, Q. Huo, J. Wang, H. Sun, Y. Cao, and J. Liao, "Personalized sketch-based image retrieval by convolutional neural network and deep transfer learning," IEEE Access, vol. 7, pp. 16 537–16 549, 2019, DOI: 10.1109/access.2019.2894351.
K. Kuppala, S. Banda, and T. R. Barige, "An overview of deep learning methods for image registration with focus on feature-based approaches," International Journal of Image and Data Fusion, vol. 11, no. 2, pp. 113–135, jan 2020, DOI: 10.1080/19479832.2019.1707720.
Y. Dong, W. Jiao, T. Long, L. Liu, G. He, C. Gong, and Y. Guo, "Local deep descriptor for remote sensing image feature matching," Remote Sensing, vol. 11, no. 4, p. 430, feb 2019, DOI: 10.3390/rs11040430.
D. G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, nov 2004, DOI: 10.1023/b:visi.0000029664.99615.94.
H. Bay, T. Tuytelaars, and L. V. Gool, "SURF: Speeded up robust features," in Computer Vision – ECCV 2006. Springer Berlin Heidelberg, 2006, pp. 404–417, DOI: 10.1007/11744023_32.
S. Zagoruyko and N. Komodakis, "Learning to compare image patches via convolutional neural networks," in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2015, DOI: 10.1109/cvpr.2015.7299064.
X. Han, T. Leung, Y. Jia, R. Sukthankar, and A. C. Berg, "MatchNet: Unifying feature and metric learning for patch-based matching," in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2015, DOI: 10.1109/cvpr.2015.7298948.
E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer, "Discriminative learning of deep convolutional feature point descriptors," in 2015 IEEE International Conference on Computer Vision (ICCV). IEEE, dec 2015, DOI: 10.1109/iccv.2015.22.
V. Balntas, E. Johns, L. Tang, and K. Mikolajczyk, "Pn-net: Conjoined triple deep network for learning local image descriptors," arXiv preprint arXiv:1601.05030, 2016.
V. Balntas, E. Riba, D. Ponsa, and K. Mikolajczyk, "Learning local feature descriptors with triplets and shallow convolutional neural networks," in Procedings of the British Machine Vision Conference 2016. British Machine Vision Association, 2016, DOI: 10.5244/c.30.119.
J. L. Schonberger, H. Hardmeier, T. Sattler, and M. Pollefeys, "Comparative evaluation of hand-crafted and learned local features," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jul 2017, DOI: 10.1109/cvpr.2017.736.
C. Aguilera, A. Sappa, C. Aguilera, and R. Toledo, "Cross-spectral local descriptors via quadruplet network," Sensors, vol. 17, no. 4, p. 873, apr 2017, DOI: 10.3390/s17040873.
S. En, A. Lechervy, and F. Jurie, "TS-NET: Combining modality specific and common features for multimodal patch matching," in 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, oct 2018, DOI: 10.1109/icip.2018.8451804.
Z. Fu, Q. Qin, B. Luo, H. Sun, and C. Wu, "HOMPC: A local feature descriptor based on the combination of magnitude and phase congruency information for multi-sensor remote sensing images," Remote Sensing, vol. 10, no. 8, p. 1234, aug 2018, DOI: 10.3390/rs10081234.
Z. Fu, Q. Qin, B. Luo, C. Wu, and H. Sun, "A local feature descriptor based on combination of structure and texture information for multispectral image matching," IEEE Geoscience and Remote Sensing Letters, pp. 1–5, 2018, DOI: 10.1109/lgrs.2018.2867635.
X. Liu, J.-B. Li, and J.-S. Pan, "Feature point matching based on distinct wavelength phase congruency and log-gabor filters in infrared and visible images," Sensors, vol. 19, no. 19, p. 4244, sep 2019, DOI: 10.3390/s19194244.
B. Fang, K. Yu, J. Ma, and P. An, "EMCM: A novel binary edge-feature-based maximum clique framework for multispectral image matching," Remote Sensing, vol. 11, no. 24, p. 3026, dec 2019, DOI: 10.3390/rs11243026.
E. Walia and V. Verma, "Boosting local texture descriptors with log-gabor filters response for improved image retrieval," International Journal of Multimedia Information Retrieval, vol. 5, no. 3, pp. 173–184, apr 2016, DOI: 10.1007/s13735-016-0099-2.
E. Rosten and T. Drummond, "Machine learning for high-speed corner detection," in Computer Vision – ECCV 2006. Springer Berlin Heidelberg, 2006, pp. 430–443, DOI: 10.1007/11744023_34.
T. Mouats, N. Aouf, D. Nam, and S. Vidas, "Performance evaluation of feature detectors and descriptors beyond the visible," Journal of Intelligent & Robotic Systems, vol. 92, no. 1, pp. 33–63, feb 2018, DOI: 10.1007/s10846-017-0762-8.
S. Saleem, A. Bais, R. Sablatnig, A. Ahmad, and N. Naseer, "Feature points for multisensor images," Computers & Electrical Engineering, vol. 62, pp. 511–523, aug 2017, DOI: 10.1016/j.compeleceng.2017.04.032.
K. Mikolajczyk and C. Schmid, "A performance evaluation of local descriptors," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 10, pp. 1615–1630, oct 2005, DOI: 10.1109/tpami.2005.188.
F. Campelo and E. F. Wanner, "Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances," Journal of Heuristics, vol. 26, no. 6, pp. 851–883, aug 2020, DOI: 10.1007/s10732-020-09454-w.