Combining deep learning model compression techniques

Authors

Keywords:

Deep learning, model compression, dark knowledge distillation, pruning, quantization

Abstract

In this article, we evaluate the performance of combining several model compression techniques. The techniques assessed were dark knowledge distillation, pruning, and quantization. We found that in the scenario in which we developed the experiments, classification of chest x-rays, the combination of these three techniques yielded a new model capable of aggregating the individual advantages of each one. In the experiments we used a combination of deep models with 95.05% accuracy, a value higher than that reported in some related works but lower than the state of the art, whose accuracy is 96.39%. The accuracy of the compressed model in turn was 90.86%, a small loss compared to the gain obtained from the reduction, in bytes, in relation to the size of the original model. The size has been reduced from 841MB to 40KB, which opens up the possibility for using deep models in edge computing applications.

Downloads

Download data is not yet available.

Author Biographies

José Vitor Santos Silva, Universidade Federal de Sergipe

José Vitor Santos Silva is an undergraduate computer engineering student at the Federal University of Sergipe, where he also works as a researcher in the field of Machine Learning (ML). His interests involve deep learning, computer vision and edge computing. Some of his scientific works were awarded at events in Brazil. Currently works as a software developer.

Leonardo Matos Matos, Federal University of Sergipe (UFS), São Cristóvão, Sergipe, Brasil

Leonardo Matos is an associate professor at the Department of Computing of the Federal University of Sergipe, Brazil. His main area of interest is Machine Learning, particularly Explainable Artificial Intelligence, model compression and applications including speech recognition and computer vision. He has several articles published in conferences and scientific journals. Some of his recent work with undergraduate students has been awarded at events in Brazil. He has been teaching Machine Learning in the Computer Science  Postgraduate Program at the Federal University of Sergipe, mentoring students and participating as a member of the master's dissertation defense jury. He is a member of the  Brazilian Computer Society, a collaborator of the ISLab group at the University of Minho in Portugal and a reviewer of articles in conferences and scientific journals in the field of Computing.

Flávio Santos, Federal University of Pernambuco(UFS), Recife, Pernambuco, Brasil

Flávio Santos is a computer science PhD student at the Informatics Center (CIn) of the Federal University of Pernambuco (UFPE). In addition, he is also a researcher at Centro Algoritmi of the School of Engineering at the University of Minho, Braga, Portugal. He has developed scientific work in Deep Learning, with applications in natural language processing and computer vision. Currently, his main research interest is interpretation and robustness of the adversarial attacks of deep learning models.

Hélisson Oliveira, Federal University of Sergipe (UFS), São Cristóvão, Sergipe, Brasil

Hélisson Oliveira Magalhães Cerqueira is a computer science student at the Federal University of Sergipe and works professionally in the data science area of the Secretariat of Finance of Sergipe. His interests involve machine learning, big data and software development.

Hendrik Macedo, Federal University of Sergipe (UFS), São Cristóvão, Sergipe, Brasil

Hendrik Teixeira Macedo obtained a bachelor's degree in Computer Science from the Federal University of Sergipe (UFS) in 1998 and completed his PhD degree in Computer Science from the Federal University of Pernambuco (UFPE), with a "Sandwich" internship at the University of Paris VI, in 2002. Since 2006, Hendrik has been an effective professor at DCOMP/UFS and, since 2010, has been part of the permanent faculty of the Computer Science Postgraduate Center (PROCC/UFS). Since 2015, Hendrik has been a CNPq Productivity in Technological Development and Innovative Extension scholarship. Research areas of greatest interest are Machine Learning, Natural Language Processing and Multi-Agent Systems.

Bruno Prado, Federal University of Sergipe (UFS), São Cristóvão, Sergipe, Brasile

Bruno Otávio Piedade Prado is Associate Professor at the Department of Computing (DCOMP) at the Federal University of Sergipe (UFS) and has been working at this institution since 2012. The main focus of his research has been the use of Machine Learning (AM) for complex classification problems  in the restricted environment of embedded systems. Due to its low unit cost and scalability, research efforts are focused on reducing computational complexity, without compromising the usefulness of the classifications obtained. For this purpose, network compression strategies, use of vector instructions (SIMD) or acceleration by dedicated reprogrammable hardware with FPGA have been explored.

Gilton Silva, Federal University of Sergipe (UFS), São Cristóvão, Sergipe, Brasil

Gilton José Ferreira da Silva is a Professor (DCOMP/UFS), Postgraduate Advisor (PROCC/UFS), Mentor and Digital Influencer (@giltonmal). PhD in Intellectual Property Science. He works with the lines of research in Information Systems (IS), Smart Cities, Design Thinking, UX, Creativity, Innovation, Educational Technologies, Intellectual Property and Digital Marketing.

Kalil Bispo, Universidade Federal de Sergipe

Kalil Araujo Bispo holds a degree in Computer Science from the Federal University of Sergipe (2006), a Masters in Computer Science from the Federal University of Pernambuco (2009) and a Ph.D. in Computer Science from the Federal University of Pernambuco (2015). He is currently an adjunct professor at the Federal University of Sergipe. He has experience in Computer Science, with an emphasis on Distributed Systems.

References

P. Rajpurkar, J. Irvin, R. L. Ball, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. P. Langlotz, et al., “Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists,” PLoS medicine, vol. 15, no. 11, p. e1002686, 2018.

G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015. [3] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing

deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149, 2015.

B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704– 2713, 2018.

W. H. O. WHO, “Standardization of interpretation of chest radiographs for the diagnosis of pneumonia in children,” tech. rep., World Health Organization, 2001. [6] T. Franquet, “Imaging of pneumonia: trends and algorithms,” European Respiratory Journal, vol. 18, no. 1, pp. 196–208, 2001.

J. V. Silva and L. Matos, “Detecção de pneumonia usando redes neurais convolucionais treinadas com destilação do conhecimento obscuro,” in Anais da XX Escola Regional de Computação Bahia, Alagoas e Sergipe, pp. 51–60, SBC, 2020.

S. A. Janowsky, “Pruning versus clipping in neural networks,” Physical Review A, vol. 39, no. 12, p. 6600, 1989. [9] D. Blalock, J. J. G. Ortiz, J. Frankle, and J. Guttag, “What is the state of neural network pruning?,” arXiv preprint arXiv:2003.03033, 2020. [10] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in International conference on machine learning, pp. 1737–1746, PMLR, 2015.

I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks,” in Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 4114–4122, 2016.

D. S. Kermany, M. Goldbaum, W. Cai, C. C. Valentim, H. Liang, S. L. Baxter, A. McKeown, G. Yang, X. Wu, F. Yan, et al., “Identifying medical diagnoses and treatable diseases by image-based deep learning,”

Cell, vol. 172, no. 5, pp. 1122–1131, 2018.

R. Caruana, S. Lawrence, and L. Giles, “Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping,” Advances in neural information processing systems, pp. 402–408, 2001.

V. Chouhan, S. K. Singh, A. Khamparia, D. Gupta, P. Tiwari, C. Moreira, R. Damaševičius, and V. H. C. De Albuquerque, “A novel transfer learning based approach for pneumonia detection in chest x-ray images,” Applied Sciences, vol. 10, no. 2, p. 559, 2020.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, pp. 1097–1105, 2012.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708, 2017.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9, 2015.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818– 2826, 2016.

Downloads

Published

2021-10-04

How to Cite

Santos Silva, J. V., Matos Matos, L., Santos, F., Magalhães Cerqueira, H. O., Macedo, H., Piedade Prado, B. O., Ferreira da Silva, G. J. ., & Bispo, K. A. (2021). Combining deep learning model compression techniques. IEEE Latin America Transactions, 20(3), 458–464. Retrieved from https://latamt.ieeer9.org/index.php/transactions/article/view/5824

Issue

Vol. 20 No. 3 (2022): Ordinary Issue

Section

Articles