Study of the Effect of Combining Activation Functions in a Convolutional Neural Network

Authors

Keywords:

Convolutional Neural Networks, activation functions

Abstract

Convolutional Neural Networks (CNN’s) have proven to be an effective approach for solving image classification problems. The output, the accuracy and the computational efficiency of a CNN are determined mainly by the architecture, the convolutional filters, and the activation functions. Based on the importance of an activation function, in this paper, nine new activation functions based on combinations of classical functions such as ReLU and sigmoid are presented. Also, a study about the effects caused by the activation functions in the performance of a CNN is presented. First, every new function is described, also, their graphs, analytic forms and derivatives are presented. Then, a traditional CNN model with each new activation function is used to classify three 10-class databases: MNIST, Fashion MNIST and a handwritten digit database created by us. Experimental results illustrate that some of the proposed activation functions lead to better performances on classifying than classical activation functions. Moreover, our study demonstrated that the accuracy of a CNN could be increased by 1.18% with the new proposed activation functions.

Downloads

Download data is not yet available.

Author Biographies

Guevara Neri Maria Cristina, Universidad Autónoma de Ciudad Juárez

María Cristina Guevara Neri was born in Ciudad Juárez, Chihuahua, México, on March 17, 1988. She received the B.S. degree in electromechanical engineering from the Instituto Tecnológico de Ciudad Juárez, México, in 2012; and the M. S. degree in electric engineering at the Universidad Autónoma de Ciudad Juárez. Currently, she is studying the Ph.D. on advanced engineering.

Vianey Guadalupe Cruz Sanchez, Universidad Autónoma de Ciudad Juárez

Vianey Guadalupe Cruz Sánchez was born in Cárdenas, Tabasco, México, on September 14, 1978. She earned the B.S. degree in computer engineering from the Instituto Tecnológico de Cerro Azul, México, in 2000; the M.Sc. degree in computer science at the Center of Research and Technological Development (CENIDET) in 2004; and the Ph.D. in computer science from CENIDET in 2010. She currently serves as a professor at the Autonomous University of Ciudad Juarez, Chihuahua, México. She is a member of the IEEE Computer Society. Her fields of interest include neuro symbolic hybrid systems, digital image processing, knowledge representation, artificial neural networks and augmented reality.

Osslan Osiris Vergara Villegas, Universidad Autónoma de Ciudad Juárez

Osslan Vergara (SM’12), was born in Cuernavaca, Morelos, Mexico on July 3, 1977. He earned the BS degree in Computer Engineering from the Instituto Tecnologico de Zacatepec, Mexico, in 2000; the MSc in Computer Science at the Center of Research and Technological Development (CENIDET) in 2003; and the PhD degree in Computer Science from CENIDET in 2006. He currently serves as a professor at the Universidad Autónoma de Ciudad Juárez, Chihuahua, Mexico, where he is the head of the Computer Vision and Augmented Reality laboratory. Prof. Vergara is a level one member of the Mexican National Research System. He serves several peer-reviewed international journals and conferences as editorial board member and as a reviewer. He has coauthored more than 100 book chapters, journals, and international conference papers. Dr. Vergara has directed more than 50 BS, MSc, and PhD thesis. He is a senior member of the IEEE Computer Society and member of the Mexican Computing Academy. His fields of interest include pattern recognition, digital image processing, augmented reality and mechatronics

Manuel Nandayapa, Universidad Autónoma de Ciudad Juárez

Manuel Nandayapa received a B.S. degree in Electronics Engineering from Institute of Technology of Tuxtla Gutierrez, Chiapas, Mexico in 1997, M.S. degree in Mechatronics Engineering from CENIDET, Morelos, Mexico in 2003, and D.Eng degree in energy and environmental science from the Nagaoka University of Technology, Japan, in 2012. His research interests include mechatronics, motion control, and haptic interfaces. He is with the Department of Industrial and Manufacturing Engineering at Autonomous University of Ciudad Juarez. Dr. Nandayapa is Member of the IEEE Industrial Electronics Society and Robotics Automation Society.

Humberto de Jesus, Universidad Autónoma de Ciudad Juárez

Humberto de Jesus Ochoa received the B.Eng. degree in industrial electronics from the Technological Institute of Veracruz, México, his M.Sc. in electronics from the Technological Institute of Chihuahua, México, and Ph.D. degree in electrical engineering from the University of Texas at Arlington, USA. He is currently with the Department of Ingeniería Eléctrica y Computación at the Universidad Autónoma de Ciudad Juárez, Mexico. He worked as an Electronic Officer for the Mexican Merchant Marine. His current teaching and research interests include multirate systems for medical image analysis, images restoration and reconstruction, image and video coding, statistical signal processing and pattern recognition.

Juan Humberto Sossa Azuela, Instituto Politécnico Nacional (CIC-IPN)

Juan Humberto Sossa Azuela received his BS degree in Communications and Electronics from the University of Guadalajara in 1980. He obtained his Master’s degree in electrical engineering from CINVESTAV-IPN in 1987 and his PhD in Informatics form the INPG, France in 1992. He is currently a full-time professor at the Robotics and Mechatronics Laboratory of the Center for Computing Research of the National Polytechnic Institute from Mexico since 1996. He has more than 450 journal and conference publications. He is a Senior Member of the IEEE.

Downloads

Published

2021-06-07

How to Cite

Maria Cristina, G. N., Cruz Sanchez, V. G., Vergara Villegas, O. O. ., Nandayapa, M. ., Humberto de Jesus, & Sossa Azuela, J. H. (2021). Study of the Effect of Combining Activation Functions in a Convolutional Neural Network. IEEE Latin America Transactions, 19(5), 844–852. Retrieved from https://latamt.ieeer9.org/index.php/transactions/article/view/4134

Issue

Vol. 19 No. 5 (2021): Ordinary Issue

Section

Articles