Continual Refoircement Learning Using Real-World Data for Intelligent Prediction of SOC Consumption in Electric Vehicles
Keywords:
Meta-experience replay (MER, reinforcementlearning (RL), state of charge (SOC), electric vehicles (EV), neural networks (NN), principal component analysis (PCA)Abstract
The accelerated migration towards electric vehicles (EV) presents several problems to solve. The main aspect is the management and prediction of the state of charge (SOC) in real long-range routes of different variations in altitude for a more efficient energy consumption and vehicle recharge plan. This paper presents the implementation of a new algorithm for SOC estimation based on continuous learning and meta-experience replay (MER) with reservoir sample. It combines the reptile meta-learning algorithm with the experience replay technique for stabilizing the reinforcement learning. The proposed algorithm considers several important factors for the prediction of the SOC in EV such as: speed, travel time, route altimetry, consumed battery capacity, regenerated battery capacity. A modified principal components analysis is used to reduce the dimensionality of the route altimetry data. The experimental results show an efficient estimation of the SOC values and a convergent increase in knowledge while the vehicle travels the routes.
Downloads
References
V. T. Tran, D. Sutanto, and K. M. Muttaqi, “The state of the art of battery charging infrastructure for electrical vehicles: Topologies, power control strategies, and future trend,” in2017 Australasian Universities Power Engineering Conference (AUPEC), Nov 2017, pp. 1–6.
S. Zhang, R. Zhao, J. Gu, and J. Liu, “A numerical study of lithium-ion battery fast charging behaviors,” in2018 International Conference on Sensing, Diagnostics, Prognostics, and Control (SDPC), Aug 2018, pp.449–454.
Y. M. Yeap and D. Tran, “Analysis of driving behavior’s impact on battery discharge rate for electric vehicles,” in2019 IEEE Intelligent Transportation Systems Conference (ITSC), Oct 2019, pp. 474–479.
L. Wang, “Research on distributed parallel dimensionality reduction algorithm based on pca algorithm,” in2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference(ITNEC), March 2019, pp. 1363–1367.
M. Lamrini and M. Y. Chkouri, “Decomposition and visualization ofhigh-dimensional data in a two dimensional interface,” in2019 1stInternational Conference on Smart Systems and Data Science (ICSSD),Oct 2019, pp. 1–5.
J. Liu, X. Kong, F. Xia, X. Bai, L. Wang, Q. Qing, and I. Lee, “Artificial intelligence in the 21st century,” IEEE Access, vol. 6, pp. 34 403–34 421,2018.
O. Kotlyar, M. Pankratova, M. Kamalian, A. Vasylchenkova, J. E.Prilepsky, and S. K. Turitsyn, “Unsupervised and supervised machine learning for performance improvement of nft optical transmission,”in2018 IEEE British and Irish Conference on Optics and Photonics (BICOP), Dec 2018, pp. 1–4.
C. Jiang, H. Zhang, Y. Ren, Z. Han, K. Chen, and L. Hanzo, “Ma-chine learning paradigms for next-generation wireless networks,”IEEEWireless Communications, vol. 24, no. 2, pp. 98–105, April 2017.
C. Kaplanis, M. Shanahan, and C. Clopath, “Continual reinforcement learning with complex synapses,”arXiv preprint arXiv:1802.07239,2018.
M. Zhai, L. Chen, F. Tung, J. He, M. Nawhal, and G. Mori, “Lifelonggan: Continual learning for conditional image generation,” in2019IEEE/CVF International Conference on Computer Vision (ICCV), Oct2019, pp. 2759–2768.
D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continuallearning,” in Advances in Neural Information Processing Systems, 2017,pp. 6467–6476.
M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro,“Learning to learn without forgetting by maximizing transfer and minimizing interference,”arXiv preprint arXiv:1810.11910, pp. 1–31,2019.
D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experi-ence replay for continual learning,” in Advances in Neural Information Processing Systems, 2019, pp. 348–358.
J.Vanschoren,“Meta-learning:Asurvey,”arXivpreprintarXiv:1810.03548, 2018.
B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman,“Building machines that learn and think like people,” Behavioral and brain sciences, vol. 40, 2017.
P. Brazdil, C. G. Carrier, C. Soares, and R. Vilalta, Metalearning: Applications to data mining. Springer Science & Business Media,2008.
S. J. Stolfo, A. L. Prodromidis, S. Tselepis, W. Lee, D. W. Fan, and P. K.Chan, “Jam: Java agents for meta-learning over distributed databases.”inKDD, vol. 97, 1997, pp. 74–81.
C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in International Conference on Machine Learning. PMLR, 2017, pp. 1126–1135.
Y. Freund and R. E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,” Journal of computer and system sciences, vol. 55, no. 1, pp. 119–139, 1997.
L. Breiman, “Bagging predictors,”Machine learning, vol. 24, no. 2, pp.123–140, 1996.
D. H. Wolpert, “Stacked generalization,”Neural networks, vol. 5, no. 2,pp. 241–259, 1992.
L. Breiman, “Random forests,”Machine learning, vol. 45, no. 1, pp.5–32, 2001.
E. Bauer and R. Kohavi, “An empirical comparison of voting classifi-cation algorithms: Bagging, boosting, and variants,” Machine learning, vol. 36, no. 1, pp. 105–139, 1999.
V. Losing, B. Hammer, and H. Wersing, “Incremental on-line learning: Are view and comparison of state of the art algorithms,” Neurocomputing, vol. 275, pp. 1261–1274, 2018.
M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau,T. Schaul, B. Shillingford, and N. De Freitas, “Learning to learn by gradient descent by gradient descent,” in Advances in neural information processing systems, 2016, pp. 3981–3989.
D. Ha, A. Dai, and Q. V. Le, “Hypernetworks,”arXiv preprintarXiv:1609.09106, 2016.
L. Fei-Fei, R. Fergus, and P. Perona, “One-shot learning of object cate-gories,” IEEE transactions on pattern analysis and machine intelligence, vol. 28, no. 4, pp. 594–611, 2006.
A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap,“Meta-learning with memory-augmented neural networks,” in Interna-tional conference on machine learning. PMLR, 2016, pp. 1842–1850.
O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstraet al., “Matching net-works for one shot learning,” Advances in neural information processing systems, vol. 29, pp. 3630–3638, 2016.
Z. Li, F. Zhou, F. Chen, and H. Li, “Meta-sgd: Learning to learn quicklyfor few-shot learning,”arXiv preprint arXiv:1707.09835, 2017.
F. Zhou, B. Wu, and Z. Li, “Deep meta-learning: Learning to learn in the concept space,”arXiv preprint arXiv:1802.03596, 2018.
B. Hariharan and R. Girshick, “Low-shot visual recognition by shrink in gand hallucinating features,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 3018–3027.
A. Nichol and J. Schulman, “Reptile: a scalable meta learning algorithm,”arXiv preprint arXiv:1803.02999, vol. 2, p. 2, 2018.
Y. Tian and P. J. Gorinski, “Improving end-to-end speech-to-intent classification with reptile,”arXiv preprint arXiv:2008.01994, 2020.
C. Tian, X. Zhu, Z. Hu, and J. Ma, “A transfer approach with attention reptile method and long-term generation mechanism for few-shot traffic prediction,” Neurocomputing, vol. 452, pp. 15–27, 2021.
R. Singh, V. Bharti, V. Purohit, A. Kumar, A. K. Singh, and S. K. Singh, “Metamed: Few-shot medical image classification using gradient-based meta-learning,” Pattern Recognition, p. 108111, 2021.
X. Liu, X. Zhang, W. Peng, W. Zhou, and W. Yao, “A novel meta-learning initialization method for physics-informed neural networks,”arXiv preprint arXiv:2107.10991, 2021.
A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,”arXiv preprint arXiv:1803.02999, 2018.
R. Zhang, B. Xia, B. Li, L. Cao, Y. Lai, W. Zheng, H. Wang, andW. Wang, “State of the art of lithium-ion battery soc estimation for electrical vehicles,” Energies, vol. 11, no. 7, p. 1820, 2018.
I. Baccouche, S. Jemmali, A. Mlayah, B. Manai, and N. E. B. Amara, “Implementation of an improved coulomb-counting algorithm based on a piecewise soc-ocv relationship for soc estimation of li-ion battery,” arXivpreprint arXiv:1803.10654, 2018.
Z. Chen, S. Qiu, M. A. Masrur, and Y. L. Murphey, “Battery state of charge estimation based on a combined model of extended kalman filter and neural networks,” in The 2011 International Joint Conference on Neural Networks. IEEE, 2011, pp. 2156–2163.
W. Wang, X. Wang, C. Xiang, C. Wei, and Y. Zhao, “Unscented Kalman filter-based battery soc estimation and peak power prediction method for power distribution of hybrid electric vehicles,” IEEE Access, vol. 6, pp.35 957–35 965, 2018.
H. Tian and B. Ouyang, “Estimation of ev battery soc based on kf dynamic neural network with ga,” in2018 Chinese Control And Decision Conference (CCDC), June 2018, pp. 2720–2724.
A. Nuhic, T. Terzimehic, T. Soczka-Guth, M. Buchholz, and K. Di-etmayer, “Health diagnosis and remaining useful life prognostics of lithium-ion batteries using data-driven methods,” Journal of power sources, vol. 239, pp. 680–688, 2013.
K. Liu, X. Hu, Z. Wei, Y. Li, and Y. Jiang, “Modified gaussian processregression models for cyclic capacity prediction of lithium-ion batteries,” IEEE Transactions on Transportation Electrification, vol. 5, no. 4, pp.1225–1236, 2019.
S. Song, Z. Wei, H. Xia, M. Cen, and C. Cai, “State-of-charge (soc)estimation using t-s fuzzy neural network for lithium iron phosphate battery,” in2018 26th International Conference on Systems Engineering (ICSEng), Dec 2018, pp. 1–5.
A. Manthopoulos and X. Wang, “A review and comparison of lithium-ion battery soc estimation methods for electric vehicles,” in IECON 2020The 46th Annual Conference of the IEEE Industrial Electronics Society. IEEE, 2020, pp. 2385–2392.
V. Chandran, C. K Patil, A. Karthick, D. Ganeshaperumal, R. Rahim,and A. Ghosh, “State of charge estimation of lithium-ion battery for electric vehicles using machine learning algorithms,” World ElectricVehicle Journal, vol. 12, no. 1, p. 38, 2021.
P. Venugopal and S. Reka S, “State of charge estimation of lithium batteries in electric vehicles using indrnn,”IETE Journal of Research, pp. 1–11, 2021.
J. P. Ortiz, J. D. Valladolid, C. L. Garcia, G. Novillo, and F. Berrezueta, “Analysis of machine learning techniques for the intelligent diagnosis of ni-mh battery cells,” in2018 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC). IEEE, 2018, pp. 1–6.
Z. Lou, J. Tuo, and Y. Wang, “Two-step principal component analysis for dynamic processes,” in2017 6th International Symposium on Advanced Control of Industrial Processes (AdCONIP), May 2017, pp. 73–77.
K. Vatanparvar, S. Faezi, I. Burago, M. Levorato, and M. A. Al Faruque,“Extended range electric vehicle with driving behavior estimation in energy management,” IEEE transactions on Smart Grid, vol. 10, no. 3,pp. 2959–2968, 2018.
G. Gruosso, G. Storti Gajani, F. Ruiz, J. D. Valladolid, and D. Patino, “A virtual sensor for electric vehicles’ state of charge estimation,” Electronics, vol. 9, no. 2, p. 278, 2020.
E. Redondo-Iglesias, P. Venet, and S. Pelissier, “Modelling lithium-ion battery ageing in electric vehicle applications—calendar and cycling ageing combination effects,” Batteries, vol. 6, no. 1, p. 14, 2020.
X. Qin, M. Gao, Z. He, and Y. Liu, “State of charge estimation for lithium-ion batteries based on narx neural network and ukf,” in2019IEEE 17th International Conference on Industrial Informatics (INDIN),vol. 1, July 2019, pp. 1706–1711.
L. Wang, L. Wang, W. Liu, and Y. Zhang, “Research on fault diagnosis system of electric vehicle power battery based on obd technology,”in2017 International Conference on Circuits, Devices and Systems(ICCDS), Sep. 2017, pp. 95–99.
K. Khorsravinia, M. K. Hassan, R. Z. A. Rahman, and S. A. R. Al-Haddad, “Integrated obd-ii and mobile application for electric vehicle (ev) monitoring system,” in2017 IEEE 2nd International Conference on Automatic Control and Intelligent Systems (I2CACIS), Oct 2017, pp.202–206.
C. Sun, M. Pan, Y. Wang, J. Liu, H. Huang, and L. Sun, “Method for electric vehicle charging port recognition in complicated environment based on cnn,” in2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), 2018, pp. 597–602
