Team Modeling with Deep Behavioral Cloning for the RoboCup 2D Soccer Simulation League
Keywords:
Team Modeling, Deep Learning, Imitation Learning, RoboCup, Robot SoccerAbstract
Soccer is still considered an open problem by the AI community due to its complex stochastic real-time multiagent nature. The RoboCup Soccer Simulation 2D League has been used as a testbed for new ideas and techniques for many subjects, including team modeling. However, even though team modeling has been an indispensable part of the best league participants to date, in practice it typically consists of ad-hoc heuristics encoded as rules. This requires time-consuming manual work, does not scale to multiple teams, and does not work well at unaccounted scenarios. This paper presents a data-driven method for modeling teams by training Deep Neural Networks with large amounts of data with an Imitation Learning formulation. We demonstrate the approach by training a deep model of the Japanese team Helios using 57,578,668 state-action pairs of players. The resulting model achieved 84.5% accuracy on action selection and a small error on regression of the action parameters. The network is shown to be an effective movement predictor of Helios field players and have negligible degradation when Helios is evaluated against adversaries not seen at training time.
Downloads
References
H. Kitano, M. Asada, Y. Kuniyoshi, I. Noda, and E. Osawa, “Robocup: the robot world cup initiative,” Proceedings of the International Con- ference on Autonomous Agents, 04 1998.
S. V. Albrecht and P. Stone, “Autonomous agents modelling other agents: A comprehensive survey and open problems,” Artificial Intelligence, vol. 258, pp. 66–95, 2018.
S. Pourmehr and C. Dadkhah, “An overview on opponent modeling in robocup soccer simulation 2d,” in RoboCup 2011: Robot Soccer World Cup XV (T. Röfer, N. M. Mayer, J. Savage, and U. Saranlı, eds.), (Berlin, Heidelberg), pp. 402–414, Springer Berlin Heidelberg, 2012.
W. Plant and G. Schaefer, “An overview of neural networks in simula- tion soccer,” 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC), pp. 1620–1625, 2009.
P. Riley and M. Veloso, On Behavior Classification in Adversarial Environments, pp. 371–380. Tokyo: Springer Japan, 2000.
A. Ledezma, R. Aler, A. Sanchis, and D. Borrajo, “Predicting opponent actions by observation,” in RoboCup 2004: Robot Soccer World Cup VIII (D. Nardi, M. Riedmiller, C. Sammut, and J. Santos-Victor, eds.), (Berlin, Heidelberg), pp. 286–296, Springer Berlin Heidelberg, 2005.
A. Ledezma, R. Aler, A. Sanchis, and D. Borrajo, “Ombo: An opponent modeling approach,” Ai Communications, vol. 22, no. 1, pp. 21–35, 2009.
M. Ahmadi, A. K. Lamjiri, M. M. Nevisi, J. Habibi, and K. Badie, “Using a two-layered case-based reasoning for prediction in soccer coach,” in Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications. MLMTA’03, June 23 - 26, 2003, Las Vegas, Nevada, USA (H. R. Arabnia and E. B. Kozerenko, eds.), (Las Vegas, Nevada, USA), pp. 181–185, CSREA Press, 2003.
T. Steffens, “Adapting similarity-measures to agenttypes in opponent- modelling,” in Workshop on Modeling Other Agents from Observations at AAMAS, (1730 Massachusetts Ave., NW Washington, DCUnited States), pp. 125–128, IEEE Computer Society, 2004.
T. Steffens, “Similarity-based opponent modelling using imperfect do- main theories.,” CIG, vol. 5, pp. 285–291, 2005.
M. W. Floyd, B. Esfandiari, and K. Lam, “A case-based reasoning approach to imitating robocup players,” in Proceedings of the Twenty- First International Florida Artificial Intelligence Research Society Con- ference, May 15-17, 2008, Coconut Grove, Florida, USA (D. Wilson and H. C. Lane, eds.), (Coconut Grove, Florida, USA), pp. 251–256, AAAI Press, 2008.
T. Fukushima, T. Nakashima, and H. Akiyama, “Mimicking an expert team through the learning of evaluation functions from action se- quences,” in RoboCup 2018: Robot World Cup XXII (D. Holz, K. Genter, M. Saad, and O. von Stryk, eds.), (Cham), pp. 170–180, Springer International Publishing, 2019.
T. Fukushima, T. Nakashima, and H. Akiyama, “Evaluation-function modeling with multi-layered perceptron for robocup soccer 2d simula- tion,” Artificial Life and Robotics, vol. 25, no. 3, pp. 440–445, 2020.
L. C. Melo, M. R. Maximo, and A. M. da Cunha, Imitation learning and meta-learning for optimizing humanoid robot motions. PhD thesis, Master’s Thesis, Instituto Tecnológico de Aeronáutica, Sao José dos Campos ..., 2019.
R. Aler, J. M. Valls, D. Camacho, and A. Lopez, “Programming robosoccer agents by modeling human behavior,” Expert Systems with Applications, vol. 36, no. 2, Part 1, pp. 1850–1859, 2009.
O.As ̧ık,B.Görer,andH.L.Akın,“End-to-enddeepimitationlearning: Robot soccer case study,” in RoboCup 2018: Robot World Cup XXII (D. Holz, K. Genter, M. Saad, and O. von Stryk, eds.), (Cham), pp. 137– 149, Springer International Publishing, 2019.
S.Raza, S.Haider, and M.-A.Williams,“Teaching coordinated strategies to soccer robots via imitation,” in 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 1434–1439, 2012.
T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters, “An algorithmic perspective on imitation learning,” arXiv preprint arXiv:1811.06711, 2018.
A. Hussein, E. Elyan, and C. Jayne, “Deep imitation learning with memory for robocup soccer simulation,” in Engineering Applications of Neural Networks (E. Pimenidis and C. Jayne, eds.), (Cham), pp. 31–43, Springer International Publishing, 2018.
A. Hussein, Deep learning based approaches for imitation learning. PhD thesis, Robert Gordon University, 2018. COMPLETED.
Q. D. Nguyen and M. Prokopenko, “Structure-preserving imitation learning with delayed reward: An evaluation within the robocup soccer 2d simulation environment,” Frontiers in Robotics and AI, vol. 7, 2020.
T. Nakashima and H. Ishibuchi, “Mimicking dribble trajectories by neural networks for robocup soccer simulation,” in 2007 IEEE 22nd International Symposium on Intelligent Control, (Singapore), pp. 658– 663, IEEE, IEEE, 11 2007.
F. V. Coimbra, “Opponent Modeling by Imitation in Robot Soccer,” Bachelor’s thesis, Instituto Tecnológico de Aeronáutica, São José dos Campos, November 2021.
F. V. Coimbra, “rcg2csv, an RCG parser based on librcsc-4.1.0.” https: //github.com/FelipeCoimbra/librcsc, 2021.
F. V. Coimbra, “rcl2csv, an RCL parser for the RoboCup Soccer Simu- lation 2D League.” https://gitlab.com/FelipeCoimbra/rcl2csv/, 2021.
F. V. Coimbra, “RCSS2D Deep Imitation.” https://gitlab.com/itandroids/ open- projects/rcss2d- opp- imitation, 2021.
H. Akiyama, T. Nakashima, T. Fukushima, Y. Suzuki, and A. Ohori, “Helios2019: Team description paper,” 2019.
G. Haixiang, L. Yijing, J. Shang, G. Mingyun, H. Yuanyue, and G. Bing, “Learning from class-imbalanced data: Review of methods and applications,” Expert Systems with Applications, vol. 73, pp. 220–239, 2017.
