Big data Analysis and Dimensionality Reduction for Predict Price Trends in the Brazilian Electricity Market Considering Interdisciplinary Phenomena
Keywords:
Mutual information, big data, databased coupled in time, price, energy market trendAbstract
This paper aims to reduce the dimensionality of the features in the Brazilian electricity market when using the price trend forecasting model and thus minimize the cost of the model's complexity and the cost of computational implementation. This reduction maintains the pertinence and relevance of the data and both quantitative and qualitative information. Dimensionality reduction uses Spearman Correlation, Mutual Information, and Principal Component Analysis for features coupled with time. The pertinence and relevance of the features are guaranteed by expert analysis and validated by evaluating the data selected in the analysis period with out-of-sample verification. Analysis of features associated in time and with unique characteristics (environmental, economic, social) for application in the market of a non-storable commodity has reduced from 4000 variables to 19 variables and the nature of 13 features to 6 features. This paper presents a thorough analysis of the dimensionality of features with characteristics coupled in time and mathematically modeled for the application of mathematical and statistical techniques. The dimensionality reduction of Big Data and the method applied are well known, but the original aspect lies in the analysis of features coupled in time with a dynamic evaluation. Another original aspect is the analysis and modeling of features with very different characteristics and dimensions, as this paper analyzes Hydrological, Energetic, Climatic, Economic, and Geopolitical features. The features and database were got from 2015 to 2023 and are only relevant to the Brazilian electricity market scenario. However, the method can apply to other economic sectors and databases based on variation over time.
Downloads
References
M. Ben Tahar, S. Ben Slimane, and M. Ali Houfi, “Commodity prices and economic growth in commodity-dependent countries: New evidence from nonlinear and asymmetric analysis”, Resources Policy, vol. 72, p. 102043, aug. 2021, doi: 10.1016/J.RESOURPOL.2021.102043.
F. Zhou, L. Page, R. K. Perrons, Z. Zheng, and S. Washington, “Long-term forecasts for energy commodities price: What the experts think”, Energy Econ, vol. 84, p. 104484, oct. 2019, doi: 10.1016/J.ENECO.2019.104484.
L. G. T. Carpio, “Cointegration to estimate long-term electricity prices in periods of rationing: The case of the brazilian hydrothermal system”, Electric Power Systems Research, vol. 121, p. 351–356, apr. 2015, doi: 10.1016/J.EPSR.2014.11.022.
F. Ziel and R. Weron, “Day-ahead electricity price forecasting with high-dimensional structures: Univariate vs. multivariate modeling frameworks”, Energy Econ, vol. 70, p. 396–420, feb. 2018, doi: 10.1016/J.ENECO.2017.12.016.
F. Ziel and R. Steinert, “Probabilistic mid- and long-term electricity price forecasting”, Ren and Sust Energy Reviews, vol. 94, p. 251–266, oct. 2018, doi: 10.1016/J.RSER.2018.05.038.
E. O. Jåstad, I. M. Trotter, and T. F. Bolkesjø, “Long term power prices and renewable energy market values in Norway – A probabilistic approach”, Energy Econ, vol. 112, p. 106182, aug. 2022, doi: 10.1016/J.ENECO.2022.106182.
R. Huisman, “The influence of temperature on spike probability in day-ahead power prices”, Energy Econ, vol. 30, no 5, p. 2697–2704, sept. 2008, doi: 10.1016/J.ENECO.2008.05.007.
A. Livas-García, O. May Tzuc, E. Cruz May, R. Tariq, M. Jimenez Torres, and A. Bassam, “Forecasting of locational marginal price components with artificial intelligence and sensitivity analysis: A study under tropical weather and renewable power for the Mexican Southeast”, Electric Power Systems Research, vol. 206, p. 107793, may 2022, doi: 10.1016/J.EPSR.2022.107793.
H. Lu, X. Ma, M. Ma, e S. Zhu, “Energy price prediction using data-driven models: A decade review”, Comput Sci Rev, vol. 39, p. 100356, fev. 2021, doi: 10.1016/J.COSREV.2020.100356.
R. Weron, “Electricity price forecasting: A review of the state-of-the-art with a look into the future”, Int J Forecast, vol. 30, no 4, p. 1030–1081, oct. 2014, doi: 10.1016/J.IJFORECAST.2014.08.008.
J. Nowotarski, J. Tomczyk, and R. Weron, “Robust estimation and forecasting of the long-term seasonal component of electricity spot prices”, Energy Econ, vol. 39, p. 13–27, sept. 2013, doi: 10.1016/J.ENECO.2013.04.004.
J. N. P. Swisher, G. M. Jannuzzi, and R. Y. Redlinger, Tools and Methods for Integrated Resource Planning: Improving Energy Efficiency and Protecting Environment. Riso National Lab, Denmark: UNEP Collaborating Centre on Energy and Environment, 1997.
J. C. Piai, R. D. M. Gomes, and G. D. M. Jannuzzi, “Integrated resources planning as a tool to address energy poverty in Brazil”, Energy Build, vol. 214, p. 109817, may 2020, doi: 10.1016/J.ENBUILD.2020.109817.
G. M. Jannuzzi, J. N. P. Swisher, and R. Y. Redlinger, Planejamento Integrado de Recursos Energéticos: Oferta, Demanda e suas Interfaces, 2a ed. Campinas: IEI, 2018. [Online]. Available: https://iei-brasil.org/livro-pir/
E. T. Sica, “Integrated planning of water resources for electricity production: a decision support system by multicriteria and system dynamics”, Thesis, Federal University of Santa Catarina, Florianópolis, 2009. [Online]. Available: http://repositorio.ufsc.br/xmlui/handle/123456789/92994
ONS, “Map of the Interconnected Power Transmission and Production System - Horizon 2027”, National Electrical System Operator. [Online]. Available: https://www.ons.org.br/paginas/sobre-o-sin/mapas
C. Kath and F. Ziel, “The value of forecasts: Quantifying the economic gains of accurate quarter-hourly electricity price forecasts”, Energy Econ, vol. 76, p. 411–423, oct. 2018, doi: 10.1016/J.ENECO.2018.10.005.
T. Serafin, G. Marcjasz, and R. Weron, “Trading on short-term path forecasts of intraday electricity prices”, Energy Econ, vol. 112, p. 106125, ago. 2022, doi: 10.1016/J.ENECO.2022.106125.
ONS, “General Data: operation results”, National Electrical System Operator. [Online]. Available: https://www.ons.org.br/paginas/resultados-da-operacao/historico-da-operacao/dados-gerais
AMPERE, “Ampere Consultoria”, Energy planning, development and meteorology. [Online]. Available: https://ampereconsultoria.com.br/
BC, “Selected Economic Indicators”, Brazilian Central Bank. [Online]. Available: https://www.bcb.gov.br/en/statistics/selectedindicators
J. Wu, N. Levi, R. Araujo, e Y. G. Wang, “An evaluation of the impact of COVID-19 lockdowns on electricity demand”, Electric Power Systems Research, vol. 216, p. 109015, mar. 2023, doi: 10.1016/J.EPSR.2022.109015.
M. Carvalho, D. Bandeira de Mello Delgado, K. M. de Lima, M. de Camargo Cancela, C. A. dos Siqueira, and D. L. B. de Souza, “Effects of the COVID-19 pandemic on the Brazilian electricity consumption patterns”, Int J Energy Res, vol. 45, no 2, p. 3358–3364, feb. 2021, doi: 10.1002/ER.5877.
V. Mahler, R. Girard, and G. Kariniotakis, “Data-driven structural modeling of electricity price dynamics”, Energy Econ, vol. 107, p. 105811, mar. 2022, doi: 10.1016/J.ENECO.2022.105811.
L. Mitridati e P. Pinson, “A Bayesian Inference Approach to Unveil Supply Curves in Electricity Markets”, IEEE Transactions on Power Systems, vol. 33, no 3, p. 2610–2620, 2018, doi: 10.1109/TPWRS.2017.2757980.
C. Ruiz, A. J. Conejo, and D. J. Bertsimas, “Revealing Rival Marginal Offer Prices Via Inverse Optimization”, IEEE Transactions on Power Systems, vol. 28, no 3, p. 3056–3064, 2013, doi: 10.1109/TPWRS.2012.2234144.
R. Chen, I. Ch. Paschalidis, M. C. Caramanis, and P. Andrianesis, “Learning From Past Bids to Participate Strategically in Day-Ahead Electricity Markets”, IEEE Trans Smart Grid, vol. 10, no 5, p. 5794–5806, 2019, doi: 10.1109/TSG.2019.2891747.
F. Saâdaoui, S. Mefteh-Wali, e S. Ben Jabeur, “Multiresolutional statistical machine learning for testing interdependence of power markets: A Variational Mode Decomposition-based approach”, Expert Syst Appl, vol. 208, p. 118161, dec. 2022, doi: 10.1016/J.ESWA.2022.118161.
EPU, “Geopolitical Risk Index”, Economic Policy Uncertainty Index. [Online]. Available: https://www.policyuncertainty.com/gpr.html
S. Awaworyi-Churchill, J. Inekwe, K. Ivanovski, and R. Smyth, “Breaks, trends and correlations in commodity prices in the very long-run”, Energy Econ, vol. 108, p. 105933, apr. 2022, doi: 10.1016/J.ENECO.2022.105933.
F. Saâdaoui and S. Ben Jabeur, “Analyzing the influence of geopolitical risks on European power prices using a multiresolution causal neural network”, Energy Econ, vol. 124, p. 106793, aug. 2023, doi: 10.1016/J.ENECO.2023.106793.
O. A. Adeosun, M. I. Tabash, X. V. Vo, and S. Anagreh, “Uncertainty measures and inflation dynamics in selected global players: a wavelet approach”, Qual Quant, vol. 57, no 4, p. 3389–3424, aug. 2023, doi: 10.1007/S11135-022-01513-7/TABLES/10.
N. AlNuaimi, M. M. Masud, M. A. Serhani, and N. Zaki, “Streaming feature selection algorithms for big data: A survey”, Applied Computing and Informatics, vol. 18, no 1–2, p. 113–135, jan. 2022, doi: 10.1016/J.ACI.2019.01.001/FULL/PDF.
M. Rong, D. Gong, and X. Gao, “Feature Selection and Its Use in Big Data: Challenges, Methods, and Trends”, IEEE Access, vol. 7, p. 19709–19725, 2019, doi: 10.1109/ACCESS.2019.2894366.
T. Verdonck, B. Baesens, M. Óskarsdóttir, and S. vanden Broucke, “Special issue on feature engineering editorial”, Mach Learn, vol. 113, no 7, p. 3917–3928, jul. 2021, doi: 10.1007/S10994-021-06042-2/METRICS.
L. Santos Coelho, H. V. Hultmann Ayala, and V. Cocco Mariani, “CO and NOx emissions prediction in gas turbine using a novel modeling pipeline based on the combination of deep forest regressor and feature engineering”, Fuel, vol. 355, p. 129366, jan. 2024, doi: 10.1016/J.FUEL.2023.129366.
Z. Wang, L. Xia, H. Yuan, R. S. Srinivasan, and X. Song, “Principles, research status, and prospects of feature engineering for data-driven building energy prediction: A comprehensive review”, Journal of Building Engineering, vol. 58, p. 105028, out. 2022, doi: 10.1016/J.JOBE.2022.105028.
J. Wang, Y. Dong, e J. Liu, “A novel multifactor clustering integration paradigm based on two-stage feature engineering and improved bidirectional deep neural networks for exchange rate forecasting”, Digit Signal Process, vol. 143, p. 104258, nov. 2023, doi: 10.1016/J.DSP.2023.104258.
G. Yan et al., “A comparative study of machine learning models for respiration rate prediction in dairy cows: Exploring algorithms, feature engineering, and model interpretation”, Biosyst Eng, vol. 239, p. 207–230, mar. 2024, doi: 10.1016/J.BIOSYSTEMSENG.2024.01.010.
Y. Q. Wang et al., “Machine learning framework for intelligent aeration control in wastewater treatment plants: Automatic feature engineering based on variation sliding layer”, Water Res, vol. 246, p. 120676, nov. 2023, doi: 10.1016/J.WATRES.2023.120676.
D. Dai et al., “Using machine learning and feature engineering to characterize limited material datasets of high-entropy alloys”, Comput Mater Sci, vol. 175, p. 109618, apr. 2020, doi: 10.1016/J.COMMATSCI.2020.109618.
L. Jiang and G. Hu, “Day-Ahead Price Forecasting for Electricity Market using Long-Short Term Memory Recurrent Neural Network”, em 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), 2018, p. 949–954. doi: 10.1109/ICARCV.2018.8581235.
Y. Sun, F. Haghighat, and B. C. M. Fung, “A review of the-state-of-the-art in data-driven approaches for building energy prediction”, Energy Build, vol. 221, p. 110022, aug. 2020, doi: 10.1016/J.ENBUILD.2020.110022.
N. Kim, H. Shin, and K. Lee, “Feature engineering process on well log data for machine learning-based SAGD performance prediction”, Geoenergy Science and Engineering, vol. 229, p. 212057, out. 2023, doi: 10.1016/J.GEOEN.2023.212057.
Y. Li et al., “A county-level soybean yield prediction framework coupled with XGBoost and multidimensional feature engineering”, International Journal of Applied Earth Observation and Geoinformation, vol. 118, p. 103269, apr. 2023, doi: 10.1016/J.JAG.2023.103269.
O. Björneld, M. Carlsson, e W. Löwe, “Case study - Feature engineering inspired by domain experts on real world medical data”, Intell Based Med, vol. 8, p. 100110, jan. 2023, doi: 10.1016/J.IBMED.2023.100110.
J. Zheng, J. Zhu, and H. Xi, “Short-term energy consumption prediction of electric vehicle charging station using attentional feature engineering and multi-sequence stacked Gated Recurrent Unit”, Computers and Elect Eng, vol. 108, p. 108694, may 2023, doi: 10.1016/J.COMPELECENG.2023.108694.
D. Elavarasan, P. M. Durai Raj Vincent, K. Srinivasan, and C. Y. Chang, “A Hybrid CFS Filter and RF-RFE Wrapper-Based Feature Extraction for Enhanced Agricultural Crop Yield Prediction Modeling”, Agriculture 2020, Vol. 10, Page 400, vol. 10, no 9, p. 400, set. 2020, doi: 10.3390/AGRICULTURE10090400.
M. Zulfiqar, M. Kamran, M. B. Rasheed, T. Alquthami, and A. H. Milyani, “A hybrid framework for short term load forecasting with a navel feature engineering and adaptive grasshopper optimization in smart grid”, Appl Energy, vol. 338, p. 120829, may 2023, doi: 10.1016/J.APENERGY.2023.120829.
A. Forootani, M. Rastegar, and A. Sami, “Short-term individual residential load forecasting using an enhanced machine learning-based approach based on a feature engineering framework”, Electric Power Systems Research, vol. 210, p. 108119, sept. 2022, doi: 10.1016/J.EPSR.2022.108119.
H. Ebrahimian, S. Barmayoon, M. Mohammadi, and N. Ghadimi, “The price prediction for the energy market based on a new method”, Economic Research, vol. 31, no 1, p. 313–337, jan. 2018, doi: 10.1080/1331677X.2018.1429291.
W. Gao and H. Pan, “Multi-Label Feature Selection Based on Min-Relevance Label”, IEEE Access, vol. 11, p. 410–420, 2023, doi: 10.1109/ACCESS.2022.3231871.
O. Abedinia, N. Amjady, and H. Zareipour, “A New Feature Selection Technique for Load and Price Forecast of Electrical Power Systems”, IEEE Transactions on Power Systems, vol. 32, no 1, p. 62–74, 2017, doi: 10.1109/TPWRS.2016.2556620.
N. Amjady and F. Keynia, “Day-Ahead Price Forecasting of Electricity Markets by Mutual Information Technique and Cascaded Neuro-Evolutionary Algorithm”, IEEE Transactions on Power Systems, vol. 24, no 1, p. 306–318, 2009, doi: 10.1109/TPWRS.2008.2006997.
C. Zhang, L. Cao, e A. Romagnoli, “On the feature engineering of building energy data mining”, Sustain Cities Soc, vol. 39, p. 508–518, may 2018, doi: 10.1016/J.SCS.2018.02.016.
D. Amengual, E. Sentana, and Z. Tian, “Gaussian Rank Correlation and Regression”, vol. 43B, p. 269–306, jan. 2022, doi: 10.1108/S0731-90532021000043B012.
B. Everitt and A. Skrondal, The Cambridge dictionary of statistics, 4th ed. Cambridge, UK ; New York: Cambridge University Press, 2010.
S. Li, P. Wang, and L. Goel, “A Novel Wavelet-Based Ensemble Method for Short-Term Load Forecasting with Hybrid Neural Networks and Feature Selection”, IEEE Transactions on Power Systems, vol. 31, no 3, p. 1788–1798, 2016, doi: 10.1109/TPWRS.2015.2438322.
