Using Generative Pre-Trained Transformers (GPT) for Electricity Price Trend Forecasting in the Spanish Market

1. Introduction

The forecast of the medium to long-term price trend of the electricity market is subject to considerable uncertainty due to the influence of multiple complex factors - geopolitical events, climatic phenomena, social developments, regulations, technical factors, economic cycles, etc. [1].

Analyzing the evolution of the Spanish electricity market (OMIE) from 2018 to 2023 [2] unravels key factors behind heightened volatility. Understanding the intricate evolution of energy prices between 2018 and 2023 (Figure 1), sheds light on the multifaceted factors that influenced this journey. According to the International Energy Agency [3], during the initial years, from 2018 to 2021, energy prices were shaped by a confluence of variables, including rising costs across various fuels and technologies, supply chain pressures, labor market constraints, and fluctuations in critical mineral supplies and construction materials. Notably, clean energy costs, which had previously witnessed steady declines, exhibited a distinct uptrend during this period.

However, the narrative dramatically turned after 2020, with energy prices experiencing heightened volatility. As reported, the primary catalyst for this volatility was the growing disparity between surging demand and a limited pipeline of new conventional projects within the oil industry. This imbalance introduced a significant risk of price spikes, casting shadows on the global economy's stability. Moreover, competitive electricity markets grappled with a widening gap between revenue from electricity sales and total generation costs, further contributing to price fluctuations.

The COVID-19 pandemic, a global disruptor, played its part by dampening overall energy demand, particularly in the case of carbon-intensive fuels like coal and oil. Conversely, renewable energy sources proved more resilient in the face of the pandemic's impacts. CO2 emissions were notably reduced, and the energy sector witnessed a decline in capital investment, primarily affecting oil and natural gas supply projects. These repercussions are expected to reverberate through energy markets in the years to come.

Furthermore, the world's current energy crisis, initiated by Russia's invasion of Ukraine, has added another layer of complexity to the energy landscape. High energy prices, particularly in the natural gas sector, have led to wealth transfers from consumers to producers, affecting electricity generation costs globally. The crisis has also posed challenges in ensuring access to modern energy for many, with lingering uncertainties about its duration and fossil fuel price trends. While short-term shifts have seen increased demand for oil and coal as alternatives to costly gas, the long-term trajectory points toward low-emissions sources such as renewables, nuclear energy, and heightened efficiency measures.

Previous time series forecasting approach models used to determine energy prices prior to the COVID-19 pandemic [4,5,6] faced significant challenges during and after the crisis. They struggled to adapt to the abrupt disruptions in energy demand, increased price volatility, supply chain pressures, and changes in the energy mix brought about by the pandemic. Additionally, uncertainties in the economic and policy landscapes further hindered their accuracy. The pandemic exposed the limitations of these forecasting models in coping with such unforeseen disruptions, emphasizing the need for more adaptable and robust forecasting approaches to navigate the evolving energy market dynamics effectively.

Scientific evidence suggests that having prior information about these qualitative factors can be of great importance in understanding the possible future evolution of the market. For example, understanding what affect the consumption patterns [7,8], the demand density [9], and the electricity generation [10,11] can be critical. In fact, many market specialists publish their understanding of recent market developments and possible trends based on factors such as social phenomena affecting demand, conflicts, government agreements, natural disasters, legislative actions, nuclear plant shutdowns, weather conditions, etc., either through journals or company reports available on the web.

These reports can be processed to apply sentiment analysis techniques. Typically, sentiment analysis techniques applied to markets (stock markets, oil markets, electricity markets) capture only headlines and classify the possible price direction. Research insights suggest that including sentiment analysis results improves quantitative predictions of price prediction models. There is a growing trend towards using deep learning techniques for market sentiment analysis. Machine learning-based methods involve training a model on a dataset of text where the sentiment of each text fragment is known. The trained model can then be used to predict the sentiment of new text. Examples of machine learning techniques include Convolutional Neural Networks (CNN), Recursive Neural Networks (RNN), and Transformer models [12].

Large Language Models (LLMs) are based on Transformer architecture trained on a large volume of data (GPT), which are then fine-tuned for specific contexts [13]. Current LLMs have been trained on large and varied datasets, giving them the capability to understand the functioning of the economy and markets. To evaluate sentiment in a particular market, they need to be fine-tuned with specific new datasets derived from specialist reports and related news to derive sentiments of the specific market [14,15]. Breitung et al. [16] apply this approach to the oil market by generating a specific dataset with 1600 records.

So far, we have not found publications dealing with sentiment analysis applied to trends in electricity markets.

Advanced natural language processing methods have transformed AI interaction and Generative Pre-trained Transformers (GPT) are ground-breaking for generating human-like text and comprehending natural language [17]. GPT models can be extended and refined with specialized news and reports analysis [18], which can significantly enhance energy price prediction models, in addition to sentiment analysis, in several ways:

• Data Enrichment: GPT-based models can analyze and extract valuable insights from a vast corpus of specialized news articles and reports on the energy market. This data enrichment provides a broader context for energy price forecasting models.
• Event Detection: GPT models can detect and highlight significant events [19], such as geopolitical developments, supply disruptions, or regulatory changes, that may impact energy markets. These detected events can be used as input variables for forecasting models.
• Market News Summarization: GPT can generate concise summaries of complex news articles and reports [20] making it easier for analysts and traders to stay informed about market developments. These summaries can serve as valuable inputs for forecasting models.
• Identifying Influential Factors: GPT can identify, and rank factors mentioned in the news and reports likely to influence energy prices. This information can guide feature selection and help prioritize variables in forecasting models.
• Customized reports: In the case of OpenAI's GPT, users can provide customized prompts to extract specific information or insights from news and reports. This allows for tailored analysis based on the unique requirements of the forecasting model.

The assumption is that by incorporating GPT-based specialized news and reports analysis into energy price prediction models allows for a more comprehensive and timely understanding of market dynamics [21]. This, in turn, may improve the models' accuracy and helps energy market participants make informed decisions in an increasingly complex and dynamic environment. The application of GPT technology in predicting energy prices in the Spanish Market is the focus of our research.

In this paper, we analyze how the reasoning capabilities of large language models (LLMs) can be used to derive context-specific trend predictions from specialized news and expert reports. We construct a dataset of electricity news and reports and compare the forecasting performance of two different approaches for domain knowledge enhancement: prompting engineering and fine-tuning. Our findings indicate that LLMs can successfully leverage their reasoning capabilities to contextualize the evolution of electricity market prices.

We contribute to the literature on advanced Natural Language Processing (NLP) tools applied to management science, illustrating how LLMs can be effectively used to extract business experts' perspectives on the future and consolidate their consensus, and provide tactical guidance to drive these models towards achieving optimal results.

In what follows, we present the methods used to adjust the GPT, the implementation details (the original code and the dataset are openly available¹), the metrics that we have defined to evaluate the different dimensions of the sentiment extracted from the analysis of the texts, the results obtained, the future developments that would be interesting to address and the main conclusions of this article.

2. Materials and Methods

OpenAI's GPT models, renowned for their natural language understanding and generation capabilities, provide a robust foundation for various applications. Two primary paradigms emerge regarding integrating private data and tailoring these models to specific tasks: context learning and fine-tuning [18]. These paradigms unlock the potential to enhance model performance, adapt to domain-specific nuances, and leverage the unique characteristics of proprietary datasets, all while preserving the underlying capabilities of the pre-trained GPT model.

2.1. Paradigm 1: In-Context Learning

In-context learning, also called prompt engineering or prompt design, is a paradigm that allows users to interact with GPT models by providing context-specific instructions or queries [22]. This approach enables the customizations of model responses without directly modifying the model. By crafting tailored prompts, users can elicit responses that align with their specific requirements, making it a versatile tool for leveraging the pre-trained knowledge of GPT models while adding a layer of domain specificity. In-context learning is precious when quick, context-aware responses are needed, such as in chatbots, customer support, or generating domain-specific content. However, it relies on the skillful design of prompts and may require iterative adjustments to achieve optimal results, as understanding and controlling model behavior through prompts can be nuanced.

2.2. Paradigm 2: Fine-Tuning

Fine-tuning represents a deeper level of customization, where the pre-trained GPT model is adapted to perform specific tasks or Excel domains [23]. The model is further trained on domain-specific data, including proprietary datasets, specialized text, or even structured information in this paradigm. Fine-tuning allows the model to learn from this additional data while preserving its general language understanding capabilities. This approach results in a more refined and specialized model capable of providing nuanced and context-aware responses in a particular domain. Fine-tuned models have applications in medical diagnosis, legal document analysis, and financial forecasting, where precise and domain-specific insights are essential. However, fine-tuning requires careful data curation, domain expertise, and a thorough understanding of the trade-offs involved in specialization, as excessive fine-tuning can risk overfitting to specific datasets and limit model generality.

2.3. Implementation Details

To conduct a comprehensive comparative analysis and measure the efficacy of OpenAI technologies into time series trend forecasting, we created two sets of predictions with the in-context examples and fine-tuning approaches of OpenAI's customization, using a collection of news articles [24,25] and expert analysis reports specifically focused on the Spanish energy markets [26,27].

The OpenAI models calculated three variables for each news article content:

• Impact on Electricity Price (Scale 0-10): The first variable quantifies the perceived impact of each news article on the price of electricity within a scale ranging from 0 (no impact) to 10 (high impact). This quantification allows us to discern the potential influence of each piece of news on energy prices, a critical factor in our forecasting model.
• Direction of Impact (Up, Down, None): We evaluated whether the news articles indicated a potential price impact in the form of an increase ("Up"), a decrease ("Down"), or no discernible impact ("None"). Understanding the Direction of influence is paramount for making informed predictions in the dynamic energy market.
• Impact Period (Past, Short-term, Mid-term, Long-term, None): The third variable delves into the temporal aspect of impact, categorizing it into various periods—Past, Short-term, Mid-term, Long-term, or None. This temporal classification aids in determining when the anticipated price effects are likely to materialize, further enhancing the precision of our forecasts.

Simulating the impact of news on the energy market involves understanding market dynamics and investor behavior when they receive updates [28]. While no formula perfectly captures this, we used a simplified “exponential decay” model to represent how the impact of news might fade over time for short-term news and a “Gaussian model” for mid-long-term impacts (Figure 2). This model assumes that the impact of the news will be most significant immediately after or close to its release and will gradually diminish over time.

2.3.1. In-context Implementation

For the in-context model, we added to the instructions to calculate the mentioned variables, 8 examples of the market news and articles dataset and provided the values of Impact, Direction and Period from an energy expert point of view (Figure 3). We executed this prompt passing as parameter each of the collected news and articles in the full dataset and enrich each news record with the new information calculated by GPT (model: text-davinci-003).

Taking the “Impact” value of the news on the date of release, we calculated the average impact over the following period depending on the identified “Duration”: for short-term up to 3 weeks; for mid/long-term, up to 3 months. Then we grouped all these values calculating the average of each interval (weeks for short-term, months for mid/long term). The symbol of the value indicated if it was a positive or negative impact (increase or decrease of the energy price).

2.3.2. Fine-Tuned Implementation

To create the fine-tuned model, for each news or article, we completed a first run with OpenAI using the “In-context examples prompt” to identify its assessment of “Impact Period” for each record (short-term, mid/long-term, past, None), keeping aside 25 randomly selected records for each possible value for the fine-tuning.

In parallel, calculated for each date of the time-series range (2018-2023) what difference (%) the price has suffered with the previous time interval to the news event (weeks for short-term, months for mid/long term) as per the following formulas²:

$$\begin{gathered} {Impact}_{week} = 0.5·{∆Price}_{week t}+0.3·{∆Price}_{week t+1}+0.2·{∆Price}_{week t+2} \\ {Impact}_{month} = 0.5·{∆Price}_{month t}+0.3·{∆Price}_{ month t+1}+0.2 \\ ·{∆Price}_{month t+2} \end{gathered}$$

With this information, we populated each news sample record to be used for fine-tuning, including the “Direction” field (sign of the impact), obtaining a dataset as the one shown below (Figure 4):

From here we were ready to follow the Fine-tuning steps explained by OpenAI to create a custom model [23]:

• Dataset preparation: Every instance within the dataset should represent a conversation structured in a manner consistent with OpenAI’s Chat Completions API. This structure entails organizing the conversation as a list of messages, where each message comprises a role, content, and the possibility of including a name.
• Validate data formatting and divide training and testing datasets
• Upload dataset file and create the fine-tuning job using the OpenAI SDK.
• Use the new fine-tuned model with the rest of the news and articles to enrich the dataset with calculated variables.

After applying the new model to the rest of the news and article dataset (Figure 5), we calculated the average Impact decay based on Impact Period (short-term: weekly average; mid/long-term: monthly average).

3. Results

From the results of analyzing the news and articles, we extract information like its impact on the price, the direction of that impact, and the period it will occur.

We have grouped the insights provided by this analysis in short-term and mid/long-term, calculating the average impact for each interval and the sign indicating Direction, and we evaluate the accuracy of that prediction based on the variation of price over the following period after the news occurred.

We have defined three different types of metrics to measure the accuracy of the Direction of the Impact prediction:

• Close Price: If the Direction indicates that the Price will go UP (Figure 6), the OPEN PRICE at the beginning of the first interval when the news is published (interval t) should be LOWER than at least 1 of the CLOSE PRICE values of the current or the following two intervals (t, t+1, t+2). The intervals will be weeks for short-term and months for mid/long-term.

If the Direction indicates that the Price will go DOWN (Figure 7), the OPEN PRICE at the beginning of the first interval when the news is published (t) should be HIGHER than at least 1 of the CLOSE PRICE values of the current or the following two intervals (t, t+1, t+2).

• High/Low: If the Direction indicates that the Price will go UP (Figure 8), the OPEN PRICE at the beginning of the first interval when the news is published (interval t) should be LOWER than at least 1 of the HIGH PRICE values of the current or the following two intervals (t, t+1, t+2). The intervals will be weeks for short-term and months for mid/long-term.

  

  Figure 8. OpenAI High/Low Price Metric: Direction UP.

  Figure 8. OpenAI High/Low Price Metric: Direction UP.

  

If the Direction indicates that the Price will go DOWN (Figure 9), the OPEN PRICE at the beginning of the first interval when the news is published (t) should be HIGHER than at least 1 of the LOW PRICE values of the current or the following two intervals (t, t+1, t+2).

Threshold: Same as High or Low, but there should be a minimum difference between the Open Price value and the High or Low, depending on the Direction of 2% for short-term and 5% for mid/long term.

3.1. Short-Term Analysis

The following figures (Figure 10 and Figure 11) and Table 1 show the analysis results of short-term identified impacts on price using OpenAI and comparing both approaches: in-context examples and fine-tuning. The color of the arrows indicates OpenAI prediction to go UP (green) or DOWN (red), next to the price timeseries.

The in-context model detects 117 impact points, while the fine-tuned model only detects 31 short-term impacts.

Concerning the scores, we can see that the fine-tunned approach performs slightly better but with considerably fewer captured points.

3.2. Mid/Long-Term Analysis

The following figures (Figure 12 and Figure 13) and Table 2 show the results of the analysis of mid/long-term identified impacts on price using OpenAI and comparing both approaches: in-context examples and fine-tuning.

The in-context model detects 71 impact points, while the fine-tuned model only detects 59 mid/long-term impacts. The scores show that the fine-tuned approach is performing again better, although with slightly fewer points.

Overall, OpenAI detects the trend in the price time series, especially in the mid/long term. The in-context approach seems more sensitive, but the fine-tuned model provides slightly more accurate results.

4. Discussion

Our findings indicate the potential of GPT models to provide valuable insights and improve predictions, particularly in understanding mid-term price trends. Therefore, we conclude that continued exploration and optimization of OpenAI's capabilities are essential to unlock their full potential in energy price forecasting, and particularly electricity market price. To validate and to generalize the observed results further research is needed.

Building on the insights gained from this study it should be enhanced the contextual understanding and data sources used as input for the GPT models by incorporating additional relevant news and reports sources and increased periodicity and optimizing the accurate impact calculation and decay over time methods to translate real impact.

The key benefit of increasing LLM awareness about the market evolution by the in-context approach is that it avoids the need to re-modify LLM parameters for this specific task application. Instead, developers can append an external knowledge repository, enriching the input and thus refining the output accuracy of the model. Therefore, in-context is seen as a more practical and economical approach, with a lower barrier to entry and independent of the specific LLM model. However, for more extensive and professional applications, it is necessary to automate the workflows that allow scalability and better precision of responses. A RAG architecture [29] can be developed for this purpose.

RAG has become one of the most popular architectures in LLM systems, combining automated information retrieval mechanisms and in-context learning to bolster LLM performance. In this framework, a query initiated by a user requests the retrieval of relevant information through search algorithms. This information is then integrated into the LLM indications, providing additional context for the generation process [30].

In basis of an RAG architecture, we plan to develop the following additional functionalities:

• the incorporation of GPT-calculated features into multivariate time series prediction models as input variables
• influential event detection as early warning signals (natural disasters, geopolitical conflicts, regulatory changes)
• automatic generation of reports that describe the recent evolution of the electricity market price and the prediction of price trends.

5. Conclusions

Our research explored the crucial realm of energy price prediction within the Spanish electricity market, which has significant implications for the nation's economy, sustainability goals, and energy security. We utilized GPT models, specifically OpenAI's ChatGPT, to create a new approach to energy price trend forecasts. This new approach can be used to create innovative hybrid models merging GPT insights with domain-specific knowledge and real-time data feeds to address the unique complexities of the Spanish electricity market.