Data discovery and access key to achieving net zero

Across the sectors that matter when it comes to addressing climate change, there is an opportunity to apply data science and AI to accelerate progress. Take the electricity sector for example. As we get more renewable energy on the grid we are going to need to focus on how to optimise the balancing of renewables. Successful balancing will require thousands, potentially hundreds of thousands of energy storage and demand-side-response assets to be connected to the grid. To ensure that all of these assets are optimised and collectively achieve their goal of balancing the system there will be a need for them all to be optimised using intelligent software. The Centre for AI & Climate has written previously about the need for AI-optimised electricity markets. However, to enable these companies' AI algorithms to work effectively it will be critical to ensure that there is much greater data visibility across the system, such that any one part of the system can understand the wider context in which it is offering services. This will require a step change in discovery of and access to energy-related data. For example developers of balancing assets will need a better understanding of where there are constraints on the grid now and where there are likely to be risks of supply and demand imbalances in the future. Developers of AI trading algorithms for energy assets would be able to offer a step change in effectiveness if there was more data on current and forecast supply and demand across the networks. EV charging infrastructure developers need visibility of where they can connect their assets to networks at lowest possible cost. DNOs need better visibility of distributed energy resources connected to their networks. The system operator needs much better forecasting of supply and demand to optimise the final stage of balancing through the Balancing Mechanism and ancillary markets. Suppliers offering agile tariffs will need greater visibility of current and forecast network constraints to understand where to focus the roll-out of such services. These are just a small sample of how improved data access will be critical for the transition to a net zero grid, but there will be thousands of opportunities created if data is made more available.

Beyond energy the same is true for other sectors such as transport, land use and industry.

For data scientists working on climate change, one of the most significant time sinks, is simply finding the right data as it is often held by a myriad of different data providers.

If we could simply aggregate links to climate transition data, we could radically reduce the time overhead data scientists experience in finding data. At a basic level this could be a simple online, open-access data catalogue, of links that connect data scientists to already-open data that can be found online. Beyond this initial approach however, to fully enable optimal data discovery, a data platform is needed that can support the following evolutions to support net zero-relevant data access:

• From individual datasets to an interconnected data graphs that captures the relationships between different datasets;

• From basic to increasingly in-depth data context to allow those coming to the field fresh to understand the data landscape easily;

• From a limited number of datasets to a very large number of datasets, as the system is increasingly well monitored;

• From static datasets to a combination of historic data and live data streams via APIs: to support real time system monitoring;

• From high-level metadata standards to increasingly detailed data guidelines: to allow common datasets to be machine-readable and easily discoverable;

• From lists of datasets to geospatial visualisations of datasets: because many of the questions that need answering on system optimisation are geospatial in nature;

• From a system that accepts limited quality data to one with increasing standards for cleaned and labelled datasets;

• From showcasing open data to facilitating a market for commercial energy data, to allow data owners to recover some of the costs from collecting data.

A platform that enabled these transitions would fully address the problem of data discovery and data access for climate relevant data.

It is clear we will need optimal data discovery to support a rapid transition to net zero. Such a platform would help unleash a Cambrian explosion of digital and AI innovation in support of the transition to net zero and climate resilience. CAIC is considering carefully how to develop such a platform.

Peter Clutton-Brock (@pcbrock) is co-founder of the Centre for AI & Climate.

Net zero electricity systems will be based on AI-Optimised Electricity Markets

In their proposal for the UK’s 6th Carbon Budget, the Climate Change Committee (CCC) recently recommended that the UK seek to reduce emissions by 78% by 2035. They estimated that to achieve this the UK will need to have almost completely decarbonised the UK electricity system and at the same time annual electricity demand is going to increase from 300 TWh in 2020 to 460 TWh by 2035. They posed a scenario where 70% of generation is renewable by 2035, with the bulk of this (265 TWh) coming from offshore wind. It is clear that large amounts of flexible balancing capacity will be needed to manage such system requirements and large public and private investments are being made in the physical assets that can offer balancing capacity.

The key question now is how can all of this balancing capacity be managed and delivered without compromising grid function and with minimal cost for the consumer? The answer is that in addition to all of the physical assets required, we will need AI-optimised energy markets to allow the new energy system to operate efficiently and deliver affordable energy for consumers.

The development of the wholesale electricity market and balancing markets has been shaped by the dominant sources of energy generation, which are now rapidly changing. The design of electricity markets will need to adapt quickly to manage new generation and flexibility sources.

Whilst it is increasingly clear that the UK’s future generation profile will involve large amounts of variable renewables it is less clear what the flexibility mix will look like. The potential outcomes vary widely. On the one hand we could imagine a system where we have vast numbers of households and EVs offering balancing services, requiring accurate real-time price signals. On the other hand it may prove to be the case that the most cost-efficient way of delivering balancing services involves fewer, larger storage and DSR assets strategically positioned at key locations on the networks, together with much simpler price signals to consumers to, for example, charge EVs at night.

Provided the rules are set in the right way, ultimately the market will determine which approach is the most economic. But regardless of the outcome of this issue, the key question that needs to be answered is how can we ensure the optimal use of whatever balancing mix wins? The answer is that optimal balancing of supply and demand will require AI optimised markets. What do we mean by this?

We mean that all buyers and sellers on the electricity markets that are able to supply balancing services will need to use AI optimised trading algorithms, AI-optimised forecasting algorithms, and in some instances AI-optimised asset planning. Generators will need to optimise their selling of electricity against the market. There are already examples where AI has been used to increase the value of wind assets. This will become the norm. Suppliers will need to optimise their buying of electricity from generators and flexibility providers on the wholesale market. Once again, this is already happening to some extent with Octopus and Ovo leading the way in the UK. Flexibility providers and aggregators will need to use AI to optimise their trading against their assets’ internal chemistry / physics, and against the markets into which they are trading. For example Habitat Energy is using AI optimised trading algorithms to optimise its battery trading against grid batteries’ chemistry and against the wholesale and balancing markets. Networks (ESO and DNOs) will need to optimise their purchasing of services with real-time auctions for balancing services, by allowing for optimal algorithmic trading into these markets by flexibility providers and by supporting data flows to ensure optimal, cost-efficient balancing.

AI-Optimised trading algorithms will in turn need good data: good data on any asset(s) that they are offering services against, and good data on the wider market. They will also need to access optimised forecasts of both demand and supply, which in turn will need to be optimised by AI. Such forecasts will become increasingly important as percentages of variable generation on the grid increase.

What we are seeing is that market participants at the grid edge are increasingly deploying AI to optimise their trading performance. However, they can only work with the data and trading environment that is currently available. To successfully offer the balancing services needed to deal with much larger proportions of variable renewables, they will need access to data that can radically improve their understanding of current and future supply and demand across the system, and a more technologically up to date process for trading on the electricity markets.

There is an increasingly important role for the network companies to facilitate the development of this system. This is a critical piece of the puzzle if we are to achieve efficient net-zero electricity grids.

So how do we get from where we are now to AI-optimised electricity markets? Here are some suggestions on the immediate priorities:

• All market participants will need to continue to optimise their trading strategies using machine learning, and clarify the data streams that will allow them to do this.

• Network companies could usefully focus on facilitating optimised purchasing of balancing services. To achieve this they will each individually need a far better understanding of all of their assets’ location, requiring a digital map of assets, real time data collection from across their networks, and much better forecasting capabilities of supply and demand. Network companies should consider developing roadmaps towards achieving AI-optimised balancing markets and consider working together to improve their demand and supply forecasting capabilities.

• Policy makers in government need to ensure that energy data that will allow for improved understanding of current and forecast demand, supply and constraints across the system is opened up to allow for more efficient balancing and optimised planning for future assets.

• Electricity markets need to move closer to real-time trading, exchanges need to offer APIs to improve market optimisation and agree on common data frameworks to make it easy for assets to easily trade on multiple markets. Simpler markets are easier to optimise.

• Network companies need to rethink how they develop software, quickly moving towards a more agile and iterative way of developing software.

We will need all of these changes in place by 2035, but why wait? The UK is already a leader both in the transition of its energy system to net zero and in AI. We could be a world leader in AI-optimised energy markets. Wouldn’t that be a good story to tell at COP26?

Peter Clutton-Brock (@pcbrock) is a co-founder of the Centre for AI & Climate. To find out more about the Centre’s work you can check out www.c-ai-c.org.

A.I. for Net Zero Electricity

In November 2020, the Centre for AI & Climate (CAIC) organised a workshop on AI for Net Zero Electricity. The workshop was attended by over 35 participants from the electricity sector, including representatives from Ofgem, Energy Systems Catapult, National Grid, SSE, ScottishPower, EDF, BP, UK Power Networks, UCL, E.On Future Energy Ventures, and others.

The CAIC team are grateful to all of the participants who contributed their time and ideas before, during and after the workshop.

The workshop focussed on the opportunities and challenges faced by participants organised around the following topics:

1. Future vision of AI in the electricity system

2. Data Governance

3. Grid balancing

4. Moving from concept to deployment

5. Building data science capabilities

Overall, all parties were optimistic about the potential benefits of digitalisation and recognised its potential role in reaching a net zero electricity system. Read on for a summary of the workshop discussions and its resulting recommendations for actions to address some of the issues identified.

---

High level outcomes

There was a high level of alignment across the sector on the need to decarbonise and digitalise the electricity system and awareness of the potential benefits of data science in this process. While different businesses are at different stages in their digitalisation journey, many have made progress in developing their foundational data governance procedures and basic data science capabilities.

There is uncertainty regarding the role of AI in the future electricity system. Although there are many potential use-cases, AI applications remain at an early stage of development and deployment, and there is no shared view of what the grid will look like in 5-10 years’ time. It is therefore difficult for businesses to plan and receive funding for their future digitalisation activities.

Recommendation 1: Develop a vision and roadmap for the role of AI in reaching a net zero electricity system, in consultation with businesses and subject matter experts. This should feed into the government’s proposed energy data and digitalisation strategy.

Internal cultural barriers to digitalisation was a topic that came up many times. Energy businesses have been traditionally engineering-focused, with a reliance on staff who make decisions based on deep technical expertise. There is therefore a cultural barrier to digitalisation and data science, with mistrust in the outputs of “black-box” AI models, a reluctance to overhaul legacy IT and data governance systems, and a lack of understanding of the value of collecting and using data.

Recommendation 2: Create and distribute energy-AI training resources to improve data literacy across the sector, and build links with universities to attract data science talent.

Recommendation 3: More research is needed into explainable AI to improve trust and usability of AI applications in the energy sector.

Although there has been progress in improving sector-wide data governance through initiatives such as the ENA data working group and the Modernising Data Access competition, there is an appetite for improved guidance on sector-wide data standards (e.g., meta-data standards and SIMs). Open data is widely identified as a key enabler for sectoral collaboration in developing improved and larger-scale AI applications.

Recommendation 4: Coordinate the development of data standards and prioritise access to key datasets, building on existing initiatives in the sector.

Finally, although it is recognised that Ofgem and BEIS are well placed to lead on addressing many of the identified barriers (e.g., improving innovation funding processes), there is a desire from industry for greater engagement from Government in co-creating solutions to these problems.

Recommendation 5: Ofgem and BEIS should engage more closely with businesses to address shared challenges and opportunities across the sector.

---

Conclusions

There is a huge appetite for digitalisation and decarbonisation in the sector, with promising initiatives happening both within and across businesses to build data science capabilities, improve data governance processes and plan for a digital future. To build on and coordinate these initiatives, we believe that the sector would benefit from a roadmap detailing the role of AI in achieving a net zero electricity system.

These changes require evolving regulatory processes (data standards, software development, and innovation funding) which were designed for traditional energy businesses, as well as a cultural shift towards data literacy across the workforce and recognising the value of data. Both are struggling to keep pace with the sector’s ambition.

The CAIC will communicate these workshop outcomes to Ofgem and BEIS and begin enacting these recommendations. We will continue to engage with businesses, researchers and policymakers to enable the adoption of AI to reach a net zero electricity system.

Please get in touch with us if you would like to be involved in any aspect of this work, or if you would like to discuss how we can support your organisation in its data science and AI journey.

---

The Centre for AI & Climate is a cross-functional facilitator and incubator that aims to connect capabilities across technology, policy, and business to accelerate the application of AI to solve climate challenges. Please visit icaiec.org to find out more