top of page

Blog

Our writing and thinking about the role of AI and data in the transition to net-zero and climate resilience.

Writer's pictureJon Thompson

Today, we’re excited to announce the launch of Weave, a data solution designed to help data scientists in the energy sector access data with just a few lines of code. Developed through a partnership between the Centre for AI and Climate and CEIMIA, Weave simplifies the notoriously complex process of gathering energy data, reducing the timeline from months to minutes.


Weave aggregates and publishes energy data in the geoparquet file format, making it easily accessible to data scientists. This innovation enables more detailed, higher-resolution demand forecasts and the analysis of energy data, helping to unlock new insights in energy consumption and management.


The long term vision is to consolidate a wide range of datasets that will improve the accuracy of energy forecasting models. As of today, Weave focuses on smart meter data that has recently been published by the UK DNOs (Distribution Network Operators). By making this data more accessible, Weave will help data scientists make smarter, quicker decisions that will shape the future of energy use.


Key features of Weave include:

  • Cloud native file storage: Geoparquet files enable access to datasets that would otherwise take days to download and process.

  • High resolution energy consumption data: Unlocks access to previously elusive high resolution smart meter data.

  • Open and free: No credit card, account creation or email address needed. Start using the data immediately.


Weave is now live and free to use. To learn more or try out Weave for yourself, visit https://weave.energy


This launch marks an important milestone in our mission to democratise energy data access, and we invite data scientists to join us in exploring the endless possibilities of Weave.

Artificial intelligence is the darling of the tech world, but when it comes to the energy sector, it's much more than just a buzzword. AI has the potential to transform how we produce, distribute, and consume energy. It can optimise grid operations, accelerate the integration of renewable energy, and make our path to net zero smoother and faster. But there's a catch: AI is only as powerful as the data we feed it.


To really unlock AI's potential in the energy transition, we need to focus on capturing large, clean and diverse sets of energy data. Otherwise, we’re just spinning our wheels, letting AI fall short of its transformative promises. So, what are the essential types of energy data we need to supercharge AI? Let’s break it down.


1. Energy Consumption Data


If AI is going to help us optimise energy use, reduce waste, and predict future demand, we need high-quality energy consumption data. This data provides a detailed picture of how much energy is being used, where, when, and by whom.


AI can take this data and analyse it to identify trends, forecast future consumption patterns, and recommend ways to reduce energy use. For example, AI can predict when demand will spike and help grid operators prepare for it, or it can help consumers and businesses cut energy costs by adjusting usage during peak times.


However, this data needs to be real-time and granular. High-level aggregate data isn’t enough. AI needs to understand energy consumption down to the level of individual buildings, devices, and even appliances. If we’re serious about using AI to tackle energy efficiency and demand management, we need to ensure that our consumption data is up to the task.


Up until very recently, this was a pipe dream. But thanks to Ofgem taking the initiative and driving change, we're starting to see a glimmer of hope. The UKs Distribution Network operators (DNOs) have been tasked with publishing high-resolution (or at least high-er resolution) energy consumption data. It's still early days and there are some challenging technical obstacles to overcome but we're really excited about the potential of this dataset.


2. Renewable Generation Data


Renewable energy is the backbone of any serious plan to decarbonise our energy systems. But renewables are unpredictable by nature. The sun doesn’t always shine, and the wind doesn’t always blow. This variability is a challenge for grid operators who need to balance supply and demand in real-time.


That’s where renewable generation data comes in. This data tells us how much energy is being generated by wind, solar, and other renewable sources at any given moment. By feeding this data into AI models, we can predict how much renewable energy will be available in the future and adjust grid operations accordingly.


AI can use renewable generation data to optimise the integration of renewables into the grid, manage storage systems, and even predict the best times to dispatch energy. But to do this effectively, we need high-resolution, real-time data.


As adoption of small-scale renewable energy generation increases, and the majority of roofs are covered with solar PV installations, we need better visibility of historic and future generation. Small scale renewable generation will play an important part in a decentralised, flexible energy system, but if we don't know how much energy is likely to be generated in different scenarios, balancing supply and demand will be incredibly difficult.


3. Low Carbon Technology and Distributed Energy Resources


The future of energy is decentralised. Low-carbon technologies like air and ground-source heat pumps and electric vehicles are proliferating. At the same time, distributed energy resources (DERs), like behind-the-meter batteries and solar PV installations, are becoming more common.


This shift towards decentralisation presents both opportunities and challenges. On one hand, DERs can make our energy systems more resilient and reduce the strain on centralised infrastructure. On the other hand, managing a grid with millions of small, decentralised energy producers is a logistical nightmare.


AI can help us manage this decentralised future, but it needs data on low-carbon technologies and DERs to do so. This data includes information on where DERs are located, how much energy they’re producing, how much they’re consuming, and when they’re connected to the grid. 


With this data, AI can optimise the operation of DERs, ensure that they’re used efficiently, and even help with demand response programs. For example, AI could predict when a fleet of EVs will be plugged in for charging and adjust grid operations to accommodate the increased demand. But to make this vision a reality, we need detailed, real-time data on every distributed energy resource out there.


The challenge comes from the lack of visibility of low carbon technologies installed today. With adoption set to increase rapidly over the coming years, the problem is only going to increase. When we spoke to the DNOs, they estimated that they had at best 50-60% visibility of installed low carbon technologies in their region. 


The reason for this gap in knowledge is due to it being optional for the consumer or installer to notify the DNO following an install, rather than it being a mandatory step. Only in the case of solar, where it's a requirement to receive grant funding, is this notification regularly completed. 


4. Weather and Environmental Data


When it comes to energy consumption and generation, weather and environmental data are absolutely critical. Both depend almost entirely on the weather, and any AI model that tries to optimise the grid without considering weather conditions is doomed to fail.


Weather data is more than just a nice-to-have, it’s a must-have for AI in the energy sector. AI can use this data to forecast renewable energy production, predict demand spikes (think heatwaves and cold snaps), and optimise storage and backup systems.


The key here is accuracy and timeliness. We need real-time weather data that’s hyper-localised to be truly effective. Global forecasts won’t help much when we’re trying to optimise a solar farm in a specific region or manage the output of a wind farm based on local wind conditions.


5. Grid Infrastructure Data


Finally, we come to grid infrastructure data, the backbone of our entire energy system. If we don’t know what’s happening on the grid, we can’t manage it effectively. AI needs detailed information about the physical infrastructure of the grid, including transmission lines, transformers, substations, and even the health of individual components.


Grid infrastructure data allows AI to optimise the operation of the grid, predict and prevent outages, and ensure that energy flows smoothly from producers to consumers. AI can also use this data to help prioritise maintenance and upgrades, reducing the risk of unexpected failures and blackouts.


But again, this data needs to be real-time and granular. AI can’t work with outdated or incomplete information. We need to know the status of every component in the grid at all times if we want to use AI to keep the lights on and the energy flowing.


Where do we go from here?


The promise of AI in the energy sector is huge, but it won’t happen on its own. We need to get serious about collecting, managing, and sharing the right kinds of data. Energy consumption data, renewable generation data, low-carbon technology and DER data, weather and environmental data, and grid infrastructure data are all essential pieces of the puzzle.


The bottom line is this: if we’re serious about accelerating the clean energy transition with AI, we need to invest in the right data infrastructure. That means real-time, high-resolution data from every corner of the energy system, and making sure it gets into the right hands, with as little friction as possible. 

When future generations look back on the 2020’s, I like to think they’ll say it was the time when humans achieved two major breakthroughs:

  1. the mass-adoption of artificial intelligence

  2. the transition to sustainable energy


With artificial intelligence set to revolutionise every industry, there is a huge opportunity to apply AI to accelerate the transition towards sustainable energy.


The harsh reality is that today, this couldn’t be further from the truth. There are surprisingly few real-world applications of artificial intelligence to address energy-system related challenges. But why?


Energy-systems of the future


Ditching fossil fuels and moving to renewable energy sources sounds like a no brainer. But the infrastructure overhaul and systemic change required to facilitate a low carbon energy-system is where the real challenge lies.


A low-carbon energy-system is also a decentralised energy-system, relying on millions of distributed energy resources, instead of the handful of centrally managed power stations that we have today. Tens of millions of small-scale solar PV installations, batteries and electric vehicles will provide the flexibility the energy system needs to perfectly balance supply with demand.


How exactly this happens is still up for debate, but there’s no doubt that an operation of this complexity will need to be a digital-first solution, utilising a cutting-edge technology-stack that may not even exist today. One thing we can be certain of is that it won’t be human dependent.


The age of artificial intelligence


AI-based solutions have been optimising our news feeds and recommendation engines for some time, but it’s only with recent advances in compute-power that AGI (Artificial General Intelligence) has become a real possibility. OpenAI’s ChatGPT stole all the headlines, but it’s the development of neural-networks and the GPUs they're trained on that has enabled the breakthrough.


The foundation of machine learning and AI models is a large, clean and diverse training dataset. When I say large, I mean absolutely enormous. The first version of ChatGPT (GPT-3) was trained on 1.3 billion parameters. Meta has just released Llama 3.1, which was trained on a staggering 405 billion parameters. These models take weeks to run and cost tens of millions of dollars in compute resources -  which is why we only get a new model a few times a year.


The main reason we’ve seen large language models take off is the availability of training data. If we want to create a similar level of intelligence within the energy sector, we need to drastically improve data availability.


As of right now, it’s pretty much impossible for data scientists to use energy system data in the most basic forms of analysis. We’re a million miles away from enabling the creation of an effective training dataset to deploy AI-based solutions. That needs to change.


Data scientists working in the energy sector spend 80% of their time collecting, parsing and cleansing data. The process of collecting the data needed for a given project takes weeks, if not months. Data scientists that are new to the energy-sector are immediately overwhelmed by industry jargon and having to navigate the numerous industry bodies. We’re actively disincentivising AI-talent from working on energy related challenges.


Energy-system data challenges


Energy-system data exists, it’s just not being used because it can’t be accessed easily. Yes, there are probably more datasets that should be created through additional monitoring. But for now at least, we should focus on extracting the enormous potential value from existing datasets.


The simplest way to do this is put it in the hands of people who can extract value, i.e data scientists. Ofgem’s drive for open data is an important first step, but it’s not enough in isolation.


I’ve spent the last four weeks speaking with data scientists in the energy sector to learn about their specific challenges and pain points. Everyone I spoke to described issues with every aspect of the system. The frustration was palpable. Some described our call as being like a therapy session.


After consolidating pages and pages of notes, I summarised the issues into four areas:


  1. Discoverability

    1. Poor knowledge as to what data exists and where to find it.

    2. Datasets are not published with a clear use-case in mind.

  2. Accessibility

    1. Lots of red tape and bureaucracy.

    2. Energy data is treated as overly sensitive.

  3. Reliability

    1. Datasets aren’t maintained or kept up to date.

    2. Solutions proposing to solve the problem are defunded and disappear. 

    3. Estimation techniques aren’t clearly explained.

  4. Usability

    1. Data isn’t standardised or consistent in any way and has poor documentation.

    2. Multiple disparate sources are needed to compile complete datasets.

    3. Where APIs do exist, they are complex and varied and therefore require further processes.


In summary, data scientists can’t find the data they need, when they can find it, it’s hidden behind paperwork and complex processes. In the rare case that they manage to get hold of it, it’s in an unusable condition, and even then, it’s unlikely to be maintained long term. 


3 ways we can improve access to energy-system data


At its core, improving access to energy data is all about removing friction. Data science is a creative endeavour. When a data scientist is in flow, asking them to wait for 3 weeks for approval, or presenting them with hundreds of files broken down by day and geographic-region, is going to stop them in their tracks.


To incentivise our brightest minds to work on energy-system related problems we need to lower the barrier to entry and encourage experimentation.


We believe there are three key areas for improvement:


  1. Simplification - Datasets should be available from a single point of access with documentation that’s easy to understand. It should be queryable through cloud-object storage, instead of having to store the data locally.

  2. Standardisation - Data structures should remain consistent over time and should be regularly maintained and kept up to date. Language should be accessible and consistent across data providers.

  3. Consolidation - Similar datasets from different sources should use the same methodology and file format, removing the need for merging, parsing or unnecessary processing.


At the Centre for AI & Climate, we’re working on a solution with these core characteristics to address the frustrations of data scientists, and help them work on energy system related problems.


If you’re a data scientist working in the energy sector, we’d love to hear from you. You can either email me at jon@c-ai-c.org, or book a slot in my diary.


Stay tuned for the launch of our prototype coming very soon!

bottom of page