Detours in Data

Comparing the electricity market data repositories from around the world for... reasons?

A reminder — we’re doing a trivia event in Melbourne on Tuesday 17 June. There are tickets still available, but they’re selling fast!

We’d love you to join us for a rowdy night of networking and nerdy trivia. Plus it’s for a good cause — all profits will go to the First Nations Clean Energy Network.

Over the last few years I’ve spent a bit of time exploring and extracting electricity data pertaining to different markets globally. Some of this has been for work, more of it has been as a… hobby? 🤷‍♂️

So today I’d like to talk about the good, the bad and the plain weird.

This is by no means a comprehensive review – more a collection of meandering observations.

Transparency

Most electricity markets make some efforts to make data publicly available under the guise of ‘visibility’ or perhaps even ‘accountability’. It’s generally easier to convince citizens that liberalised electricity markets are working in their favour if the data is somewhat accessible.

This is however easier said than done. For starters most market participants (e.g. generators, retailers) don’t access the market data in these ways. They usually interface with the market via specific APIs, file servers or other systems developed for and only available to market participants (and usually with some cost associated). There is thus limited commercial incentive to maintain public-facing datasets.

Access to market data can also be complicated by the organisations hosting the data. Australia’s NEM is blessed with the simplicity of a compulsory gross pool and a single (regional) spot price, managed by a single market operator.

However many other electricity markets allow trading on one or more power exchanges, or bilateral trades between participants where the contractual terms are opaque to the public.

For example India has three power exchanges — the Indian Electricity Exchange, which accounts for 95% of all trades, and the smaller Power Exchange of India and Hindustan Power Exchange. Further complicating matters is that only about 15% of energy is traded on the power exchanges — the remainder is contracted through long-term power purchase agreements.

So, let’s say you’re a non-market participant seeking detailed data about the pricing and operation of various markets. How do they stack up? Let’s explore a few case studies.

The NEM

Starting at home, we have the venerable NEMWeb.¹ When you first land on the page it has the attractive purple colour scheme of the modernised AEMO branding, but quickly plunges you into a torrid ftp server of a million zip files.

There are a few layers to the NEMWeb data, but arguably the most important is the ‘Data Model Archive’, which is a series of consolidated monthly datasets.

Cutting edge technology in 1990. Also, to the person who put # symbols in the filenames, I hope your pillow is warm on both sides and the barista burns your coffee and misspells your name.

The genesis of the Data Model Archive is not actually for the convenience of public users — the zipped monthly files were put onto DVD and mailed out to market participants in order to use as backups to their servers. I don’t know if AEMO still mails out monthly DVDs² — these days a month of zipped data is ~ 20 GB, so emphasis on DVDs, plural.

While the format of a million zipped csv files (or, in the Archive section, zips of zips of csvs!) can be inconvenient, NEMWeb is a standout for the sheer amount of market data freely available. Not every single dataset available to market participants lives on NEMWeb, but it’s about 95% complete (excluding private participant-specific datasets). I don’t believe there’s any other electricity market with this much data available.

And there are a huge range of open source tools which have been built on the back of this data, the poster child being Open Electricity (née OpenNEM), as well as providing a significant and important window into the market for non-market participants.

If I had to nitpick (I will) — the NEMWeb data only goes back to 2009 (when AEMO started). The data prior to 2009 was managed by NEMMCO and is not publicly available.³

The WEM

Our western cousins have a system of market data which provides some excellent contrast.

The Wholesale Electricity Market underwent a market reform in October 2023, so the data is sectioned into pre- and post-reform silos.

Pre-reform market data is available via conveniently consolidated annual csv files. The range of available datasets are not as extensive or as detailed as those of the NEM, but there’s more than enough there to undertake some detailed market analysis.

Delightful! Easy to read and scrap csv files and accompanying explanations!

Post-reform the WEM decided to implement the NEM’s retro-futurist ftp file server approach, which I shall call WEMWeb, but arguably has the much sexier offical title of NORREWMWAPP1.wem.local. 😋 Yummy.

Post-reform WEMWeb has a much greater amount of data available, which is hardly shocking since the WEM has been managed by AEMO since 2016 and the reform involved implementing a number of existing features of the NEM, like security-constrained economic dispatch and market-based ancillary services.

Where WEMWeb differs is that the data is not provided as csv files, instead it is available via JSON files.

And look, there’s probably some good technical arguments around JSON being a better standard for transmitting large amounts of data than csv files⁴, but it does make it much harder for the average analyst to casually pull data.

And it becomes all the more frustrating when you go searching for the new frequency control essential system service (FCESS) prices, only to find them buried deep within the dispatch solution files — a daily zip file which is a casual 250 MB and contains 288 individual JSON files for each 5-minutes of the day.

To the person who designed that system, may your front yard be full of upturned rakes.

NEMS

The National Electricity Market of Singapore arguably sets the bar for what most market data interfaces look like. There’s a simple and clean dashboard with high level market metrics like price and demand, and market data can be downloaded via the drop down.

Also, how’s that price capture?!

It lacks the detailed data of the NEM or WEM, but there’s enough data there to get to work and it’s easy to access.

Having to manually click a button is of course troublesome if you’re looking to implement an automated data feed or similar. It would be convenient if there was an API, but sometimes you should be careful what you wish for…

ENTSO-E

The European Network of Transmission System Operators for Electricity has the transparency platform, which provides an excellent range of data pertaining to the various European electricity markets.

They have a bunch of dashboards. And a new map! Bravo.

We’re visual learners here.

ENTSO-E even has a free and easily accessible API; amazing! This a feature lacking in almost all other publicly available market datasets.

Except the data is provided in XML files. 🤮 Whyyyyyyyyyyyy?

You can transmit data via XML files, much like I could do the school drop off on a unicycle. But let’s be real — it’s impractical and inconvenient.⁵

At least they’re nice enough to provide detailed schema. But still 🤮

The convenience of the metric system this is not.

ERCOT

The Electric Reliability Council of Texas (ERCOT) has a pretty decent range of market data, which is relatively easily accessible. It’s not quite NEMWeb standard, but it’s very good.

That is of course if you’re located in the United States, because they’ve geoblocked it for… reasons?

Cheers mates. I thought Texas was big on freedom?

NZEM

Finally, we have my favourite electricity market data repository, the Electricity Market Information website of the New Zealand Electricity Market.

It’s good — lots of datasets available conveniently via a series of uncomplicated csv files. Excellent.

But wait, what is that at the bottom of the page? Are those… social features? 🤔

Just in case you wanted to keep tabs your peers?

Wait, wait, wait. Why are there ‘Trending Datasets’? 🧐

Why is the Electricity Authority gamifying market datasets?

And there’s a forum! 😱

All market data portals should have a forum.

Honestly jealous — a forum seems like a genuinely useful feature of market operations.

NEMWeb might have the crown for the widest range of detailed data available, and an ability to fetch data in a systematic fashion, but there’s clearly a few things we could learn.

The forum. Give us a forum please AEMO.

1. Also sometimes written ‘NEMWEB’ or ‘Nemweb’ or called some variant of ‘MMS data’ or ‘Infoserver’.

2. Is anyone able to confirm?

3. Currently Speaking is officially soliciting for data donations. Please sir or madam.

4. Repeat after me. Apache Parquet. Apache. Parquet. Please.

5. See note 4. PAAAAAAAAAARQUEEEEEEEEEET!