Automation in energy
But mostly a long-winded analogy
Quick note: if you’re reading this in Outlook, some of the formatting seems to get lost in the mail. Opening it in a browser looks much nicer.
Automation features pretty heavily in the energy system. For example, most big batteries on the grid are controlled through fully automated bidding systems. These systems ingest a bunch of forecasts to determine an “optimal” strategy to maximise value for the battery, dictating when it should charge/discharge/keep the powder dry over the course of the day.
At a more distributed level, automation feels like it’s at a bit of a precipice. Electrification of heating and transport is picking up and more distributed storage is being rolled out. With this comes opportunities for computers and algorithms to implement more “optimal” decisions, working out the most economical times to heat or charge. There’s so much happening in this space, blog posts could be dedicated to a myriad of specific technologies. Instead of diving into that, I wanted to speculate about how these kinds of shifts can force the market to evolve to keep up.
To get there, there’s a fascinating (if somewhat tangential) historical example involving Wall St, Bandits and old computer programs, that gives us a loose analogy to think through what automation might mean for energy market design.
The SOES Bandits
‘Day trading’ and ‘retail investors’ entered the mainstream vernacular during the pandemic. Cashed up and trapped inside, Reddit traders took on Wall St, or speedrun bankruptcy by piling into cryptocurrency.
The concept of using computers for “day trading” (buying and selling stocks or financial instruments and only holding them for short periods of time) isn’t new. It’s early days date back to the 1980s. One notorious group who terrorised the NASDAQ around this time were called the SOES Bandits. These traders were the first to really use computer programs for high-frequency trading.
In the mid-1980s, trades through the NASDAQ stock exchange were placed with market makers over the phone. These market makers would list buy and sell prices for the listed stocks and if you wanted to trade, you had to call one of them up to place your order. These market markers would make money on the differences between the prices stocks were bought and sold for.
Having to get a human on the phone to trade was incredibly manual, especially when compared to the processes we have today. The flaws of this process was exposed at times when markets were stressed, like during the 1987 stock market crash (a.k.a Black Monday)1 where US$1.7 trillion was wiped off markets in a panic. In the midst of this chaos, it was very difficult for traders to get market makers on the phone to make their trades. At the bottom of the pile were the small retail traders, whose orders were left unprocessed while market makers dealt with large firms.2 These small traders were hung out to dry.
After the crash, there was a flood of bad press directed at the market makers for failing to maintain access to trading. In response, the NASDAQ mandated that market makers implement automated trading platforms for small traders. These platforms would require the market makers to list their buy and sell prices, and could execute orders without any human involvement. This automated platform was called the Small Order Execution System (SOES).
An opportunistic group of traders recognised the implications of this new system. Instead of having to sit in a queue waiting to call through to a market maker, small trades could be made automatically. By sidestepping the cumbersome interactions with the market makers, a range of trading strategies could be designed to exploit the system for profit. These traders came to be called the SOES Bandits.
The SOES Bandits made their money by arbitraging the different market makers. If one market maker had been slow to update the prices they listed in SOES (as this was also a manual process), these traders could quickly exploit the out-of-date prices. For example, if a company’s stock price had fallen, but one of the market makers hadn’t listed it as falling, the bandits would buy it cheaper and sell to the market maker who moved too slow. In addition, they developed software that would exploit rules within the NASDAQ for the orders in which trades are executed, allowing them to jump the queue, frontrunning bigger orders.
Unsurprisingly, market makers hated them. These bandits were making money specifically by picking on the market makers and profiting wherever a market maker was slow, lazy or incompetent. The market makers complained to the NASDAQ, and the NASDAQ tried to encumber the Bandits by adding rules and requirements onto anyone trading through the SOES. However, despite the NASDAQ’s efforts, the traders continued to find ways to use technology to work the system.3
Depending on who you asked, the SOES Bandits were either a group of exploitative traders who took advantage of the system or wily financial innovators who took on the market makers who’d favoured dealing with large firms over retail traders.
The instinct at the time was, perhaps understandably, to see their impact as negative. They were halfway-house Robin Hoods - stealing from the rich and keeping it. They made life difficult for market makers, and the response from NASDAQ was to try to undercut them with regulation.
But, with hindsight, their actions (while obviously individually profitable) were instrumental in chasing out inefficiencies. Before the SOES Bandits, stocks on the NASDAQ traded in prices of eighths (i.e. increments of $0.125), and manually over the phone. By introducing technology into their trading, they forced the hand of the NASDAQ and market makers — by the time the SOES Bandits had made out like bandits, stock prices ran to several decimals and market making was done via fully automated systems. These systems allowed for faster and more accurate trading, improving the efficiency of the market and its accessibility, particularly for retail traders.
Bringing it home
Automation is coming for energy in a similar way. It’s already integral to the bidding and operation of grid-scale batteries and renewables. Large scale batteries and renewables are required to bid to generate in the wholesale market, and use bidding software to adjust, refine and optimise how they are dispatched into the market. For example, the number of rebids (where generators change their offers to generate, typically following some change in market conditions) exploded after large-scale batteries entered the market.
This graph shows the outsized contribution batteries make to rebidding. Unlike traditional dispatchable generators (coal, gas, hydro) batteries are energy constrained. The current grid-scale batteries only have one or two hours of juice, so a highly optimised bidding strategy involves rapidly responding to the changing market conditions. Automation allows these assets to chase efficiencies at the margins — every 5-minutes counts.
The next frontier of automation extends into our homes; as electrification continues the options for optimising energy use will not only grow, but will become critical in chasing these efficiencies at the margins. Inefficient charging of a residential EV might only cost a few dollars, but smart charging offers the chance consistently and automatically reduce these costs day in day out. Extend that idea of automation and control to pool pumps, heat pumps, water heaters and many other devices and the margins are looking quite a bit larger.
Undoubtedly, there will be hiccups, but this should ultimately offer customers electricity at lower costs for the same or better services.4 Shifting electricity use into periods of renewable energy generation is great for everyone. Customers get access to cheaper, renewable electricity and the renewable generators have more customers wanting to take their power.
We might be going through our SOES Bandits/NASDAQ moment.5 Automation offers energy customers huge opportunities — lowering energy costs and streamlining services. Equally, it conjures concerns about the impact on the system and the established way of doing things. Coordinated smart charging for example, could look scary to distribution networks or AEMO, whose primary focus is maintaining the stable operation of the power system.
At risk of massively oversimplifying, there are broadly two pathways forward. One, is to look to regulation to provide guardrails. Adding in control and requirements can give AEMO and the networks reassurance that automation won’t have adverse impacts on network voltages or on system security etc. A simple example is the introduction of centralised control of rooftop solar, where PV can be switched off when needed to protect the system. The second path is to lean into automation. Automation offers the ability for very complex decisions to be made regarding when or how loads are used. If we had the right market signals in place, this automation could improve the grid — it can help soak up excess renewables, help maintain network voltages and improve the resilience of the system.
Embracing automation
I’ve worked on the regulator and regulated sides of the energy market and in both positions, this oversimplification is a false dichotomy. There are competing, valid interests and concerns that need to be considered. It is both important to put guardrails in place to protect the energy system and to recognise that the growth of home automation offers huge opportunities for consumers and the grid. Designing markets that reward the right behaviours is also much easier said than done.
The Australian Energy Market Commission published a paper in 2020 called ‘How digitalisation is changing the NEM’ (link) which tried to dig into how the balance between control and markets could be struck.6
An opportunity raised in the paper is that with more automation, it paves the way for more complex and efficient pricing. Electricity prices for households and businesses are typically simple to the detriment of efficiency. Simplicity is important to provide users with understandable pricing, but these pricing models limit the rewards for demand management.7 However, when price signals are managed through computers, more complex pricing signals can be utilised to increase efficiency.
For example, since 2014, distribution networks have been required to develop cost-reflective pricing. In practice, network tariff reform has been slow. This can be partially blamed on the slow uptake of smart meters, but it’s also due to gradualism — an insistence on slowly transitioning to more complex cost reflective prices to help consumers understand the changing tariffs. This trepidation is understandable when considering all energy consumers. But automation offers a pathway to sharper price signals that end users can respond to without having to think about them.
If more loads were automated and networks offered more cost-reflective prices, a virtuous cycle can emerge. Loads being managed around network constraints would save that end-user costs, and reduce the strain on the network. This leads to lower overall network costs, pushing down costs for all.
Another area that has been sorely neglected is market access. Energy consultancy Grids highlighted a range of changes that would provide a clearer pathway for distributed resources to participate in the market:
Increasing access to, and accessibility of market data
Make it easier to register to participate
Reduce the size of aggregations.
If we are going to embrace automation, we could learn a little from the experience with SOES Bandits. When the NASDAQ was set up to allow automated trading to occur, these traders could utilise technology to innovate, create opportunities and, over time, force the marketplace to digitise faster. In energy, if we can set up the system that provides the right access and price signals, innovation and automation will flourish. Inevitably it will cause some headaches, but on the other side is a marketplace that’s more efficient and more democratised.
Things happen
Not strictly an energy thing, but it has been fascinating watching the drama with PWC unfold. The AFR has a pretty good podcast recapping the whole thing.
The cable to nowhere (Sun Cable) should have new (old) owners by the end of this week. Some scuttlebutt we missed earlier was that Softbank was alleged to be helping out the Grok side, which while apparently not true, seems like a good rumour because Softbank will apparently invest in just about anything.
Mixed messages on the IRA. According to the AFR, Albanese and Biden have apparently hashed out a deal for Australian companies to get some of the billions of tax credits available under the landmark US Inflation Reduction Act legislation. However, the Daily Telegraph is reporting that this isn’t the case. Hard to keep up!
Known in Aus and NZ as Black Tuesday because of time zones.
Small traders could electronically lodge trade requests to market makers at the time instead of having to call, but these orders still needed to be manually processed. During the crash, they were often ignored for extended periods of time.
I got most of this story from a podcast called Patrick Boyle On Finance. It’s a great podcast for anyone interested in current affairs and history lessons from financial markets.
For the record, I am not saying smart charging is like old school 80s Wall St traders. Analogies are never perfect.
Full disclosure, I contributed to this paper while I worked at the AEMC.
Like say when a significant market event happens and it takes months and months for the effect of these costs to be felt, with interest.
I just love the way you boys write, I guess that's partly you and partly the joys of a university education. Keep these interesting analogies coming, I'm loving it!
I've already got a smart home automation system that puts the EV onto charge when the suns out, flips the dishwasher on, puts a load of washing on, and then pauses it all when the clouds come over. It's called "Dylan".