The Unified Theory of Grid Coordination:
Putting the Customer First
Bruce Nordman
Building Technologies and Urban Systems Division
Lawrence Berkeley National Laboratory
Berkeley, CA 94720
bnordman@lbl.gov
Summary—How to coordinate between the utility grid and customers has been a topic of discussion and deployment for many decades. Among the many mechanisms proposed to manage supply and demand of energy, only pricing and aggregators are on the rise. Cost-effective scalability and interoperability requires global solutions—as we have already experienced with Internet technology. This paper outlines an approach to grid coordination that supports both mechanisms, addresses emerging needs of utility grids to address distribution system capacity, and puts customer choice and benefit as primary.
Keywords—grid coordination; highly dynamic pricing; dynamic capacity management..
Over the past half century or more, many mechanisms have been proposed or deployed to coordinate customers and customer devices with the utility grid. This can have a variety of benefits including reducing wholesale energy costs, delaying or avoiding capital investments on the grid, facilitating more integration of renewable energy sources (on the customer and grid sides of the meter), improving power quality, and more.
Some of these mechanisms such as direct load control and event-based demand response are declining in use. Others such as complex two-way transactive energy have never taken off. The two mechanisms that are increasing in use are dynamic retail pricing and aggregators (Virtual Power Plants).
In physics, the Unified Field Theory [1] was proposed to bring disparate forces and particles into a single consistent framework. The presentation here is a way to bring all grid / customer coordination into such a framework[1].
Without such a framework, grid coordination has been explored and deployed in widely disparate ways around the globe, often also different within the same country with different regions, retailers, and over time. This relative chaos has greatly impaired clarity of thinking about the topic and in deployment of hardware, software, and other systems, and creation of policies that facilitate flexibility that is high performing and low cost.
The approach described here can be utilized in any country and in any application context (e.g. grid connected always, never, or intermittently), so can truly be universal. Its other core characteristic is to be simple. The combination of these two is central to the system's goals and design.
Many discussions about grid coordination confuse and conflate wholesale markets and retail markets, and services that can or should be provided by retail customers vs. wholesale (grid-scale) entities. For ordinary customers, it is helpful to disaggregate this to:
Power is only involved with devices that export power into the grid, so primarily PV and devices that include batteries. IEEE 2030.5 and associated standards cover that topic already, and those functions are mostly or entirely separate from needed coordination about energy and capacity, so Power is not covered in this paper.
This paper is informed by the idea that due to its simplicity and universality, pricing will be eventually recognized as the highest performing and lowest cost mechanism to manage flows of energy over space and time. It is not the intention of this paper to try to prove this hypothesis. Rather, this paper recognizes that pricing is an important part the future. With the principle that simpler solutions are better, the burden of proof should be for alternative approaches to explain why they are lower cost and higher performing than pricing.
The utility grid and the old phone system were invented in almost the same place at almost the same time. Both were marvels of their time and were perhaps the best systems that could be created with technology available at the time. Much later, as technology and goals advanced, Internet technology arrived and it became clear that it was a much better way to communicate than the old phone system. Over the past half century we have been phasing out the latter, dispensing with obsolete concepts and technology. Critically, Internet technology provides capabilities not possible (at reasonable cost) with the old technology.
Power distribution is our mechanism for moving electrons over space and time, both at wide area scale—the utility grid—and locally within buildings. The technology basis for power distribution today is amazingly old. While our mechanisms for generating electricity and for using it would be largely unrecognizable to those from the 1880s when the grid was invented, the ways that electricity is moved around has changed little. Within buildings we have moved from fuses to circuit breakers, but for most power loads, that is the only change. On the grid, we have more advanced technologies for specific purposes, but the overall structure hasn’t evolved significantly. The features that power distribution provides have hardly changed—we just have better ways to control or manage it. It is challenging to find an area of technology that has seen less advancement than power distribution in buildings over the past 140 years.
In contrast, for communications, not only have the devices at the edge of the network changed dramatically—from phones to computers—but the infrastructure has changed fundamentally in its basic technology, organization, and system architecture. Critically, the phone company did not invent Internet technology—incumbents almost never invent the much better future. The phone company just invented a better phone line—ISDN—which had some new capabilities but was not a network connection. ISDN lines were also expensive and so not scalable.
The structural similarities between the utility grid and the old phone system raise an obvious question. What insights about electricity can we gain from our experience with reinventing how we distribute information?
Internet technology was not developed as an adaptation of the phone system, but rather from a blank slate as the best way to move data around at any scale—in geography, topology, or bandwidth. It was not designed around any existing institutions or business models. Because it was developed separately from the phone system, there were not pressures for it to conform to the incumbent technology’s business models or system architecture principles. Originally, Internet technology was layered on top of the phone system, with acoustic couplers and then in parallel on the same wires with DSL. Eventually, the roles were flipped and phone calls began to be transited across Internet Protocol (IP) connections. Thus, there was a long transition period, but the underlying new technology was not compromised.
The phone company could have been a partner in all this, but as with so many incumbent institutions, it was incapable of seeing the much better future. In the 1950s, more than one person approached Bell Labs proposing that packet switching was an important part of the future. The response was approximately ‘we are aware of this idea, but we know it can’t possibly work so it needs no further consideration’.
The success of Internet and related technologies have many lessons for us in electricity and other technology domains, but one central one is that the best things in life are simple and universal. Everyday technology such as email addressing, web browsing, Ethernet, Wi-Fi, and USB are all quite simple at their core and used in all countries in any application context. Having many standards rather than one would be a huge detriment to interoperability, costs, security, and more. Each of these technologies does have considerable complexity internally, but the complexity is contained and does not intrude on those who simply use the technology, and does not impair the overall system architecture. The entire description of the Internet Protocol (RFC 791 [2]), including the system architecture within which it operates and many examples, is only 40 pages of sparse text. Part of achieving simplicity is that when there is complexity, the architecture contains it. Technology layering, as the OSI model specifies, enables this.
Internet technology is based on fundamentally different principles than the phone system. It is natively digital rather than analog, it has intelligence at the edges much more than the center (the phone system had no intelligence at the edge), it is a network model rather than a unitary model, it is a loosely coupled system rather than a tightly coupled one, and more.
Critically, the phone system was designed in an era long before digital data storage existed, and so operated without it. Internet technology relies on data storage everywhere. Related to this, the phone system was completely deterministic—once a circuit was set up between two telephones, assuming no hardware failures then there is no chance that call information would be lost. The Internet is non-deterministic—IP packet delivery is just ‘best effort’. However, with sufficient infrastructure deployed, packets are rarely dropped, and when they are, with storage and packet retransmission with TCP, the data are recovered and the user is never aware of this. Simply stated, storage changes everything. For Internet technology, digital communication was not enough—it also needed storage.
Our electric system was designed around the idea that storage doesn’t exist, which was a needed assumption at the time. Now that we have electricity storage that can be deployed at a cost and scale that makes its integration reasonable, we have system architecture options that previously did not exist. We also have digital technologies with which we can orchestrate the storage.
As we evolve our electricity system, we need to consider the specific technologies that we would like to have in the future as well as the processes—and impediments—to achieving that. With communications, we long had multiple parallel networks and links, not just the phone system, so there was a degree of diversity already in place. With this, it is harder to innovate with electricity separately from the grid—except inside of individual customer sites.
This raises the question of why it is so hard to change how we distribute power. As noted, improvement in this technology has been so very slow over the past century. Part of this is because of the nature of the grid being one large unitary system, tied together by synchronous AC power. The need for backwards compatibility severely limits the ability to explore or adopt new technology approaches. The places where we have seen the most dramatic change—such as USB and in vehicles—are largely decoupled from the AC grid. Not surprisingly, both use DC power which is easier to segment into distinct power domains.
In addition, AC power is necessarily regulated due to monopoly concerns, which makes change even slower and more difficult, and encourages organizations to resist change. Because power distribution has seen such little technology evolution, the relevant institutions are not set up to embrace and encourage it—energy policy makers, regulators, utilities, and even researchers.
Early on, Internet technology was not practical for the vast majority of applications for which we use it for today, but its clear value allowed it to be developed to drive down costs, drive up capabilities, until they could be reasonably used by more individuals and organizations for more purposes. The technology was not sidelined because it couldn’t immediately serve most purposes.
Price-Based Grid Coordination [3] aims to deliver the maximum value to customers by giving them 100% of the value of load flexibility (in contrast, aggregators take a large share of this value leaving customers with much less). This model also maximizes privacy and cybersecurity. Customers have the freedom to do as little or as much load shifting (or shedding) and can change this as often as desired. All customer devices can provide flexibility. In this paper the term Customer Energy Resource (CER) is used as an adaptation of the term in use in Australia of Consumer Energy Resource [4] and a replacement for the common term Distributed Energy Resource (DER). CER makes clear that the device is owned and controlled by the customer for their own benefit. CER are internal to the customer site and it is unnecessary for the grid to know about the existence of CER, much less have access to their status or exercise control.
Pricing is the only mechanism that can effectively work at any scale and in any application context. Pricing is used in wholesale electricity markets; it is used for retail[2], at the meter; and critically, pricing can be used inside of customer sites—whether grid connected or not—with ‘local prices’ [5]. Price is a unit of measurement; it can be used in cash exchange or transactions with no cash exchange, just as kilograms or meters can be used in cash exchange or for other purposes. Price is a measure of resource availability. The latter always exists; prices usually do, and can always be used.
A Highly Dynamic Price (HDP [3]) is a retail price that is:
These are necessary for the price to be reflective of actual grid conditions. Otherwise, the price is not reflective of actual grid conditions and so will lead to suboptimal performance in terms of the cost and reliability of the grid at large.
The ongoing communication between the grid and the customer should include:
The ‘local price’ inside of a building is even simpler. A building central device will compute the local price taking into account the power flow at the meter and the import and export prices, accounting for GHG costs, and other factors. Emergency alerts are really an application-layer data, not power distribution, but should be also relayed to devices.
Of course, we also need utilities to publish HDP prices with a price server. The state of California already has one operating [7], as does the Danish grid operator [8] and others.
If we were designing the grid today, with today’s technology and with the lessons from the history of Internet technology so apparent, we would be likely to quickly identify pricing as the superior mechanism. After all, pricing is used in most parts of the economy. The major impediment to having good retail electricity prices is 140 years of having bad prices.
Figure 1 shows a graphical illustration of the price-based model. Note that Figure 2 shows all possible communication paths of information to a CER. Any individual CER will use only a single path from the price server to the CER. The orange lines in the figure show functional control commands (see Table 1) sent after an entity other than the CER itself has combined the prices with functional operating considerations. Any of the four devices in the bottom half of the diagram can do the translation from price to functional control. The orange lines are functional control for when the optimization is external to the CER itself.
Figure 1. Price-Based Grid Coordination
While Figure 1 shows the communication paths, it doesn’t specify what protocol(s) are to be used. The recently introduced OpenADR 3.0 [9] can become the ordinary way for transiting data along all of the green price communication paths, particularly those inside the customer site. OpenADR 3.0 is so simple that it could be readily integrated into a Wi-Fi light bulb. The Matter standard [10] will also have the ability to carry price information inside of customer sites and is an important complement. Matter, BacNET, and other application-layer protocols will also carry the functional control signals.
Pricing balances supply and demand over a regional scale. However, for hyper-local issues as can arise on individual substation feeders (and perhaps downstream of individual transformers), pricing cannot guarantee that capacity limits won’t be exceeded. For this reason, another mechanism is needed.
Distribution system capacity issues first arose as a widespread issue in Australia, as they have the highest penetration of rooftop PV (about a third of single family residences). An increasing number of feeders had so much excess PV that customers were being prohibited from installing more so as to not risk exceeding capacity constraints. As a solution, they began deploying Dynamic Operating Envelopes (DOE [11]), which is limit-based capacity management. DOE was first added to IEEE 2030.5 [12], but support for it in OpenADR 3.0 [9] was added soon thereafter. With DOE, the grid broadcasts interval export limits to each customer on constrained feeders. Customers who might otherwise have to curtail PV could instead increase load.
EV charging raises the opposite potential problem—exceeding import constraints. A Level 2 EV charger can easily use more power than the rest of an all-electric home uses for all other loads at any time—and many times the average. The load is not constant—it only lasts for a few hours. EV charging can also be synchronized across customers due to rates with sharp drops in prices, human behavior, and other reasons. This all means that they can readily drive wires or transformers in the distribution system over their rated power levels. Australia is now studying import capacity management. EV charging stations—with many stalls and very high power levels—raise the same issue.
One possible solution is to take a cue from USB [13] and Ethernet [14] and make capacity permission-based. For both technologies, connecting devices are initially only allowed to draw minimal power, to enable communications. Before drawing more, they must request and be granted permission to use more by the supplying device. We know that “ordinary” loads don’t pose a problem, so there is no need to involve them in this, in either changing electricity prices or their behavior. We only need to manage the impact of the bursty loads (and new anomalous customers such as EV charging stations).
One possible solution is described below. It is proposed as a candidate to test, along with others. Choosing what mechanism is ultimately used should be driven by empirical testing.
In this system [15], each customer has a subscription to an amount of capacity that their traditional devices never exceed. This can be assessed through examination of recent smart-meter data (capacity management requires a smart meter to work at all). This subscription might be changeable annually, and would have a fee associated with it (though the first few kW might be free). This subscription is likely to be several times the average consumption—that is, typically the margin between typical use and the subscription will be (much) larger than the current use.
When the customer acquires an EV, a controller can monitor the current at the meter, and modulate the maximum EV charging rate to stay at or below the subscription level. If the customer wants to charge faster, an automatic request can be made to the grid for a reservation of more capacity, with a specified start time and duration, and a specified additional amount of power. The grid’s response could be any of:
Whenever a request is made, the grid responds with a list of intervals with availability and cost for each. With this, a customer may choose a different time and/or incremental value. Multiple reservations can be made to span times of different reservation prices. The specifics of this exchange are described in the OpenADR 3.0 User Guide [9].
Several countries are dominated by HDP today, such as in Scandinavia and Spain. In others, such as the UK and the US, it is available in a few places but has only limited use to date. In general, these places have switched to the new tariff before there was any significant automation in place to use it. The automation came after. This is not surprising as product manufacturers and service providers will be reluctant to introduce features that customers can’t use, but once prices are available, then there is value in offering the feature or service. Any product that can get a routine software update can easily have the price-responsive feature added.
Customer sites will commonly use many of the communication paths shown in Figure 1, though for each CER it will generally use only one. Cloud-based optimization is likely to dominate initially, with many algorithms moving to on-site devices for a variety of reasons. Price-response and capacity management also lend themselves to facilitating microgrid operation.
We will also likely see more and more use of direct current domains of power internal to buildings, to more easily and efficiently integrate combinations of PV, storage, EV charging, and loads. Capacity management and price communication are already present in some DC technologies.
Deployment of capacity management can be very surgical. The only device in a building that needs to contemplate capacity management is the EV charger. Only feeders that are constrained need anyone to do anything. On most feeders, capacity will only be in short supply occasionally.
The above is all that is needed when CER optimize to prices internally. However, for the foreseeable future, we will also make significant use of the other three paths, with the optimization done with an external control device, a central building device, or in the cloud. In these cases a standard control interface is needed[3]. Such a control interface is also needed for aggregator control (Virtual Power Plant is taken as a synonym for an aggregator [16]). Thus, standard control interfaces are needed for both pricing and VPPs. Defining such interfaces, and using policy and market tools to ensure their inclusion in products, will be a subject of much future work. The CTA-2045 standard [17] provides an example such interface for water heating (and to a limited degree thermostats), though it could be improved and extended. These semantics are replicated in OpenADR 3.0.
While pricing is higher functioning and lower cost than VPPs, for the foreseeable future both will be common, so a policy of “coexistence” needs to be established. This can assure that the customer is always in control, and that the customer can always choose—for each CER—whether they want it managed by the tariff or by a VPP. Perhaps customers could change this only once a year to provide stability to the system and protect VPP operators who earn revenue seasonally.
How VPP operation can well intersect capacity management is not known. With prices, the customer is a unitary entity with only one financial and control goal. When a VPP is introduced, then a second entity (or several more) is competing for control, and may do actions that are counterproductive for the customer as a whole, even as they are positive for the CER being controlled. An alternative is to only have VPPs that are responsible for the whole customer site, which severely limits customer choice, so is not considered desirable.
This paper describes a set of coordination mechanisms that fully cover grid needs for addressing energy use (and export) and distribution system capacity issues (power issues with inverters need to be also addressed). It covers both highly dynamic pricing and VPPs/aggregators as mechanisms. It only describes those interactions which affect the customer, so do not cover others such as how utility grids and wholesale markets interact with VPP operators.
With the introduction of the OpenADR 3.0 protocol, and the expected emergence of Matter mechanisms for energy in 2024, we now also have the protocols needed to cost-effectively deploy this widely in products. Demand flexibility can now be rapidly accelerated by focusing on the mechanisms described here, with the language, technology, and design principles presented. Demand flexibility can deliver great value to customers, utility grids, the environment, and economy, if only we try to allow it to succeed.
[1] A few customers may usefully have a more complex relationship with the grid, such as large EV charging stations and large industrial facilities. However, the existence of these need not add complexity to the other 99%+ of customers.
[2] Retail prices usually have poor correspondence to the actual value of power at a particular place and time, but they are prices nonetheless and could readily be improved.
[3] Proprietary communication could be used, including from a manufacturer cloud, but these reduce interoperability and can limit which device is allowed to control the device at all and so is a substandard solution.