Energy, Entropy and Artificial Intelligence: Are LLM AIs Sustainable?

​

Kevin Lambert,

Liberal Studies Dept.,

California State University Fullerton

The Cost of AI

• From Aloyka Kanungo, “The Green Dilemma: Can AI Fulfil Its Potential Without Harming the Environment?” Earth .Org, July 18, 2023 (accessed 2025, Feb. 9).

Claude Shannon (1916-2001)

Shannon with Norbert Wiener (1894-1964) the founder of cybernetics

Cybernetics and feedback

• Feedback is a foundational idea for cyberneticists

​

The Cybernetics community

Every community has its own understanding of the past that is both particular to the group and an expression of the historical moment in which it existed.

• The historian’s task is to understand artificial intelligence as the result of human design.
• The task is not to guess where a technology is inevitably leading but encourage potential users to ask how and if they want to use it.

Shannon’s Mathematical Theory of Communication

“Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design.”

Shannon, The Mathematical Theory, p. 32

Jorge Luis Borges’s Library of Babel

A European alphabet as a Shannon Information Source

The Shannon Entropy of an Information Source

• A switch has two possible states: 0 (closed) and 1 (open).
• Two switches have four possible states: 00, 01, 10, 11.
• Three switches have 8 possible states: 000, 001, 010, 011, 100, 101, 110, 111.
• Adding a switch causes the information source to grow exponentially.
• One source has two states, two switches have four states, three switches have eight states and so on.
• That exponential growth in information as switches are added to the systems can be expressed in algebraic symbols as a logarithmic function.
• Thus, a general measure of the possible states of an information source becomes H= −Σp_i log (p_i). where p_i is the probability that the source occupies a particular state, say a closed switch in the simplest case of one switch, which is one half
• If a message is selected from an information source the uncertainty or entropy will fall. That requires work.

What Shannon calls natural languages have a stochastic structure �

• The stochastic structure of natural language requires that although some letters can be chosen freely, other choices will be dictated by the preceding symbols.
• Shannon calls those kinds of dependencies transition probabilities.

Markov Chain or Process

• To generate a ngram or sequence of words using transition probabilities is called a Markov chain or process (named for the Russian mathematician Andrei Markov).
• “There exist a finite number of possible “states” of a system S₁, S₂, …, S_n. In addition there is a set of transition probabilities, p_i (j), the probability that if the system is in state S_i it will go to State S_j. To make this Markoff process into an information source we need only assume that a letter is produced for each transition from one state to another. The states will correspond to the ‘residue of influence’ from preceding letters.”

Shannon, The Mathematical theory of Communication, 45

​

• LLM AIs are Markov chains with feedback (backpropagation).

Theseus the Mouse

Theseus was a wooden mouse with copper-wire whiskers that was able to find and remember its way around a labyrinth to a “cheese,” a switch that turned off its motor.

Cybernetic machines are “outside in.”

• Even though Shannon knew Theseus and its maze was a machine going through a sequence of configurations, he and his audiences still saw the mouse as an actor.
• For Shannon, just like materialist philosophers such as Daniel Dennett and leading LLM AI researchers such as Geoffrey Hinton, the mind is an illusion like that produced by a magic trick
• “The magic of consciousness, like stage magic defies explanation only so long as we take it at face value. Once we appreciate all the non-mysterious ways in which the brain can create benign “user-illusions,”we can begin to imagine how the brain creates consciousness,” Daniel Dennett, “Explaining the Magic of Consciousness,” Journal of Cultural and Evolutionary Psychology (April, 2003):7-19
• I have no opinion on soft and hard explanations of consciousness but doubt that LLM AIs uses many of the same “tricks” the brain uses to make language.

​

LLMS are expensive

• There are different pathways through the phase space of the English language to the state which selects what Shannon would call a message, and the pathway LLMs follow is expensive.
• US data centers, the environments on which AIs are “trained” account for 105 million tons of CO₂ a year, this is of the same order as that produced by domestic airlines (about 131 million tons). About 4.6% of US energy consumption is by data centers, and that figure has doubled since 2018.

Conclusions

• The stories about evolutionary changes or transitions, from wood to coal in the industrial revolution, or to a new information age, is an ideology justifying “More and more and more” energy consumption.
• That is unsustainable.
• The environmental cost should be paramount in considering the uses of LLM AIs

Disclosure Statement

• I did not use an AI to write this paper except when I searched for things on the web and then, more often than not, AI got in the way.

​