How many angels can dance on the head of a pin?

The question “How many angels can dance on the head of a pin?” is typically used as a mocking retort to questions that are thought to be of little usefulness. It’s especially used in reference to philosopher theologians of the Middle Ages like the Scholastics or to theology in general. It’s not a question any of the Scholastics ever actually asked. But medieval philosophy did have plenty of talk about angels, and for good reason. They used angels as subjects for thought experiments to explore concepts like cognition and identity in the most generalized way possible, in the way modern philosophers talk about brains in a vat, brains separated from the body and sent to another planet, philosophical zombies, or people living in a black-and-whiteworld. Their topics are just as relevant today as we develop technologies like artificial intelligence and deepen our understanding of the brain and the mind.

The question “How many angels can dance on the head of a pin?” is typically used as a mocking retort to the sorts of philosophical and especially theological questions that are thought to be of little usefulness and a general waste of effort. It’s especially used in reference to philosopher theologians of the Middle Ages like Scholastics including Thomas Aquinas and Duns Scotus. While the material conditions of people in the world were those of misery and squalor (so it is supposed) these folks were sitting in their ivory towers thinking about useless questions instead of doing science and inventing things. As you might guess, I don’t share this perspective and as a kind of subversion of the retort I’d like to appropriate it. The question “How many angels can dance on the head of a pin?” is not a question anyone actually ever asked. It’s a straight up caricature. But Medieval Scholastic philosophy did have plenty of talk about angels. Why? And could angels have any modern intellectual relevance?

In what follows I propose that angels were used in Medieval Scholastic philosophy as subjects for thought experiments. In such thought experiments the properties of angels were not those primarily of angels as described in biblical texts but more of an idealized notion serviceable to philosophical exercises. I propose that these sorts of philosophical angels can still be used to explore questions we find interesting today in thought experiments pertaining to cognition and consciousness. With an understanding of thought experiments as idealizations which transcend particularity in order to achieve generality I’ll go through various stages of generalization of consciousness from its particular human form to its most general form. This process moves into the complete abstraction of consciousness from any of its particular physical instantiations in order to explore its most essential features, though with our present knowledge this can only be the outline of a conceptual scaffolding since we don’t currently know what the essential features of consciousness actually are, or what it even is. And this will lead back to speculation about the nature of angels and what their nature might be.

When Medieval philosophers like Aquinas and Scotus talked about angels they did not get into historical scholarship about the way angels were understood at the times that the biblical texts were written. Or the way angels were understood during the Second Temple period with the writing of pseudepigraphal texts like the Book of Enoch. This is a fascinating subject and on this subject I’d recommend the work of Michael S. Heiser in his books Angels and The Unseen Realm. In Biblical texts angels are messengers. That’s what the Greek angelos (ἄγγελος) means. Also the Hebrew malak (מֲלְאָךְ). There’s not much information given about their metaphysical nature. But it is their metaphysical nature that is most interesting to the Medieval philosophers.

The most important attribute of angels for philosophical purposes is that they are non-corporeal. Angels do not have physical bodies. So they are very unlike us humans. Yet they are also like us humans in a very significant aspect: they are conscious beings. They’re not only conscious but also self-conscious, intelligent, and rational, again like humans. In our regular experience we only know of one kind being that is self-conscious, intelligent, and rational: human beings. And we have good reason to believe that these attributes are essentially connected to our physical bodies and especially our brains. The idea that a being could be self-conscious, intelligent, and rational without a physical body or even just a physical brain conflicts with our regular experience. But that’s why it’s interesting.

An excellent resource on the use of angels in Medieval philosophy is Martin Lenz and Isabel Iribarren’s Angels in Medieval Philosophical Inquiry: Their Function and Significance, which is a collection of essays on the subject. In particular I recommend the chapter by Dominik Perler titled Thought Experiments: The Methodological Function of Angels in Late Medieval Epistemology. In his essay Perler works with a definition of the thought experiment from The Routledge Encyclopedia of Philosophy, given by Timothy Gooding, whom he quotes: “A thought experiment is an idealization which transcends the particularity and the accidents of worldly human activities in order to achieve the generality and rigour of a demonstrative procedure.” I think this is an excellent definition for a thought experiment. If you want to get at the essence of a concept a thought experiment is a way of looking at it in the most generalized way possible. There are certain concepts like language, rationality, and possibly self-consciousness (of the most reflective sort) that we only find in human beings. How can we think about these concepts in their general form when we only have one kind of example from actual experience? We have to use our imaginations. And so we make thought experiments.

Perler notes that thought experiments take different forms in different ages. I would say that they make use of the images that are readily available in the culture. “Today, of course, philosophers hardly speak about angels. They prefer talking about brains in the vat, brains separated from the body and sent to another planet, zombies, or people living in a black-and-whiteworld.” We take our ideas from the culture: from religion, myth, literature, and film. But are these necessarily fictional? Perler makes an interesting distinction: “Of course, one needs to make a crucial distinction when talking about thought experiments. They can be understood either as scenarios involving purely fictitious entities (e.g., brains in the vat), or as scenarios appealing to entities that have real existence or could in principle have real existence, but are considered under ideal conditions (e.g., the scientist Mary who has all knowledge about colours). Since medieval authors took angels to be real entities, endowed with real causal power and interacting with other real entities, they were certainly not interested in thought experiments in the first sense [fictitious entities]. They were rather focusing on thought experiments in the second sense, analyzing angels as real creatures that transcend the material world and therefore enable us to examine cognitive activities in its purest and most ideal form, which is not subject to material constraints.”

I mentioned before that no Medieval person really asked “How many angels can dance on the head of a pin?”. But we can find examples of something kind of close. In the Summa Theologiae Thomas Aquinas asked the question: “Whether several angels can be at the same time in the same place?” (Summa Theologiae, First Part, Question 52, Article 3) This he answered in the negative: “I answer that, There are not two angels in the same place.” I’d actually answer the question differently I think. But what’s important is that the question makes sense to ask. We assume here that angels are immaterial. So there are questions that arise regarding the relation of immaterial things to space. Does something immaterial take up space? Could it take up space under certain circumstances? If the specific question about angels seems too remote, think about other immaterial things. First, consider mathematical sentences like 1+1=2. Does that take up space? It would seem not to. It’s just not a spatial kind of thing. Second, consider a field as understood in physics? A field is “a physical quantity, represented by a scalar, vector, or tensor, that has a value for each point in space and time.” Some fields probably have actual physical existence. In quantum field theory certain fields are understood to be the most basic building block of physical reality that give rise to everything else. But other fields are more abstract, especially since we can invent all kinds of fields to suit our purposes. For example, we can imagine a temperature field where every point in a given volume has a certain scalar temperature value. This kind of field would obviously be spatial in nature since it is defined over a given region of space. But it’s not exactly a physical thing either. It’s just a set of numbers. These two kinds of immaterial things have very different relations to space. For any kind of immaterial thing, including angels, it’s reasonable to ask which kind of relation to space it has. How many mathematical sentences, such as 1+1=2, can fit on the head of a pin? In the asking of that question we can see that mathematical sentences just aren’t the kinds of things that take up space or subsist in space at all. That’s an interesting feature of mathematical sentences.

Let’s look at another Medieval example. In his article Dominik Perler looks at the work of Duns Scotus and William of Ockham on the subject of perception. They framed the issue as one of how angels could have cognition of things in the world. Physical beings perceive the world through their physical senses. But nonphysical beings wouldn’t have sense organs of this sort. They would need other sorts of cognitive devices “to make things cognitively present to them.” Both Scotus and Ockham held that “every cognition requires a cognitive act and a cognitive object.” But they had different views about what that cognitive object would be. For Scotus, “the appropriate object for an intellectual act is the essence of a thing.” For Ockham, “the appropriate object for an intellectual act is the individual thing with its individual qualities.” Both Scotus and Ockham then think about how cognitive objects become cognitively accessible. And they use angels in their thought experiments to make this as general as possible. They don’t even refer to sense perception. Rather, there’s simply something that makes the cognitive object cognitively accessible. That something could be sense perception but it doesn’t have to be. So it’s thoroughly general. For me, as a physical being with physical senses, before I cognize something like a chair I perceive it with my senses. But after I perceive it with my senses there’s an intellectual act by which I cognize it. For Scotus the object of this intellectual act is the essence of a chair. For Ockham the object of this intellectual act is the individual object, which happens to be a chair. These are two very different ways of understanding cognition. And by using angels, nonphysical beings, in their thought experiments they bracket everything that comes before the intellectual act because it’s that intellectual act specifically that interests them. It doesn’t matter if what makes the cognitive object cognitively accessible is sight, smell, echolocation, electroreceptors, or what have you. This can apply to humans, bats, sharks, computers, or angels. That’s why it’s fully generalized. They are just interested in what happens in the intellect.

So how does the cognitive object become cognitively accessible to the intellect? For Scotus, the cognitive object is the essence of a thing and “to make the essence of a thing cognitively accessible, the intellect needs a cognitive device: the intelligible species.” For Ockham, the cognitive object is the individual thing and “to make an individual thing cognitively accessible, the intellect simply needs a causal relationship with that thing.” For Scotus, “the intelligible species determines the content of an intellectual act”. For Ockham, “the individual thing itself determines the content of an intellectual act.” What does this look like in practice? I expect that Ockham’s view will seem most plausible up front. Going back to the chair example, the essence of a chair sounds kind of fictional. Isn’t it just the thing itself we’re dealing with that we then understand to be a chair? Nevertheless, I’d say that in the history of philosophy these two views are pretty evenly represented. Just to comment on it briefly, it’s arguably that our cognition always has some intentionality to it. It’s always directed towards something. We don’t just cognize the thing but also what we can do with it. A chair, for example, is not just an elevated platform with four legs and a backing. A chair is something to sit on. And it’s arguable that that usefulness is the first thing we cognize.

Perler comments that for both Scotus and Ockham, “their detailed analysis was motivated by an interest in the general structure of cognition… They both wanted to know how cognition works in principle, i.e. what kinds of entities and relations are required in any cognitive process.” The philosophy of mind understandably tends to be very human-focused. But that leaves out many other ways of cognizing, both real and imagined, from bats to sharks to computers. By using angels in their thought experiments Scotus and Ockham were able to think about these issues in a way that transcended the particularity of all these cases and think about cognition as such. I think that Medieval people had an easier time thinking about human attributes in this more general way because more of them believed in a universe populated by beings of an unseen realm that had many human-like attributes. Part of the “discarded image” as C.S. Lewis called it. The anthropologist Marshall Sahlins called these kinds of beings “metapersons”, something he proposed recovering and studying in a “new science of the enchanted universe” (see The New Science of the Enchanted Universe: An Anthropology of Most of Humanity). This worldview of a heavily populated unseen realm led Medievals to come up with useful concepts that they otherwise might not have. This is something Joseph Koterski has argued, with the case of Boethius in particular (see Natural Law and Human Nature). In his writings about these various kinds of human-like beings Boethius had to come up with a technical term and stipulate an appropriate definition to refer to all of them, both human and non-human. He used the term “person”, which he defined as an individual substance of a rational nature. A wonderfully general definition that preserves the essential core.

Could we make similar use of angels in modern thought experiments? I think they could be used in a similar way as used by the Medievals, i.e. to transcend the particularity and accidents of human beings and explore concepts in their most general form. One topic that interests me and where I can see application of this is to consciousness, or more especially self-consciousness. This is a distinction that Roger Scruton has made (see The Soul of the World, p. 40). Consciousness is the state of being aware of and able to perceive one’s environment, thoughts, and feelings. This is something that many animals would seem to have.  Self-consciousness is a higher level of awareness where an individual is not only conscious but also aware of themselves as a distinct entity with their own thoughts, feelings, and identity. It involves reflection on one’s own mental states and actions. Humans are certainly self-conscious. As humans we know this because we have direct, first-person access to this higher level of awareness and reflection. And we assume, rightly and non-solipsistically, that this is common to other humans as well. We can also talk to each other and infer from our conversations that we share this self-conscious nature. Other animals might be self-conscious in this way as well. Or they might not be. We don’t really know because we are not of the same species and we can’t talk with them. No other animals have language. It’s possible we Homo sapiens are unique in our self-consciousness.

Let’s consider a progression of abstraction from human consciousness to hypothetical consciousness in other physical beings, to general consciousness as such, abstracted from all particular instances.

We start with human beings. We know a lot about human physiology, psychology, and neurology. All these contribute to the human experience, including our experience of self-consciousness. We know our mental operations are highly if not entirely dependent on our brains. At a very basic level our brains are composed of neurons and the synaptic relations between neurons. Our brains are also further organized into structures whose activity we can observe under various scenarios and activities. We’ve amassed a good deal of knowledge about our brains but there is still a great deal we don’t know. And ultimately we don’t have an answer to what David Chalmers has called the “easy problem of consciousness”: how all our mental experiences like perception, memory, and attention correlate to specific brain mechanisms and processes. We don’t have a complete “map” of the brain. We can’t read people’s minds by looking at their brains. So we still have to speak in rather general terms.

I think the next step removed from human beings would be other species that are closely related to us in the genus Homo. Unfortunately all other species of our genus are extinct so we can’t observe them or talk with them. It’s reasonable that it may have been possible to talk to some of them. Neanderthals seem to have been very similar to humans. They may have had very similar brains, similar enough to have similar capabilities but different enough to be interesting and to make some generalizations about our common features. That’s no longer possible, but that was a very close step removed from human beings that would have been useful for generalization.

To move another step from humans we may have to look beyond our planet and our evolutionary relatives. This would be an organic alien lifeform. We may never encounter such a being but I put it before an artificial lifeform, which we may encounter much sooner, simply because we can imagine such beings being, for lack of a better word, organic: organisms composed of organic molecules with water-based biochemistry, probably composed of cells, with genetic information encoded in molecules, and evolutionary history. Basically, not computers or machines. We can only speculate what such beings might be like. And I suspect we’ll have more insight into the nature of their consciousness after we develop artificial consciousness, for reasons I’ll explain. What might their brains be like? Would they have the basic cellular unit (like neurons) with a complex structure built up on the relations between them?

The next level removed would be beyond any organic physical entity, human or alien, to artificial entities: artificial intelligence or artificial consciousness. The consciousness of artificial consciousness would be considered artificial because it would not be naturally occurring but instead be a product of human engineering and design. What I’m talking about here is not just artificial intelligence of the sort that started showing up everywhere in 2023, when it seemed like almost everything was getting AI capability of some kind. These are large language models (LLMs) like ChatGPT. A large language model’s artificial intelligence processes and generates human-like text based on patterns learned from large sets of text data. But LLMs lack actual awareness and subjective experience. Artificial consciousness, on the other hand, would entail a system having self-awareness, subjective experiences, and a sense of understanding its existence and environment.

We may well encounter artificial consciousness before we encounter alien life. So even though alien life might be more similar to us, being organic, if we don’t encounter it we won’t be able to learn much from it or make comparisons to the particular physical features that give rise to their consciousness. Nevertheless we may be able to infer certain features that a self-conscious alien species would have to have from features held in common between human consciousness and artificial consciousness.

What might be analogous to the brain in artificial consciousness? We can imagine that an artificial consciousness would have physical similarities to modern computers. They would probably have processing units and memory systems. Although existing artificial intelligence in the form of large language models may not be conscious, artificial consciousness may end up having similar features like a neural network structure consisting of layers of interconnected nodes (like neurons) that process data through weighted connections.

What kinds of features would be held in common between all sorts of conscious entities: humans, Neanderthals, aliens, and artificial consciousness? Right now we can’t know. But we can speculate. I suspect that there will be some kind of structure common to all. This might not be the case, which would be very confusing indeed. But let’s suppose there would be some kind of common structure. I would further speculate that it would have the form composed of basic objects plus the relations between objects. For example, in human brains the basic object is the neuron. And the brain is organized as a system of relations between neurons, the synapses. What constitutes the object and the relations might be rather different from one entity to the next. But I suspect that basic structure will apply to all conscious beings. The definition of the structure will be very complex. It’s not just that there are objects and relations between objects. Many structures would meet that description without being conscious. For example, a three-dimensional matrix of neurons in which every neuron was simply connected synaptically to its nearest neighbor wouldn’t be much of a structure. Consciousness-producing structures are much more complex. In all these cases so far each entity is physical and has some consciousness-imparting structure. In Aristotelian terms these entities are composites of matter and form, a notion called hylemorphism. In each case there is a material substratum for the consciousness-producing form or structure.

This brings me to the final level of abstraction where we pull away the material substratum leaving only the consciousness-producing structure itself; the form without the matter. This moves beyond all physical conscious entities to the completely abstract and nonphysical. The features that I’ve speculated are held in common between conscious entities have the features of a mathematical structure: a set of objects having some additional features such as relations between objects. Fully abstracted, the actual objects themselves don’t matter. These instead become open slots. This is how Verity Harte describes structures generally. Structures are the sorts of things that make available open slots that can be filled by particular objects. When the slot is filled by a physical entity, like a neuron, the structure has a particular physical instantiation. But when the slot is empty the structure is abstracted from physical instantiation. Here the structure is at its most generalized state.

The highest levels of abstraction bring up some interesting philosophical issues. Let’s start with artificial consciousness. Here the question of the number of angels dancing on the head of a pin, or the matter of space, comes up again. The volume of an adult human brain is around 1300 cubic centimeters. That’s not even counting the rest of the body that’s needed for the brain to survive and operate. Our consciousness requires space to operate. Artificial consciousness would also take up space. Maybe a single artificial conscious entity would require even more space than a human body or human brain. But let’s hope that it could be less. Could there be a kind of corollary to Moore’s Law for consciousness. Moore’s Law is the trend in the semiconductor industry for the number of transistors in integrated circuits to double every two years. Since individual transistors are being made to be progressively smaller, more can fit in a given area. Could an artificial consciousness fit in the volume of a modern smartphone? Or maybe even smaller? Could multiple artificial consciousnesses fit on the head of the pin? That might give renewed relevance to the never-actually-asked question of angels dancing on the head of a pin. How many artificial consciousnesses could operate in a space the size of the head of a pin?

Artificial consciousness also brings up questions about embodiment. In humans this is a question of the way our minds relate not just to our brains but to our whole bodies and even to our environments. Hubert Dreyfus was an important contributor to philosophical work on this question, notably in his book What Computers Can’t Do. Dreyfus was drawing on the thought of Martin Heidegger. Maurice Merlau-Ponty would also be relevant to the subject. Dreyfus referred to Heideggerian concepts such as being-in-the-world, ready-to-hand, present-to-hand, and mood to explain the necessarily embodied nature of consciousness. Dreyfus argued that human understanding is fundamentally embodied and situated in a social and physical context, that much of human expertise is tacit and non-reflective. We tend to be very brain-centered in our thinking about human beings. But we are much more than just our brains. If a human brain could be separated from the rest of the body and survive it’s hard to speak for that quality of life, if it could even qualify as a life. Notably, even Dreyfus wasn’t arguing that artificial intelligence or artificial consciousness weren’t possible. Just that they would have to have those sets of features he identified. Would an artificial consciousness then require not only processing and memory but also motility and some form of embodied existence in the world?

Then there’s the highest, nonphysical level of abstraction. An abstract, nonphysical entity that has all the essential structures of consciousness. What is the nature of such an abstraction? Could such an abstract entity actually be conscious? Or would it have mere hypothetical consciousness, in the event that the open slots happen to be filled by physical objects? Does it even make sense to think of such nonphysical, abstract entities as existing at all? This last question is the basic question of platonism? Do abstract objects in general have any existence when we’re not thinking about them or using them in some way?

We can imagine the case where this nonphysical abstraction only becomes conscious when it is physically instantiated. It’s embodied in some way, whether that body by organic or computer. This would be like the relationship of the concept of a machine to the machine itself. We might have the concept of a machine like an engine but this concept doesn’t actually perform work. For that we actually need to build the physical machine. But even with the abstract concept of a machine we could say true and false things about it. Any concept for a steam engine, for example, has to have features for heat addition, expansion, heat rejection, and compression. Any concept lacking these features could be said to be nonfunctional, even in the abstract. In the abstract neither the concept for the functional design nor the nonfunctional design produce any work. But if we were to build them, one would work and the other would not. A philosophical angel might be the same way. It’s not actually conscious in the abstract but the abstract structure would be conscious if it were physically instantiated. So we can still refer to it, like Scotus and Ockham, and talk about cognition and consciousness, and how they would work in any kind of physical instantiation, be it human, Neanderthal, alien, or computer.

Or we can imagine the more extravagant case where this nonphysical abstraction actually is conscious, even when unembodied. That’s difficult to imagine. And there are a number of objections we can make. Maybe not the most obvious of these objections but one I think is important is the matter of temporality. Consciousness seems to be essentially temporal in nature. It’s a process. Our brains aren’t static but change from moment to moment. And our brain states are constantly changing. So how would a nonphysical abstraction do the things that conscious entities do across time? We think of abstractions as being atemporal. They don’t ever change. That’s true in a sense. They don’t change with respect to actual time. But some abstract objects could have multiple elements that are organized in a way like a process. For example, an algorithm with multiple steps is a kind of process. The steps in an algorithm aren’t moments in time but they have an ordering and sequence similar to temporal processes. To take another example, a piece of music, when played, is played in time. But we can also look at a whole page of music at once. In the abstract the piece of music is no longer temporal, but it still has order and sequence.

Another objection is that it seems like a nonphysical abstraction just isn’t enough. It needs something more. Stephen Hawking put this thought quite poetically in A Brief History of Time: “Even if there is only one possible unified theory, it is just a set of rules and equations. What is it that breathes fire into the equations and makes a universe for them to describe? The usual approach of science of constructing a mathematical model cannot answer the questions of why there should be a universe for the model to describe. Why does the universe go to all the bother of existing?” Of philosophical angels we might ask, what breathes fire into them? Do they need physical instantiation to be conscious?

In the foregoing I’ve also operated under an assumption that consciousness is a structure of some kind. And that we find these kinds of structures instantiated in multiple ways that constitute individual conscious beings. This is reasonable. We always find them together. We don’t observe human consciousness without human brains. But what if that assumption is not correct? Maybe consciousness is not the structure but the structure is a condition for consciousness. And consciousness itself is something more basic. Maybe consciousness itself is a kind of monad, a fundamental, indivisible unit that possesses perception and self-awareness. A theory of such monads was developed in the thought of Gottfried Wilhelm Leibniz. There are some modern theories based on the idea that consciousness might be a basic component of the universe, rather than something reducible to anything else. Not a structure but a unit. See David Chalmers as one prominent philosopher in this area. That’s a very different picture.

In the monad view the complex structures associated with brains and computer systems would not have some structural essence identical to consciousness. Instead they might produce conditions under which it can appear. As an analogy, a generator built to create an electric current has a complex structure. But this structure is not the electric current. The electric current is something else, and something much simpler. The generator produces the conditions under which an electric current can appear. But neither the generator nor its structure are the electric current itself.

In all of this my thoughts about angels have been instrumental. Not so much a question about what actual angels might be like, but about the utility of a certain concept of angels for philosophical exercises. But what of actual angels? Could any of these ideas about angels be correct about actual angels? Granted, that question will only be interesting if you think angels might actually exist. Well, I do. So it is an interesting question for me. I like to think these kinds of metaphysical speculations are on the right track. But I definitely won’t make any strong claims to that effect. As in scripture, the angels will just have to tell us themselves what they are like, if they feel so inclined.

But to be honest what got me onto this whole subject and thinking about it was just that derisive line about angels dancing on the head of a pin. I enjoy reading Medieval philosophy and I just don’t find the characterization of their thought as endless circling around useless questions to be at all accurate. Their thinking was deep and touched on the most foundational issues of knowledge and existence. And when they did talk about angels they were talking about timeless issues on the subjects of cognition, individuation, and language. So one response to the derisive line could be that it’s not really the “own” you think it is. Medieval philosophers didn’t actually ask how many angels could dance on the head of a pin. But even if they had, I’m sure it would have been interesting. 

Human Language and Artificial Neural Networks

The recent developments in AI are quite impressive. If someone had told me a couple years ago about the capabilities of something like ChatGPT I wouldn’t have believed them. AI certainly has enormous practical benefit. But since artificial neural networks were inspired by biological neural networks they can also be useful models for them. In this episode I share some recent studies investigating the behavior of the brain using AI models and evaluating their possible underlying computational similarities.

This is a follow-up to some things discussed in our last group episode on artificial intelligence. Since that conversation I’ve been digging more into the subject and wanted to share some ideas about it. I’ve been interested in artificial intelligence for a number of years. Part of that interest is because of its practical usefulness, which we’re really seeing explode now, with ChatGPT in particular. But I’m also interested in artificial intelligence as a model that could give us insights about human intelligence.

I have to say that the performance of these most recent models, like ChatGPT-3 and especially ChatGPT-4, is something that has really surprised me. If someone had told me a couple years ago that in 2022 & 2023 a deep learning model would be able to perform as well as these do I wouldn’t have believed it. I’m typically predisposed to doubt or at least be very critical about the capabilities of artificial intelligence. But in this case I think I was wrong and I’m happy to have been wrong about that. I don’t mean to swing too far to the other extreme and get too exuberant about it and overstate the capabilities of these models. But just a little bit of excess excitement might be excusable for the moment.

One claim that would be too extreme would be that these deep learning models are actually self-conscious already. Now I have no philosophical reason to suppose that an artificial device could not be self-conscious. I just don’t think we’re there yet. Another, less extreme claim, but one that would still go too far would be that deep learning models actually replicate the way human speech is produced in the brain. I think the implementations are still distinct. But that being said, I think there are enough similarities to be useful and interesting.

For comparison, there are insights we can gain into sight and hearing from cameras and audio recorders. Obviously they are not the same as our own sense organs but there are some similar principles that can help us think about how our senses work. The comparisons work both at the level of physical mechanisms and at the level of data processing. For example, I think there are some interesting insights about human senses from perceptual coding. Perceptual coding is a method used in digital signal processing that leverages the limitations and characteristics of the human sensory systems (auditory and visual) to provide data compression. For example, in audio, certain sounds are inaudible if they’re masked by louder sounds at a similar frequency. Similarly, in an image, subtle color differences in areas with high spatial detail are less noticeable than in smooth areas. Perceptual coding takes advantage of this by selectively removing the less noticeable information to reduce the data size, without significantly impacting perceived quality. This is done with MP3s and JPEGs. Extending this comparison to large language models, I’d propose that models like ChatGPT might be to human language production what cameras, JPEGs, audio recorders, and MP3s are to sight and sound. They aren’t the same but there are some parallels. ChatGPT is not the same as a human brain any more than a camera is an eye or an audio recorder is an ear. But, more modestly, ChatGPT may have some interesting similarities to human language production.

The most significant developments in this technology are so recent that the most useful reading material I’ve had to go to on the subject is peer-reviewed literature from the past year. Even there a lot of the research was done with GPT-2, which was a much less advanced model than we have available today. So it will be interesting to see what studies come out in the next year and beyond. The papers I want to focus on are 3 research papers from 2022 that present the results of experiments and 2 (just slightly older) perspective papers that offer some broad reflections and theoretical considerations.

In what follows, I’ll proceed in three parts: (1) philosophical background, (2) an overview of neural networks: biological and artificial, and (3) recent scientific literature.

Philosophical Background

Most of the philosophy I’d like to discuss is from the 20th century, in which there was considerable philosophical interest in language in what has been called the “linguistic turn”. But first something from the 18th century.

Something that stood out to me in all the research articles was the issue of interpretability. Artificial neural networks have been shown to have remarkable parallels to brain patterns in human brain production. That’s nice because the brain is so complex and little understood. The only problem is that ANNs themselves are also extremely complex and opaque to human comprehension. This challenges a notion going back to the 18th-century Italian philosopher Giambattista Vico: the Verum factum principle.

The phrase “verum factum” means “the true is the made,” which refers to the notion that truth is verified through creation or invention. In other words, we can only know with certainty that which we have created ourselves, because we understand its origins, structure, and purpose. Vico developed this principle as a critique of the Cartesian method of knowing, which, in Vico’s view, emphasized the abstract and ignored the concrete, humanistic dimensions of knowledge. By asserting that true knowledge comes from what humans create, Vico highlighted the role of human agency, creativity, and historical development in the creation of knowledge.

However, applying the verum factum principle to complex human creations like modern industrial economies, social organizations, big data, and artificial neural networks poses some interesting challenges. These creations certainly reflect human ingenuity and creativity, but they also possess a complexity that can make them difficult to fully comprehend, even for those directly involved in their creation. Artificial neural networks are inspired by our understanding of the human brain, but their function, especially in deep learning models, can be incredibly complex. It’s often said that these networks function as “black boxes,” as the pathway to a certain output given a certain input can be labyrinthine and largely inexplicable to humans, including their creators. So, while the verum factum principle encapsulates the role of human agency and creativity in the construction of knowledge, artificial neural networks illustrate that our creations can reach a level of complexity that challenges our ability to fully comprehend them.

Now turning to the 20th century I think four philosophers are especially relevant to the subject. These are Martin Heidegger, Ludwig Wittgenstein, Ferdinand de Saussure, and Hubert Dreyfus. Of these four Hubert Dreyfus was the one who most directly commented on artificial intelligence. But Dreyfus was also using ideas from Heidegger in his analysis of AI.

Let’s start with Dreyfus and Heidegger. Dreyfus’s main arguments were outlined in his influential 1972 book, What Computers Can’t Do. The core of his critique lies in what he sees as AI’s misguided reliance on formal symbolic reasoning and the assumption that all knowledge can be explicitly encoded. Dreyfus argued that human intelligence and understanding aren’t primarily about manipulating symbolic representations, as early AI research assumed. Instead, he believed that much of human knowledge is tacit, implicit, and tied to our embodied experience of “being in the world”, an important Heideggerian concept. These are aspects that computers, at least during that time, couldn’t easily replicate.

Dreyfus drew heavily on the philosophy of Martin Heidegger to make his arguments. Heidegger’s existential phenomenology, as expressed in his 1927 book Being and Time describes human existence (“Dasein”) as being-in-the-world—a complex, pre-reflective involvement with our surroundings. This contrasts with the traditional view of humans as subjects who perceive and act upon separate objects. According to Heidegger, we don’t usually encounter things in the world by intellectually representing them to ourselves; instead, we deal with them more directly.

Dreyfus related this to AI by arguing that human expertise often works in a similar way. When we become skilled at something, we don’t typically follow explicit rules or representations—we just act. This aligns with Heidegger’s notion of ‘ready-to-hand’—the way we normally deal with tools or equipment, not by observing them as separate objects (‘present-at-hand’), but by using them directly and transparently in our activities.

Another philosopher relevant to this topic is Ludwig Wittgenstein. He was one of the most influential philosophers of the 20th century. He is considered to have had two major phases that were quite different from each other.  His early work, primarily represented in Tractatus Logico-Philosophicus, proposed that language is a logical structure that represents the structure of reality. But in his later work, chiefly Philosophical Investigations, Wittgenstein advanced a very different view.

In Philosophical Investigations, Wittgenstein introduces the concept of language as a form of social activity, what he called “language games.” He argues that language does not have a single, universal function (as he had previously believed) but is instead used in many different ways for many different purposes.

Language, Wittgenstein claims, should be seen as a myriad of language games embedded in what he called ‘forms of life’, which are shared human practices or cultural activities. Different language games have different rules, and they can vary widely from commands, to questions, to descriptions, to expressions of feelings, and more. These language games are not separate from our life but constitute our life.

Wittgenstein also introduced the idea of ‘family resemblances’ to discuss the way words and concepts gain their meanings not from having one thing in common, but from a series of overlapping similarities, just like members of a family might resemble each other.

He also challenged the idea that every word needs to have a corresponding object in the world. He argued that trying to find a definitive reference for each word leads to philosophical confusions and that words acquire meaning through their use in specific language games, not through a one-to-one correspondence with objects in the world. So, for Wittgenstein, the meaning of a word is not something that is attached to it, like an object to a label. Instead, the meaning of a word is its use within the language game. This was a notion similar to a theory of language called structuralism. 

The leading figure of structuralism was the Swiss linguist Ferdinand de Saussure. His ideas laid the groundwork for much of the development in linguistics in the 20th century and provided the basis for structuralism. In his course in general linguistics, compiled from notes taken by his students, Saussure proposed a radical shift in the understanding of language. He proposed that language should be studied synchronically (as a whole system at a particular point in time) rather than diachronically (as a historical or evolutionary development). According to Saussure, language is a system of signs, each sign being a combination of a concept (the ‘signified’) and a sound-image (the ‘signifier’). Importantly, he emphasized that the relationship between the signifier and the signified is arbitrary – there is no inherent or natural reason why a particular sound-image should relate to a particular concept.

Regarding the creation of meaning, Saussure proposed that signs do not derive their meaning from a connection to a real object or idea in the world. Instead, the meaning of a sign comes from its place within the overall system of language and its differences from other signs; It’s within the structure of the language. That is, signs are defined not positively, by their content, but negatively, by their relations with other signs. For example, the word “cat” doesn’t mean what it does because of some inherent ‘cat-ness’ of the sound. Instead, it gains meaning because it’s different from “bat,” “cap,” “car,” etc. Moreover, it signifies a particular type of animal, different from a “dog” or a “rat”. Thus, a sign’s meaning is not about a direct link to a thing in the world but is about differential relations within the language system.

Saussure’s ideas about language and the generation of meaning can be interestingly compared to the techniques used in modern natural language processing (NLP) models, such as word2vec and to cosine similarity. For example, in word2vec, an algorithm developed by researchers at Google, words are understood in relation to other words.

Word2vec is a neural network model that learns to represent words as high-dimensional vectors (hence “word to vector”) based on their usage in large amounts of text data. Each word is assigned a position in a multi-dimensional space such that words used in similar contexts are positioned closer together. This spatial arrangement creates ‘semantic’ relationships: words with similar meanings are located near each other, and the differences between word vectors can capture meaningful relationships.

A measure of the similarity between two vectors is called cosine similarity. In the context of NLP, it’s often used to measure the semantic similarity between two words (or word vectors). If the word vectors are close in the multi-dimensional space (meaning the angle between them is small), their cosine similarity will be high, indicating that the words are used in similar contexts and likely have similar meanings. There are some interesting parallels between Saussure’s linguistics and AI language models. Both approaches stress that words do not have meaning in isolation but gain their meaning through their relations to other words within the system; language for Saussure and the trained model for word2vec.

Neural Networks: Biological and Artificial

Recall Hubert Dreyfus’s critique of formal symbolic reasoning in artificial intelligence and the assumption that all knowledge can be explicitly encoded. His critique is most relevant to traditional programming in which explicit program instructions are given. In machine learning, however, and in artificial neural networks program rules are developed in response to data. To whatever degree this is similar to the human mind, biology is at least the inspiration for artificial neural networks.

What is the structure of biological neural networks (BNNs)? In the brain connections between neurons are called synapses. Synapses are the tiny gaps at the junctions between neurons  in the brain where communication occurs. They play a vital role in the transmission of information in the brain. Each neuron can be connected to many others through synapses, forming a complex network of communicating cells.

Neurons communicate across the synapse using chemicals called neurotransmitters. When an electrical signal (an action potential) reaches the end of a neuron (the presynaptic neuron), it triggers the release of neurotransmitters into the synapse. These chemicals cross the synapse and bind to receptors on the receiving neuron (the postsynaptic neuron), which can result in a new electrical signal in that neuron. This is how neurons interact with each other and transmit information around the brain.

Synapses form during development and continue to form throughout life as part of learning and memory processes. The creation of new synapses is called synaptogenesis. This happens when a neuron extends a structure called an axon toward another neuron. When the axon of one neuron comes into close enough proximity with the dendrite of another neuron, a synapse can be formed.

The strength of synapses in the brain can change, a phenomenon known as synaptic plasticity. This is thought to be the basis of learning and memory. When two neurons are activated together frequently, the synapse between them can become stronger, a concept known as long-term potentiation (LTP). This is often summarized by the phrase “neurons that fire together, wire together”.

On the other hand, if two neurons aren’t activated together for a while, or the activation is uncorrelated, the synapse between them can become weaker, a process known as long-term depression (LTD).

Multiple factors contribute to these changes in synaptic strength, including the amount of neurotransmitter released, the sensitivity of the postsynaptic neuron, and structural changes such as the growth or retraction of synaptic connections. By adjusting the strength of synaptic connections, the brain can adapt to new experiences, form new memories, and continually rewire itself. This is a dynamic and ongoing process that underlies the brain’s remarkable plasticity.

How then do biological neural networks compare to artificial neural networks? In an artificial neural network, each connection between artificial neurons (also called nodes or units) has an associated weight. These weights play a role somewhat analogous to the strength of synaptic connections in a biological brain. A weight in an ANN determines the influence or importance of an input to the artificial neuron. When the network is being trained, these weights are iteratively adjusted in response to the input the network receives and the error in the network’s output. The goal of the training is to minimize this error, usually defined by a loss function.

The process of adjusting weights in an ANN is a bit like the changes in synaptic strength observed in biological neurons through processes like long-term potentiation (LTP) and long-term depression (LTD). In both cases, the changes are driven by the activity in the network (biological or artificial) and serve to improve the network’s performance – either in terms of survival and behavior for a biological organism, or in terms of prediction or classification accuracy for an ANN.

Of courses there are still multiple differences between biological neural networks and artificial neural networks. ANNs usually involve much simpler learning rules and lack many of the complex dynamics found in biological brains, such as the various types of neurons and synapses, detailed temporal dynamics, and biochemical processes. The biological synaptic plasticity is a much richer and more complex process than the adjustment of weights in an ANN. Also, in most ANNs, once training is complete, the weights remain static, while in biological brains, synaptic strength is continually adapting throughout an organism’s life. Biological and artificial neural networks share computational principles but they certainly don’t implement these computations in the same way.  Brains and computers are simply very different physical things, right down to the materials that compose them.

Artificial neural networks have been in development for several decades. But it is in very recent years that we’ve seen some especially remarkable advances, to which we’ll turn now.

Recent Scientific Literature

I’d like to share 5 papers that I’ve found useful on this subject. Three are research papers with experimental data and two are perspective papers that offer some broad reflections and theoretical considerations.

The 3 research papers are:

“Brains and algorithms partially converge in natural language processing”, published in Communications Biology in 2022 by Caucheteux & King.

“Shared computational principles for language processing in humans and deep language models”, published in Nature Neuroscience in 2022 by Goldstein et al.

“Explaining neural activity in human listeners with deep learning via natural language processing of narrative text”, published in Scientific Reports in 2022 by Russo et al.

And the 2 perspective articles are:

“Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks”, published in Neuron in 2020 by Hasson et al.

“A deep learning framework for neuroscience”, published in Nature Neuroscience in 2019 by Richards et al.

In each of the three research papers human participants read or listened to certain passages while their brain signals for specific brain regions were measured. Deep learning models were trained on this data to predict the brain signals that would result from the text. Researchers looked for instances of high correlation between actual brain patterns and the brain patterns predicted by the model and mapped where in the brain these signals occurred at various points in time before and after word onset. In particular, they noted whether the brain regions activated corresponded to those regions that would be expected from neuroscience to activate in the various stages of language processing.

In the first article, “Brains and algorithms partially converge in natural language processing”, published in Communications Biology in 2022 by Caucheteux & King the researchers used deep learning models to predict brain responses to certain sentences. Then the actual brain responses of human subjects were used as training data for the models. They used a variety of models that they classified as visual, lexical, and compositional. Then they evaluated how well these different types of models matched brain responses in different brain regions. The brain responses in the human subjects were measured using functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG).

Regarding the 3 different types of models:

Visual models are deep learning models that are primarily used for tasks involving images or videos. They are trained to recognize patterns in visual data, which can then be used to perform tasks such as image classification, object detection, image generation, and more. The most common types of visual deep learning models are convolutional neural networks (CNNs). CNNs are specifically designed to process pixel data and have their architecture inspired by the human visual cortex. 

Lexical models are models that focus on the processing of words or “lexemes” in a language. They work with individual words or groups of words (n-grams), treating them as atomic units. Lexical models can learn word representations (often called “embeddings”) that capture the semantic meanings of words, and their relationships with each other. They are often used in natural language processing (NLP) tasks such as text classification, sentiment analysis, and named entity recognition. Examples of lexical models include traditional word2vec or GloVe models, which map words into a high-dimensional vector space.

Compositional models, also called “sequential” or “recurrent” models, handle sequences of data where the order of the data points is important, such as sentences, time-series data, etc. They are designed to process one part of the sequence at a time and maintain a kind of memory (in the form of hidden states) of what has been seen so far. This allows them to capture patterns over time and use this information to make predictions about future data points in the sequence. Examples include causal language transformers (CLTs) like GPT.

Interestingly enough, the accuracy of the different types of models was observed to vary with time from the word onset. And the moments of high correlation of each model type corresponded with the activation of certain brain regions.

In early visual responses – less than 150 ms, when subjects would first see a word – brain activations were in the primary visual cortex and correlated best with activations in visual models, convolutional neural networks (CNNs).

At around 200 ms these brain activations were conveyed to the posterior fusiform gyrus. At the same time lexical models like Word2Vec started to correlate better than CNNs. This tracks with the hypothesis that the fusiform gyrus is responsible for orthographic and morphemic computations.

Around 400 ms brain activations were present in a broad fronto-temporo-parietal network that peaked in the left temporal gyrus. At this point lexical models like Word2Vec also correlated with the entire language network. These word representations were then sustained for several seconds, suggesting a widespread distribution of meaning in the brain.

Around 500-600 ms there were complex recurrent dynamics dominated by both visual and lexical representations.

After 800 ms, brain activations were present in the prefrontal, parietal, and temporal lobes. At the same time compositional models like causal language transformers (CLTs) correlated better than lexical models. The team speculated that these late responses might be due to the complexity of the sentences used in this study, potentially delaying compositional computations.

The researchers concluded from their experiment that the results show that deep learning algorithms partially converge toward brain-like solutions.

In “Shared computational principles for language processing in humans and deep language models”, published in Nature Neuroscience in 2022 by Goldstein et al. the researchers compared the responses of human participants and autoregressive deep language models (DLMs) to the text of a 30-minute podcast.

The authors note that human language has traditionally been explained by psycholinguistic approaches using interpretable models that combine symbolic elements, such as nouns, verbs, adjectives, and adverbs, with rule-based operations. This is similar to the kind of traditional programming that Hubert Dreyfus argued would not be viable for AI. In contrast, autoregressive Deep Language Models (DLMs) learn language from real-world textual examples, with minimal or no explicit prior knowledge about language structure. They do not parse words into parts of speech or apply explicit syntactic transformations. Instead, these models learn to encode a sequence of words into a numerical vector, termed a contextual embedding, from which the model decodes the next word. Autoregressive DLMs, such as GPT-2, have demonstrated effectiveness in capturing the structure of language. But the open question is whether the core computational principles of these models relate to how the human brain processes language. The authors present their experimental findings as evidence that human brains process incoming speech in a manner similar to an autoregressive DLM.

In the first experimental setup, participants proceeded word by word through a 30-minute transcribed podcast, providing a prediction of each upcoming word. Both the human participants and GPT-2 were able to predict words well above chance. And there was high overlap in the accuracy of the predictions of human subjects and GPT-2 for individual words, i.e. words that human subjects predicted well GPT-2 also predicted well. This experiment was determined to demonstrate that listeners can accurately predict upcoming words when explicitly instructed to do so, and that human predictions and autoregressive DLM predictions are matched in this context. Next the researchers wanted to determine if the human brain, like an autoregressive DLM, is continuously engaged in spontaneous next-word prediction without such explicit instruction. And whether neural signals actually contain information about the words being predicted.

In the next experimental setup, the researchers used electrocorticography (ECoG) to measure neural responses of human participants before and after word-onset. Subjects engaged in free listening, without being given any explicit instruction to predict upcoming words. The goal was to see if our brains engage in such prediction all the time as simply a natural part of language comprehension.

The results from human subjects in this experiment were also compared to models. The first model used was a static word embedding model, GloVe. The model was used to localize electrodes containing reliable responses to single words in the narrative. The words were aligned with neural signals and then the model would be trained to predict neural signals from word embeddings. A series of coefficients corresponding to features of the word embedding was learned using linear regression to predict the neural signal across words from the assigned embeddings. “The model was evaluated by computing the correlation between the reconstructed signal and the actual signal” for the word.

In the results of this experiment there was indeed found to be a neural signal before word onset. But what the model also enabled the researchers to do was ascertain some kind of semantic content from that signal, since the model had been trained to predict certain neural signals for given words. What was observed was that “the neural responses before word onset contained information about human predictions regarding the identity of the next word. Crucially, the encoding was high for both correct and incorrect predictions. This demonstrated that pre-word-onset neural activity contains information about what listeners actually predicted, irrespective of what they subsequently perceived.” Of course, sometimes the subject’s predictions were wrong. So what happened in those cases? “The neural responses after word onset contained information about the words that were actually perceived.” So “the encoding before word onset was aligned with the content of the predicted words” and “ the encoding after word onset was aligned with the content of the perceived words.” This all aligns with what we would expect under a predictive processing (PP) model of the brain.

The next level of analysis was to replace the static embedding model (GloVe) with a contextual embedding model (GPT-2) to determine if this would improve the ability to predict the neural signals to each word. It did; an indication that contextual embedding is a closer approximation to the computational principles underlying human language. And the improved correlation from contextual embedding was found to be localized to specific brain regions. “Encoding based on contextual embeddings resulted in statistically significant correlations” in electrodes that “were not significantly predicted by static embedding. The additional electrodes revealed by contextual embedding were mainly located in higher-order language areas with long processing timescales along the inferior frontal gyrus, temporal pole, posterior superior temporal gyrus, parietal lobe and angular gyrus.” The authors concluded from this that “the brain is coding for the semantic relationship among words contained in static embeddings while also being tuned to the unique contextual relationship between the specific word and the preceding words in the sequence.”

The authors submit that DLMs provide a new modeling framework that drastically departs from classical psycholinguistic models. They are not designed to learn a concise set of interpretable syntactic rules to be implemented in novel situations, nor do they rely on part of speech concepts or other linguistic terms. Instead, they learn from surface-level linguistic behavior to predict and generate the contextually appropriate linguistic outputs. And they propose that their experiments provide compelling behavioral and neural evidence for shared computational principles between the way the human brain and autoregressive DLMs process natural language.

In “Explaining neural activity in human listeners with deep learning via natural language processing of narrative text”, published in Scientific Reports in 2022 by Russo et al. human participants listened to a short story, both forward and backward. Their brain responses were measured by functional MRI. Text versions of the same story were tokenized and submitted to GPT-2. Both the brain signal data and GPT-2 outputs were fed into a general linear model to encode the fMRI signals.

The 2 outputs researchers looked at from GPT-2 were surprisal and saliency. Surprisal is a measure of the information content associated with an event, in terms of its unexpectedness or rarity. The more unlikely an event, the higher its surprisal. It is defined mathematically as the negative logarithm of the probability of the event. Saliency refers to the quality by which an object stands out relative to its neighbors. In a text it’s the importance or prominence of certain words, phrases, or topics, a measure of how much a particular text element stands out relative to others in the same context.

What they found in their results was that the surprisal from GPT-2 correlated with the neural signals in the superior and middle temporal gyri, in the anterior and posterior cingulate cortices, and in the left prefrontal cortex. Saliency from GPT-2 correlated with the neural signals for longer segments in the left superior and middle temporal gyri.

The authors proposed that their results corroborated the idea that word-level prediction is accurately indexed by the surprisal metric and that the neural activation observed from the saliency scores suggests the co-occurrence of a weighing mechanism operating on the context words. This was something previously hypothesized as necessary to language comprehension.

The involvement of areas in the middle and the superior temporal gyrus aligns with previous studies supporting that core aspects of language comprehension, such as maintaining intermediate representations active in working memory and predicting upcoming words, do not necessarily engage areas in the executive control network but are instead performed by language-selective brain areas that, in this case, are the ones relatively early in the processing hierarchy.

I found the following comment in the discussion section of the paper quite interesting: “In general, considering that the architecture of artificial neural networks was originally inspired by the same principles of biological neural networks, it might be not at all surprising that some specific dynamics observed in the former are somehow reflected in the functioning of the latter.” I think that’s an interesting point. The whole idea of artificial neural networks came from biological neural networks. We were basically trying to do something similar to what neurons do. We don’t know exhaustively how biological neural networks work but we do know that they work very well. When we are finally able to make artificial networks that work quite well it’s perhaps to be expected that they would have similar characteristics as biological neural networks.

The other two papers were perspective papers. These didn’t present the results of experiments but discussed what I thought were some interesting ideas relating to the whole interchange between language processing in the human brain and in deep learning models.

In “Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks”, published in Neuron in 2020 by Hasson et al. the authors covered several topics. One thing they addressed that I found interesting was a challenge to three basic assumptions of cognitive psychology. These assumptions are:

1. The brain’s computational resources are limited and the underlying neural code must be optimized for particular functions. They attribute this to Noam Chomsky and Jerry Fodor.

2. The brain’s inputs are ambiguous and too impoverished for learning without built-in knowledge. They attribute this to Noam Chomsky.

3. Shallow, externally supervised and self-supervised methods are not sufficient for learning. They attribute this to Steven Pinker.

In response to the first assumption the authors argue that the brain’s computational resources are actually not scarce. “Each cubic millimeter of cortex contains hundreds of thousands of neurons with millions of adjustable synaptic weights, and BNNs utilize complex circuit motifs hierarchically organized across many poorly understood cortical areas. Thus, relative to BNNs, ANNs are simplistic and minuscule.” Artificial neural networks are indeed trained on huge amounts of data. GPT-4 is essentially trained on the whole internet. Human children don’t learn to talk by reading the whole internet; that’s true. But the human brain is also a lot more complex than even the most sophisticated artificial neural networks; so far at least. So if GPT-4 is able to perform so impressively with a structure that’s less sophisticated than the human brain we can expect that the human brain’s computational resources are hardly scarce.

In response to the second assumption the authors argue that the brain’s input is not impoverished. Noam Chomsky, arguably the most important linguist of the 20th century, argued for what he called the “poverty of the stimulus,” meaning that the linguistic input children receive is often incomplete, ungrammatical, or otherwise imperfect. But they still manage to learn their native language effectively. How? Chomsky proposed that there is a “Language Acquisition Device” (LAD) within the human brain. This hypothetical module is thought to be equipped with knowledge of a “Universal Grammar,” which encapsulates the structural rules common to all human languages. But Hasson et al. argue that there is no poverty of the stimulus because deep learning models can produce direct fit with reliable interpretations using dense and broad sampling for the parameter space. The model is casting a very wide net. They state: “One of our main insights is that dense sampling changes the nature of the problem and exposes the power of direct-fit interpolation-based learning… The unexpected power of ANNs to discover unintuitive structure in the world suggests that our attempts to intuitively quantify the statistical structure in the world may fall short. How confident are we that multimodal inputs are in fact not so rich?” By the way, I was sharing a draft of this with a friend who shared another recent paper with me by UC Berkeley professor Steven Piantadosi, titled “Modern language models refute Chomsky’s approach to language”. I’m not going to get into that now but just thought I’d mention it.

In response to the third assumption the authors argue that shallow self-supervision and external-supervision are sufficient for learning. The authors cite Pinker’s book The Language Instinct: How the Mind Creates Language as an example of the view that they are challenging. Pinker’s views are very similar to Chomsky’s. Pinker argues that language learning is not just about imitation or conditioning. Instead, he believes that the human brain has an inherent structure for understanding language, which is why children are able to learn languages so rapidly and effortlessly, often making grammatical leaps that aren’t explicitly taught or present in their environment. But Hasson et al. argue that humans have a great deal of external supervision from our environment, both social and physical. They refer to the importance of embodiment to predictive processing, referring to the ideas of Andy Clark and Karl Friston, among others.

Another subject the authors address is the issue of interpretability. This goes back to the Verum factum principle from Vico. Scientific models, including those in neuroscience, are often evaluated based on two desirable features: (1) interpretability and (2) generalization. We want explanations to have good predictive power but we also want to be able to understand them. Not just verify that they work. If it’s an equation we like to be able to look at an equation and be able to intuit how it works. And this means that the equation can’t be too long or have too many parameters. However, interpretability and generalization are often in conflict. Models with good interpretability may have strong explanatory appeal but poor predictive power, and vice versa.

The authors suggest that the brain is an exceptionally over-parameterized modeling organ. Interpretability in the brain is intractable for the same reason interpretability of deep learning models is intractable. They work with a huge number of parameters. There’s quantification occurring but it’s not like a concise equation that you can look at in grasp intellectually. The authors propose that neural computation relies on brute-force direct fitting, which uses over-parameterized optimization algorithms to enhance predictive power, i.e. generalization, without explicitly modeling the underlying generative structure of the world.

One thing that’s really nice about this paper (and I highly recommend it by the way, it’s a delightful read) is its 3 “boxes” that touch on some key concepts. One box covers the biomimicry of biological neural networks by artificial neural networks. The authors state that artificial neural networks (ANNs) are learning models that draw inspiration from the biological neural networks (BNNs) present in living brains, but that ANNs are a highly abstracted version of BNNs. Some biological nervous systems include functional specialized system-level components like the hippocampus, striatum, thalamus, and hypothalamus, elements not included in contemporary ANNs. ANNs are also disembodied and do not closely interact with the environment in a closed-loop manner. While the authors concede that ANNs are indeed highly simplified models of BNNs, they propose that there exist some essential similarities: they both belong to the same group of over-parameterized, direct-fit models that depend on dense sampling for learning task-relevant structures in data. And, crucially, ANNs are currently the only models that achieve human-like behavioral performance in many domains and can offer unanticipated insights into both the strengths and limitations of the direct-fit approach. Like BNNs, ANNs are founded on a collection of connected nodes known as artificial neurons or units that loosely resemble neurons in a biological nervous system. Each connection, akin to synapses in BNNs, links one artificial neuron to another, and the strength of these connections can be adjusted through learning. The connections between artificial neurons have weights that are adjusted during the learning process based on supervised feedback or reward signals. The weight amplifies or reduces the strength of a connection. And much like BNNs, ANNs are sometimes organized into layers.

Another “box” addresses embodiment. This is something the philosopher Andy Clark has addressed a lot in his work. Not to mention, going further back, the philosopher Maruice Merleau-Ponty. At present, ANNs are disembodied and unable to actively sample or modify their world. The brain does not operate with strictly defined training and test regimes as found in machine learning. Objective functions in BNNs must satisfy certain body-imposed constraints to behave adaptively when interacting with the world. The authors suggest that adding a body to current ANNs, capable of actively sampling and interacting with the world, along with ways to directly interact with other networks, could increase the network’s learning capacity and reduce the gaps between BNNs and ANNs. Interestingly enough, they cite Wittgenstein’s “Philosophical Investigations” when addressing the way social others direct our learning processes.

One other topic in the paper that I found interesting was a discussion of “System 1” and “System 2”. This model was made most famous by Daniel Kahneman in his 2011 book Thinking Fast and Slow. The authors cite Jonathan St B. T. Evans’s 1984 paper “Heuristic and analytic processes in reasoning”. And there are earlier precedents for the general idea going back further in history. System 1 represents fast, automatic, and intuitive thinking, what Evans called heuristic processes. And System 2 represents slow, effortful, and deliberate thinking, what Evans called analytic processes. Hasson et al. propose that we can understand System 1 to be a kind of substrate from which System 2 can arise. System 2 is where things get really interesting. That’s where we find some of the most impressive capacities of the human mind. But they maintain that we have to start with System 1 and build from there. They state: “Although the human mind inspires us to touch the stars, it is grounded in the mindless billions of direct-fit parameters of System 1.” They see artificial neural networks as having the most relevance toward explaining System 1 processes. And the thing is we seem to be continually finding that System 1 includes more than we might have thought. “Every day, new ANN architectures are developed using direct-fit procedures to learn and perform more complex cognitive functions, such as driving, translating languages, learning calculus, or making a restaurant reservation–functions that were historically assumed to be under the jurisdiction of System 2.” 

In “A deep learning framework for neuroscience”, published in Nature Neuroscience in 2019 by Richards et al. the authors focus on three key features of artificial neural network design – (1) objective functions, (2) learning rules, and (3) architectures – and address how these design components can impact neuroscience.

The authors observe that when the traditional framework for systems neuroscience was formulated, they could only collect data from a small selection of neurons. Under this framework, a scientist observes neural activity, formulates a theory of what individual neurons compute, and then constructs a circuit-level theory of how these neurons integrate their operations. However, the question arises as to whether this traditional framework can scale up to accommodate recordings from thousands of neurons and all of the behaviors that one might want to explain. It’s arguable that the classical approach hasn’t seen as much success when applied to large neural circuits that perform a variety of functions, such as the neocortex or hippocampus. These limitations of the classical framework suggest that new methodologies are necessary to capitalize on experimental advancements.

At their fundamental level, ANNs model neural computation using simplified units that loosely emulate the integration and activation properties of real neurons. The specific computations performed by ANNs are not designed but learned. When setting up ANNs, scientists don’t shape the specific computations performed by the network. Instead, they establish the three components mentioned previously: objective functions, learning rules, and architecture. Objective functions measure the network’s performance on a task, and learning involves finding synaptic weights that maximize or minimize this objective function. These are often referred to as ‘loss’ or ‘cost’ functions. Learning rules offer a guide for updating the synaptic weights. And architectures dictate the arrangement of units in the network and determine the flow of information, as well as the computations the network can or cannot learn.

Richards et al. make an observation about interpretability similar to that made by Hasson et al. The computations that emerge in large-scale ANNs trained on high-dimensional datasets can be hard to interpret. An ANN can be constructed with a few lines of code, and for each unit in an ANN, the equations determining their responses to stimuli or relationships to behavior can be specified. But after training, a network is characterized by millions of weights that collectively encode what the network has learned, and it is difficult to envision how such a system could be described with only a few parameters, let alone in words. They suggest that we think about this in the following way. Theories can have a compact explanation that can be expressed in relatively few words that can then be used to develop more complex, non-compact models. They give the theory of evolution by natural selection as a comparative example. The underlying principle is fairly simple and comprehensible, even if the actual mechanics that emerge from it are very complex. For systems neuroscience we can start with these three relatively simple and comprehensible principles: objective functions, learning rules, and architecture. Then even though the system that emerges from that is too complex to comprehend at least the underlying principles are comprehensible and give some degree of intuitive understanding.

Conclusion

Something that I find exciting about all this is that it’s an interesting interface between philosophy of mind, neuroscience, and programming. I think that some of the most interesting problems out there are philosophical problems. Even many scientific problems transition into philosophical problems eventually. But our philosophy needs periodic grounding in the world of empirical observations. What we might call armchair philosophy runs the danger of getting untethered from reality. In the philosophy of mind we can speculate about a lot of things that don’t work out very well in neuroscience. That’s not to say that philosophy of mind has to be entirely bounded by neuroscience. Just because human minds work in a certain way doesn’t mean that minds of any kind would have to be constrained in the same way. There could be many different ways for minds to work. But if we’re theorizing about ways other types of minds might work we don’t, at present, have ways to verify that they actually would work. With theories about human minds we can at least try to verify them. Even that’s kind of challenging though because the brain is so complex and difficult to observe directly at high resolution.

Still, there’s a lot about our brains that we do know that we can take into account in our theories of the mind. We know that our brains have neurons and that neurons make synaptic connections. And we know that those synaptic connections can strengthen or weaken. We can at least account for that in our theories. Artificial neural networks patterned after biological neural networks are useful tools to model our brains. We can’t go into every synaptic cleft in the brain to sample its flux of neurotransmitters. Or record the firing frequency of every neuron in the brain. That would be great but we just don’t have that capability. With artificial neural networks, as imperfect approximations as they are, we at least have recorded information for billions of parameters, even if their sheer quantity defies comprehension. And we can try out different configurations to see how well they work.

Another subtopic that’s interested me for a while is the possibility of what I call a general theory of mind. “General” in the sense of applying beyond just the special case of human minds, a theory of the human mind being a “special” theory of mind. What other kinds of minds might there be? What are all the different ways that a mind can work? AI might give us the ability to simulate and test more general and exotic possibilities and to extract the general principles they all hold in common.

I think the recent success of these large language models is quite exciting. Maybe a little bit frightening. But I’m mostly excited to see what we can learn.

How to Use Entropy

Entropy is an important property in science but it can be somewhat challenging. It is commonly understood as “disorder”, which is fine as an analogy but there are better ways to think about it. As with many concepts, especially complex ones, better understanding comes with repeated use and application. Here we look at how to use and quantify entropy in applications with steam and chemical reactions.

Entropy is rather intimidating. It’s important to the sciences of physics and chemistry but it’s also highly abstract. There are, no doubt, more than a couple of students who graduate with college degrees in the physical sciences or in engineering who don’t have much of an understanding of what it is or what to do with it. We know it’s there and that it’s a thing but we’re glad not to have to think about it any more after we’ve crammed for that final exam in thermodynamics. I think one reason for that is because entropy isn’t something that we often use. And using things is how we come to understand them, or at least get used to them.

Ludwig Wittgenstein argued in his later philosophy that the way we learn words is not with definitions or representations but by using them, over and over again. We start to learn “language games” as we play them, whether as babies or as graduate students. I was telling my daughters the other day that we never really learn all the words in a language. There are lots of words we’ll never learn and that, if we happen to hear them, mean nothing to us. To use a metaphor from Wittgenstein again, when we hear these words they’re like wheels that turn without anything else turning with them. I think entropy is sometimes like this. We know it’s a thing but nothing else turns with it. I want to plug it into the mechanism. I think we can understand entropy better by using it to solve physical problems, to see how it interacts (and “turns”) with things like heat, temperature, pressure, and chemical reactions. My theory is that using entropy in this way will help us get used to it and be more comfortable with it. So that maybe it’s a little less intimidating. That’s the object of this episode.

I’ll proceed in three parts.

1. Define what entropy is

2. Apply it to problems using steam

3. Apply it to problems with chemical reactions

What is Entropy?

I’ll start with a technical definition that might be a little jarring but I promise I’ll explain it.

Entropy is a measure of the number of accessible microstates in a system that are macroscopically indistinguishable. The equation for it is:

S = k ln W

Here S is entropy, k is the Boltzmann constant, and W is the number of accessible microstates in a system that are macroscopically indistinguishable.

Most people, if they’ve heard of entropy at all, haven’t heard it described in this way, which is understandable because it’s not especially intuitive. Entropy is often described informally as “disorder”. Like how your bedroom will get progressively messier if you don’t actively keep it clean. That’s probably fine as an analogy but it is only an analogy. I prefer to dispense with the idea of disorder altogether as it relates to entropy. I think it’s generally more confusing than helpful.

But the technical, quantifiable definition of entropy is a measure of the number of accessible microstates in a system that are macroscopically indistinguishable.

S = k ln W

Entropy S has units of energy divided by temperature, I’ll use units of J/K. The Boltzmann constant k is the constant 1.38 x 10-23 J/K. The Boltzmann constant has the same units as entropy so those will cancel, leaving W as just a number with no dimensions.

W is the number of accessible microstates in a system that are macroscopically indistinguishable. So we need to talk about macrostates and microstates. An example of a macrostate is the temperature and pressure of a system. The macrostate is something we can measure with our instruments: temperature with a thermometer and pressure with a pressure gauge. But at the microscopic or molecular level the system is composed of trillions of molecules and it’s the motion of these molecules that produce what we see as temperature and pressure at a macroscopic level. The thermal energy of the system is distributed between its trillions of molecules and every possible, particular distribution of thermal energy between each of these molecules is an individual microstate. The number of ways that thermal energy of a system can be distributed among its molecules is an unfathomably huge number. But the vast majority of them make absolutely no difference at a macroscopic level. The vast majority of the different possible microstates correspond to the same macrostate and are macroscopically indistinguishable.

To dig a little further into what this looks like at the molecular level, the motion of a molecule can take the form of translation, rotation, and vibration. Actually, in monatomic molecules it only takes the form of translation, which is just its movement from one position to another. Polyatomic molecules can also undergo rotation and vibration, with the number of vibrational patterns increasing as the number of atoms increases and shape of the molecule becomes more complicated. All these possibilities for all the molecules in a system are potential microstates. And there’s a huge number of them. Huge, but also finite. A fundamental postulate of quantum mechanics is that energy is quantized. Energy levels are not continuous but actually come in discrete levels. So there is a finite number of accessible microstates, even if it’s a very huge finite number.

For a system like a piston we can set its entropy by setting its energy (U), volume (V), and number of atoms (N); its U-V-N conditions. If we know these conditions we can predict what the entropy of the system is going to be. The reason for this is that these conditions set the number of accessible microstates. The reason that the number of accessible microstates would correlate with the number of atoms and with energy should be clear enough. Obviously having more atoms in a system will make it possible for that system to be in more states. The molecules these atoms make up can undergo translation, rotation, and vibration and more energy makes more of that motion happen. The effect of volume is a little less obvious but it has to do with the amount of energy separating each energy level. When a set number of molecules expand into a larger volume the energy difference between the energy levels decreases. So there are more energy levels accessible for the same amount of energy. So the number of accessible microstates increases.

The entropies for many different substances have been calculated at various temperatures and pressures. There’s especially an abundance of data for steam, which has had the most practical need for such data in industry. Let’s look at some examples with water at standard pressure and temperature conditions. The entropy of

Solid Water (Ice): 41 J/mol-K

Liquid Water: 69.95 J/mol-K

Gas Water (Steam): 188.84 J/mol-K

One mole of water is 18 grams. So how many microstates does 18 grams of water have in each of these cases?

First, solid water (ice):

S = k ln W

41 J/K = 1.38 x 10-23 J/K * ln W

Divide 41 J/K by 1.38 x 10-23 J/K and the units cancel

ln W = 2.97 x 1024

That’s already a big number but we’re not done yet.

Raise e (about 2.718) to the power of both sides

W = 10^(1.29 x 10^24) microstates

W = 101,290,000,000,000,000,000,000,000 microstates

That is an insanely huge number.

Using the same method, the value for liquid water is:

W = 10^(2.2 x 10^24) microstates

W = 102,200,000,000,000,000,000,000,000 microstates

And the value for steam is:

W = 10^(5.94 x 10^24) microstates

W = 105,940,000,000,000,000,000,000,000 microstates

In each case the increased thermal energy makes additional microstates accessible. The fact that these are all really big numbers makes it a little difficult to see that, since these are differences in exponents, each number is astronomically larger than the previous one. Liquid water has 10^(9.1 x 10^23) times as many accessible microstates as ice. And steam has 10^(3.74 x 10^24) times as many accessible microstates as liquid water.

With these numbers in hand let’s stop a moment to think about the connection between entropy and probability. Let’s say we set the U-V-N conditions for a system of water such that it would be in the gas phase. So we have a container of steam. We saw that 18 grams of steam has 10^(5.94 x 10^24) microstates. The overwhelming majority of these microstates are macroscopically indistinguishable. In most of the microstates the distribution of the velocities of the molecules is Gaussian; they’re not all at identical velocity but they are distributed around a mean along each spatial axis. That being said, there are possible microstates with different distributions. For example, there are 10^(1.29 x 10^24) microstates in which that amount of water would be solid ice. That’s a lot! And they’re still accessible. There’s plenty of energy there to access them. And a single microstate for ice is just as probable as a single microstate for steam. But there are 10^(4.65 x 10^24) times as many microstates for steam than there are for ice. It’s not that any one microstate for steam is more probable than any one microstate for ice. It’s just that there are a lot, lot more microstates for steam. The percentage of microstates that take the form of steam is not 99% or 99.99%. It’s much, much closer than that to 100%. Under the U-V-N conditions that make those steam microstates accessible they will absolutely dominate at equilibrium.

What if we start away from equilibrium? Say we start our container with half ice and half steam by mass. But with the same U-V-N conditions for steam. So it has the same amount of energy. What will happen? The initial conditions won’t last. The ice will melt and boil until the system just flips among the vast number of microstates for steam. If the energy of the system remains constant it will never return to ice. Why? It’s not actually absolutely impossible in principle. But it’s just unimaginably improbable.

That’s what’s going on at the molecular level. Macroscopically entropy is a few levels removed from tangible, measured properties. What we see macroscopically are relations between heat flow, temperature, pressure, and volume. But we can calculate the change in entropy between states using various equations expressed in terms of these macroscopic properties that we can measure with our instruments.

For example, we can calculate the change in entropy of an ideal gas using the following equation:

Here s is entropy, cp is heat capacity at constant pressure, T is temperature, R is the ideal gas constant, and P is pressure. We can see from this equation that, all other things being equal, entropy increases with temperature and decreases with pressure. And this matches what we saw earlier. Recall that if the volume of a system of gas increases with a set quantity of material the energy difference between the energy levels decreases and there are more energy levels accessible for the same amount of energy. Under these circumstances pressure would decrease so entropy would decrease with pressure.

For solids and liquids we can assume that they are incompressible and leave off the pressure terms. So the change in entropy for a solid or liquid is given by the equation:

Let’s do an example with liquid water. What’s the change in entropy, and the increase in the number of accessible microstates, that comes from increasing the temperature of liquid water one degree Celsius? Let’s say we’re increasing 1 mole (18 grams) of water from 25 to 26 degrees Celsius. At this temperature the heat capacity of water is 75.3 J/mol-K.

Now that we have the increase in entropy we can find the increase in the number of microstates using the equation

Setting this equal to 0.252 J/mol-K

The increase is not as high as it was with phase changes, but it’s still a very big change.

We’ll wrap up the definition section here but conclude with some general intuitions we can gather from these equations and calculations:

1. All other things being equal, entropy increases with temperature.

2. All other things being equal, entropy decreases with pressure.

3. Entropy increases with phase changes from solid to liquid to gas.

Keeping these intuitions in mind will help as we move to applications with steam

Applications with Steam

The first two examples in this section are thermodynamic cycles. All thermodynamic cycles have 4 processes.

1. Compression

2. Heat addition

3. Expansion

4. Heat rejection

These processes circle back on each other so that the cycle can be repeated. Think, for example, of pistons in a car engine. Each cycle of the piston is going through each of these processes over and over again, several times per second.

There are many kinds of thermodynamic cycles. The idealized cycle is the Carnot cycle, which gives the upper limit on the efficiency of conversion from heat to work. Otto cycles and diesel cycles are the cyles used in gasoline and diesel engines. Our steam examples will be from the Rankine cycle. In a Rankine cycle the 4 processes take the following form:

1. Isentropic compression

2. Isobaric heat addition

3. Isentropic expansion

4. Isobaric heat rejection

An isobaric process is one that occurs at constant pressure. An adiabatic process is one that occurs at constant entropy.

An example of a Rankine cycle is a steam turbine or steam engine. Liquid water passes through a boiler, the steam passes through a turbine, expanding and turning the turbine, The fluid passes through a condenser, and then is pumped back to the boiler, where the cycle repeats. In such problems the fact that entropy is the same before and after expansion through the turbine reduces the number of unknown variables in our equations.

Let’s look at an example problem. Superheated steam at 6 MPa at 600 degrees Celsius expands through a turbine at a rate of 2 kg/s and drops in pressure to 10 kPa. What’s the power output from the turbine?

We can take advantage of the fact that the entropy of the fluid is the same before and after expansion. We just have to look up the entropy of superheated steam in a steam table. The entropy of steam at 6 MPa at 600 degrees Celsius is:

The entropy of the fluid before and after expansion is the same but some of it condenses. This isn’t good for the turbines but it happens nonetheless. Ideally, most of the fluid is still vapor so the ratio of the mass that is saturated vapor to the total fluid mass is called “quality”. The entropies of saturated liquid, sf, and of evaporation, sfg, are very different. So we can use algebra to calculate the quality, x2, of the fluid. The total entropy of the expanded fluid is given by the equation:

s2 we already know because the entropy of the fluid exiting the turbine is the same as that of the fluid entering the turbine. And we can look up the other values in steam tables.

Solving for quality we find that 

Now that we know the quality we can find the work output from the turbine. The equation for the work output of the turbine is:

h1 and h2 and enthalpies before and after expansion. If you’re not familiar with enthalpy don’t worry about it (we’re getting into enough for now). It roughly corresponds to the substance’s energy. We can look up the enthalpy of the superheated steam in a steam table.

For the fluid leaving the turbine we need to calculate the enthalpy using the quality, since it’s part liquid, part vapor. We need the enthalpy of saturated liquid, hf, and of evaporation, hfg. The total enthalpy of the fluid leaving the turbine is given by the formula

From the steam tables

So

And now we can plug this in to get the work output of the turbine.

So here’s an example where we used the value of entropy to calculate other observable quantities in a physical system. Since the entropy was the same before and after expansion we could use that fact to calculate the quality of the fluid leaving the turbine, use quality to calculate the enthalpy of the fluid, and use the enthalpy to calculate the work output of the turbine.

A second example.  Superheated steam at 2 MPa and 400 degrees Celsius expands through a turbine to 10 kPa. What’s the maximum possible efficiency from the cycle? Efficiency is work output divided by heat input. We have to input work as well to compress the fluid with the pump so that will subtract from the work output from the turbine. Let’s calculate the work used by the pump first. Pump work is:

Where v is the specific volume of water, 0.001 m3/kg. Plugging in our pressures in kPa:

So there’s our pump work input.

The enthalpy of saturated liquid is:

Plus the pump work input is:

Now we need heat input. The enthalpy of superheated steam at 2 MPa and 400 degrees Celsius is:

So the heat input required is:

The entropy before and after expansion through the turbine is the entropy of superheated steam at 2 MPa and 400 degrees Celsius is:

As in the last example, we can use this to calculate the quality of the steam with the equation:

Looking up these values in a steam table:

Plugging these in we get:

And

Now we can calculate the enthalpy of the expanded fluid.

And the work output of the turbine.

So we have the work input of the pump, the heat input of the boiler, and the work output of the turbine. The maximum possible efficiency is:

So efficiency is 32.32%.

Again, we used entropy to get quality, quality to get enthalpy, enthalpy to get work, and work to get efficiency. In this example we didn’t even need the mass flux of the system. Everything was on a per kilogram basis. But that was sufficient to calculate efficiency.

One last example with steam. The second law of thermodynamics has various forms. One form is that the entropy of the universe can never decrease. It is certainly not the case that entropy can never decrease at all. Entropy decreases all the time within certain systems. In fact, all the remaining examples in this episode will be cases in which entropy decreases within certain systems. But the total entropy of the universe cannot decrease. Any decrease in entropy must have a corresponding increase in entropy somewhere else. It’s easier to see this in terms of an entropy balance.

The entropy change in a system can be negative but the balance of the change in system entropy, entropy in, entropy out, and entropy of the surroundings will never be negative. We can look at the change of entropy of the universe as a function of the entropy change of a system and the entropy change of the system’s surroundings.

So let’s look at an example. Take 2 kg of superheated steam at 400 degrees Celsius and 600 kPa and condense it by pulling heat out of the system. The surroundings have a constant temperature of 25 degrees Celsius. From steam tables the entropy of the superheated steam and saturated steam are:

With these values we can calculate the change in entropy inside the system using the following equation;

The entropy decreases inside the system. Nothing wrong with this. Entropy can definitely decrease locally. But what happens in the surroundings? We condensed the steam by pulling heat out of the system and into the surroundings. So there is positive heat flow, Q, out into the surroundings. We can find the change in entropy in the surroundings using the equation:

We know the surroundings have a constant temperature, so we know T. We just need the heat flow Q. We can calculate the heat flow into the surroundings by calculating the heat flow out of the system using the equation

So we need the enthalpies of the superheated steam and saturated steam.

And plugging these in

Q = mΔh=(2)3270.2-670.6=5199 J

Now that we have Q we can find the change in entropy in the surroundings:

The entropy of the surroundings increases. And the total entropy change of the universe is:

So even though entropy decreases in the system the total entropy change in the universe is positive.

I like these examples with steam because they’re very readily calculable. The thermodynamics of steam engines have been extensively studied for over 200 years, with scientists and engineers gathering empirical data. So we have abundant data on entropy values for steam in steam tables. I actually think just flipping through steam tables and looking at the patterns is a good way to get a grasp on the way entropy works. Maybe it’s not something you’d do for light reading on the beach but if you’re ever unable to fall asleep you might give it a try.

With these examples we’ve looked at entropy for a single substance, water, at different temperatures, pressures, and phases, and observed the differences of the value of entropy at these different states. 

To review some general observations:

1. All other things being equal, entropy increases with temperature.

2. All other things being equal, entropy decreases with pressure.

3. Entropy increases with phase changes from solid to liquid to gas.

In the next section we’ll look at entropies for changing substances in chemical reactions.

Applications with Chemical Reactions

The most important equation for the thermodynamics of chemical reactions is the Gibbs Free Energy equation:

ΔG=ΔH-TΔS

Where H, T, S are enthalpy, temperature, and entropy. ΔG is the change in Gibbs free energy. Gibbs free energy is a thermodynamic potential. It is minimized when a system reaches chemical equilibrium. For a reaction to be spontaneous the value for ΔG has to be negative, meaning that during the reaction the Gibbs free energy is decreasing and moving closer to equilibrium.

We can see from the Gibbs free energy equation

ΔG=ΔH-TΔS

That the value of the change in Gibbs free energy is influenced by both enthalpy and entropy. The change in enthalpy tells us whether a reaction is exothermic (negative ΔH) or endothermic (positive ΔH). Exothermic reactions release heat while endothermic reactions absorb heat. This has to do with the total change in the chemical bond energies in all the reactants against all the products. In exothermic reactions the energy released from breaking chemical bonds is greater than the energy used to form new chemical bonds. This extra energy is converted to heat. We can see from the Gibbs free energy equation that exothermic reactions are more thermodynamically favored. Nevertheless, entropy can override enthalpy.

The minus sign in front of the TS term tells us that an increase in entropy where ΔS is positive will be more thermodynamically favored. This makes sense with what we know about entropy from the second law of thermodynamics and from statistical mechanics. The effect is proportional to temperature. At low temperatures entropy won’t have much influence and enthalpy will dominate. But at higher temperatures entropy will start to dominate and override enthalpic effects. This makes it possible for endothermic reactions to proceed spontaneously. If the increase in entropy for a chemical reaction is large enough and the temperature is high enough endothermic reactions can proceed spontaneously, even though the energy required to form the chemical bonds of the products is more than the energy released from the chemical bonds in the reactants.

Let’s look at an example. The chemical reaction for the production of water from oxygen and hydrogen is:

We can look up the enthalpies and entropies of the reactants and products in chemical reference literature. What we need are the standard enthalpies of formation and the standard molar entropies of each of the components.

The standard enthalpies of formation of oxygen and hydrogen are both 0 kJ/mol. By definition, all elements in their standard states have a standard enthalpy of formation of zero. The standard enthalpy of formation for water is -241.83 kJ/mol. The total change in enthalpy for this reaction is

It’s negative which means that the reaction is exothermic and enthalpically favored.

The standard molar entropies for hydrogen, oxygen, and water are, respectively, 130.59 J/mol-K, 205.03 J/mol-K, and 188.84 J/mol-K. The total change in entropy for this reaction is

It’s negative so entropy decreases in this reaction, which means the reaction is entropically disfavored. So enthalpy and entropy oppose each other in this reaction. Which will dominate depends on temperature? At 25 degrees Celsius (298 K) the change in Gibbs free energy is

The reaction is thermodynamically favored. Even though entropy is reduced in this reaction, at this temperature that effect is overwhelmed by the favorable reduction in enthalpy as chemical bond energy of the reactants is released as thermal energy.

Where’s the tradeoff point where entropy overtakes enthalpy? This is a question commonly addressed in polymer chemistry with what’s called the ceiling temperature. Polymers are macromolecules in which smaller molecular constituents called monomers are consolidated into larger molecules. We can see intuitively that this kind of molecular consolidation constitutes a reduction in entropy. It corresponds with the rough analogy of greater order from “disorder” as disparate parts are assembled into a more organized totality. And that analogy isn’t bad. So in polymer production it’s important to run polymerization reactions at temperatures where exothermic, enthalpy effects dominate. The upper end of this temperature range is the ceiling temperature.

The ceiling temperature is easily calculable from the Gibbs free energy equation for polymerization

Set ΔGp to zero.

And solve for Tc

At this temperature enthalpic and entropic effects are balanced. Below this temperature polymerization can proceed spontaneously. Above this temperature depolymerization can proceed spontaneously.

Here’s an example using polyethylene. The enthalpies and entropies of polymerization for polyethylene are

Using our equation for the ceiling temperature we find

So for a polyethylene polymerization reaction you want to run the reaction below 610 degrees Celsius so that the exothermic, enthalpic benefit overcomes your decrease in entropy.

Conclusion

A friend and I used to get together on weekends to take turns playing the piano, sight reading music. We were both pretty good at it and could play songs reasonably well on a first pass, even though we’d never played or seen the music before. One time when someone was watching us she asked, “How do you do that?” My friend had a good explanation I think. He explained it as familiarity with the patterns of music and the piano. When you spend years playing songs and practicing scales you just come to know how things work. Another friend of mine said something similar about watching chess games. He could easily memorize entire games of chess because he knew the kinds of moves that players would tend to make. John Von Neumann once said: “In mathematics you don’t understand things. You just get used to them.” I would change that slightly to say that you understand things by getting used to them. Also true for thermodynamics. Entropy is a complex property and one that’s not easy to understand. But I think it’s easiest to get a grasp on it by using it.

Evolutionary Biology With Molecular Precision

Evolutionary biology benefits from a non-reductionist focus on real biological systems at the macroscopic level of their natural and historical contexts. This high-level approach makes sense since selection pressures operate at the level of phenotypes, the observed physical traits of organisms. Still, it is understood that these traits are inherited in the form of molecular gene sequences, the purview of molecular biology. The approach of molecular biology is more reductionist, focusing at the level of precise molecular structures. Molecular biology thereby benefits from a rigorous standard of evidence-based inference by isolating variables in controlled experiments. But it necessarily sets aside much of the complexity of nature. A combination of these two, in the form of evolutionary biochemistry, targets a functional synthesis of evolutionary biology and molecular biology, using techniques such as ancestral protein reconstruction to physically ‘resurrect’ ancestral proteins with precise molecular structures and to observe their resulting expressed traits experimentally.

I love nerdy comics like XKCD and Saturday Morning Breakfast Cereal (SMBC). For the subject of this episode I think there’s a very appropriate XKCD comic. It shows the conclusion of a research paper that says, “We believe this resolves all remaining questions on this topic. No further research is needed.” And the caption below it says, “Just once, I want to see a research paper with the guts to end this way.” And of course, the joke is that no research paper is going to end this way because further research is always needed. I’m sure this is true in all areas of science but I think two particular fields it’s especially true. One is in neuroscience, where there is still so much that we don’t know. And the other is evolutionary biology. The more I dig into evolutionary biology the more I appreciate how much we don’t understand. And that’s OK. The still expansive frontiers in each of these fields is what makes them especially interesting to me. Far from being discouraging, unanswered questions and prodding challenges should be exciting. With this episode I’d like to look at evolutionary biology at its most basic, nuts-and-bolts level at the level of chemistry. This combines the somewhat different approaches of both evolutionary biology and molecular biology.

Evolutionary biology benefits from a non-reductionist focus on real biological systems at the macroscopic level of their natural and historical contexts. This high-level approach makes sense since selection pressures operate at the level of phenotypes, the observed physical traits of organisms. Still, it is understood that these traits are inherited in the form of molecular gene sequences, the purview of molecular biology. The approach of molecular biology is more reductionist, focusing at the level of precise molecular structures. Molecular biology thereby benefits from a rigorous standard of evidence-based inference by isolating variables in controlled experiments. But it necessarily sets aside much of the complexity of nature. A combination of these two, in the form of evolutionary biochemistry, targets a functional synthesis of evolutionary biology and molecular biology, using techniques such as ancestral protein reconstruction to physically ‘resurrect’ ancestral proteins with precise molecular structures and to observe their resulting expressed traits experimentally. This enables evolutionary science to be more empirical and experimentally grounded.

In what follows I’d like to focus on the work of biologist Joseph Thornton, who is especially known for his lab’s work on ancestral sequence reconstruction. One review paper of his that I’d especially recommend is his 2007 paper, Mechanistic approaches to the study of evolution: the functional synthesis, published in Nature and co authored with Antony Dean.

Before getting to Thornton’s work I should mention that Thornton has been discussed by biochemist Michael Behe, in particular in his fairly recent 2019 book Darwin Devolves: The New Science About DNA That Challenges Evolution. Behe discusses Thornton’s work in the eighth chapter of that book. I won’t delve into the details of the debate between the two of them, simply because that’s it’s own topic and not what directly interests me here. But I’d just like to comment that I personally find Behe’s work quite instrumentally useful to evolutionary science. He’s perceived as something of a nemesis to evolutionary biology but I think he makes a lot of good points. I could be certainly wrong about this but I suspect that many of the experiments I’ll be going over in this episode were designed and conducted in response to Behe’s challenges to evolutionary biology. Maybe these kinds of experiments wouldn’t have been done otherwise. And if that’s the case Behe has done a great service. 

Behe’s major idea is “irreducible complexity”. An irreducibly complex system is “a single system which is composed of several well-matched, interacting parts that contribute to the basic function, and where the removal of any one of the parts causes the system to effectively cease functioning.” (Darwin’s Black Box: The Biochemical Challenge to Evolution) How would such a system evolve by successive small modifications if no less complex a system would function? That’s an interesting question. And I think that experiments designed to answer that question are quite useful.

Behe and I are both Christians and we both believe that God created all things. But we have some theological and philosophical differences. My understanding of the natural and supernatural is heavily influenced by the thought of Thomas Aquinas, such that in my understanding nature is actually sustained and directed by continual divine action. I believe nature, as divine creation, is rationally ordered and intelligible, since it is a product of divine Mind. As such, I expect that we should, at least in principle, be able to understand and see the rational structure inherent in nature. And this includes the rational structure and process of the evolution of life. Our understanding of it may be miniscule. But I think it is comprehensible at least in principle. Especially since it is comprehensible to God. So I’m not worried about a shrinking space for some “god of the gaps”. Still, I think it’s useful for someone to ask probing questions at the edge or our scientific understanding, to poke at our partial explanations and ask, “how exactly?” But, perhaps different from Behe, I expect that we’ll continually be able to answer such questions better and better, even if there will always be a frontier of open questions and problems.

With complete admission that what I’m about to say is unfair, I do think that some popular understanding of evolution lacks a certain degree of rigor and doesn’t adequately account for the physical constraints of biochemistry. Evolution can’t just proceed in any direction to develop any trait to fill any adaptive need, even if there is a selection pressure for a trait that would be nice to have. OK, well that’s why it’s popular rather than academic, right? Like I said, not really fair. Still, let’s aim for rigor, shall we? Behe gets at this issue in his best known 1996 book Darwin’s Black Box: The Biochemical Challenge to Evolution. In one passage  he comments on what he calls the “fertile imaginations” of evolutionary biologists:

“Given a starting point, they almost always can spin a story to get to any biological structure you wish. The talent can be valuable, but it is a two edged sword. Although they might think of possible evolutionary routes other people overlook, they also tend to ignore details and roadblocks that would trip up their scenarios. Science, however, cannot ultimately ignore relevant details, and at the molecular level all the ‘details’ become critical. If a molecular nut or bolt is missing, then the whole system can crash. Because the cilium is irreducibly complex, no direct, gradual route leads to its production. So an evolutionary story for the cilium must envision a circuitous route, perhaps adapting parts that were originally used for other purposes… Intriguing as this scenario may sound, though, critical details are overlooked. The question we must ask of this indirect scenario is one for which many evolutionary biologists have little patience: but how exactly?”

“How exactly?” I actually think that’s a great question. And I’d say Joseph Thornton has made the same point to his fellow biologists, maybe even in response to Behe. In the conclusion of their 2007 paper he and Antony Dean had this wonderful passage:

“Functional tests should become routine in studies of molecular evolution. Statistical inferences from sequence data will remain important, but they should be treated as a starting point, not the centrepiece or end of analysis as in the old paradigm. In our opinion, it is now incumbent on evolutionary biologists to experimentally test their statistically generated hypotheses before making strong claims about selection or other evolutionary forces. With the advent of new capacities, the standards of evidence in the field must change accordingly. To meet this standard, evolutionary biologists will need to be trained in molecular biology and be prepared to establish relevant collaborations across disciplines.”

Preach it! That’s good stuff. One of the things I like about the conclusion to their paper is that it talks about all the work that still needs to be done. It’s a call to action (reform?) to the field of evolutionary biology. 

Behe has correctly pointed out that their research doesn’t yet answer many important questions and doesn’t reduce the “irreducible complexity”. True, but it’s moving in the right direction. No one is going to publish a research paper like the one in the XKCD comic that says, “We believe this resolves all remaining questions on this topic. No further research is needed.” Nature and evolution are extremely complex. And I think it’s great that Thornton and his colleagues call for further innovations. For example, I really like this one:

“A key challenge for the functional synthesis is to thoroughly connect changes in molecular function to organismal phenotype and fitness. Ideally, results obtained in vitro should be verified in vivo. Transgenic evolutionary studies identifying the functional impact of historical mutations have been conducted in microbes and a few model plant and animal species, but an expanded repertoire of models will be required to reach this goal for other taxa. By integrating the functional synthesis with advances in developmental genetics and neurobiology, this approach has the potential to yield important insights into the evolution of development, behaviour and physiology. Experimental studies of natural selection in the laboratory can also be enriched by functional approaches to characterize the specific genetic changes that underlie the evolution of adaptive phenotypes.”

For sure. That’s exactly the kind of work that needs to be done. And it’s the kind of work Behe has challenged evolutionary biologists to do. I think that’s great. Granted, that kind of work is going to be very difficult and take a long time. But that’s a good target. And we should acknowledge the progress that has been made. For example, earlier in the paper they note:

“The Reverend William Paley famously argued that, just as the intricate complexity of a watch implies a design by a watchmaker, so complexity in Nature implies design by God. Evolutionary biologists have typically responded to this challenge by sketching scenarios by which complex biological systems might have evolved through a series of functional intermediates. Thornton and co-workers have gone much further: they have pried open the historical and molecular ‘black box’ to reconstruct in detail — and with strong empirical support — the history by which a tightly integrated system evolved at the levels of sequence, structure and function.”

Yes. That’s a big improvement. It’s one thing to speculate, “Well, you know, maybe this, that, and the other” (again, being somewhat unfair, sorry). But it’s another thing to actually reconstruct ancestral sequences and run experiments with them. That’s moving things to a new level. And I’ll just mention in passing that I do in fact think that all the complexity in Nature was designed by God. And I don’t think that reconstructing that process scientifically does anything to reduce the grandeur of that. If anything, such scientific understanding facilitates what Carl Sagan once called “informed worship” (The Varieties of Scientific Experience: A Personal View of the Search for God). 

With all that out of the way now, let’s focus on Thornton’s very interesting work in evolutionary biochemistry.

First, a very quick primer on molecular biology. The basic process of molecular biology is that DNA makes RNA, and RNA makes proteins. Living organisms are made of proteins. DNA is the molecule that contains the information needed to make the proteins. And RNA is the molecule that takes the information from DNA to actually make the proteins. The process of making RNA from DNA is called transcription. And the process of making proteins from RNA is called translation. These are very complex and fascinating processes. Evolution proceeds through changes to the DNA molecule called mutations. And some changes to DNA result in changes to the composition and structure of proteins. These changes can have macroscopically observable effects.

In Thornton’s work with ancestral sequence reconstruction the idea is to look at a protein as it is in an existing organism, try to figure out what that protein might have been like in an earlier stage of evolution, and then to make it. Reconstruct it. By actually making the protein you can look at its properties. As described in the 2007 Nature article:

“Molecular biology provides experimental means to test these hypotheses decisively. Gene synthesis allows ancestral sequences, which can be inferred using phylogenetic methods, to be physically ‘resurrected’, expressed and functionally characterized. Using directed mutagenesis, historical mutations of putative importance are introduced into extant or ancestral sequences. The effects of these mutations are then assessed, singly and in combination, using functional molecular assays. Crystallographic studies of engineered proteins — resurrected and/or mutagenized — allow determination of the the structural mechanisms by which amino-acid replacements produce functional shifts. Transgenic techniques permit the effect of specific mutations on whole-organism phenotypes to be studied experimentally. Finally, competition between genetically engineered organisms in defined environments allows the fitness effects of specific mutations to be assessed and hypotheses about the role of natural selection in molecular evolution to be decisively tested.”

What’s great about this kind of technique is that it spans a number of levels of ontology. Evolution by natural selection acts on whole-organism phenotypes. So it’s critical to understand what these look like between all the different versions of a protein. We don’t just want to know that we can make all these different kinds of proteins. We want to know what they do, how they function. Function is a higher-level ontology. But we also want to be precise about what is there physically. And we have that as well, down to the molecular level. Atom for atom we know exactly what these proteins are.

To dig deeper into these experimental methods I’d like to refer to another paper, Evolutionary biochemistry: revealing the historical and physical causes of protein properties, published in Nature in 2013 by Michael Harms and Joseph Thornton. In this paper the authors lay out three strategies for studying the evolutionary trajectories of proteins.

The first strategy is to explicitly reconstruct “the historical trajectory that a protein or group of proteins took during evolution.”

“For proteins that evolved new functions or properties very recently, population genetic analyses can identify which genotypes and phenotypes are ancestral and which are derived. For more ancient divergences, ancestral protein reconstruction (APR) uses phylogenetic techniques to reconstruct statistical approximations of ancestral proteins computationally, which are then physically synthesized and experimentally studied… Genes that encode the inferred ancestral sequences can then be synthesized and expressed in cultured cells; this approach allows for the structure, function and biophysical properties of each ‘resurrected’ protein to be experimentally characterized… By characterizing ancestral proteins at multiple nodes on a phylogeny, the evolutionary interval during which major shifts in those properties occurred can be identified. Sequence substitutions that occurred during that interval can then be introduced singly and in combination into ancestral backgrounds, allowing the effects of historical mutations on protein structure, function and physical properties to be determined directly.”

This first strategy is a kind of top-down, highly directed approach. We’re trying to follow exactly the path that evolution followed and only that path to see what it looks like.

The second strategy is more bottom-up. It is “to use directed evolution to drive a functional transition of interest in the laboratory and then study the mechanisms of evolution.” The goal is not primarily to follow the exact same path that evolution followed historically but rather to stimulate evolution, selecting for a target property, to see what path it follows. 

“A library of random variants of a protein of interest is generated and then screened to recover those with a desired property. Selected variants are iteratively re-mutagenized and are subject to selection to optimize the property. Causal mutations and their mechanisms can then be identified by characterizing the sequences and functions of the intermediate states realized during evolution of the protein.”

If the first strategy is top-down and the second strategy is bottom-up, the third strategy is to cast a wide net. “Rather than reconstructing what evolution did in the past, this strategy aims to reveal what it could do.” In this approach:

“An initial protein is subjected to random mutagenesis, and weak selection for a property of interest is applied, enriching the library for clones with the property and depleting those without it. The population is then sequenced; the degree of enrichment of each clone allows the direct and epistatic effects of each mutation on the function to be quantitatively characterized.”

Let’s look at an example from Thornton’s work, which followed the first, top-down approach. The most prominent work so far has been on the evolution of glucocorticoid receptors (GRs) and mineralocorticoid receptors (MRs). See for example the 2006 paper Evolution of Hormone-Receptor Complexity by Molecular Exploitation, published in Science by Jamie Bridgham, Sean Carroll, and Joseph Thornton.

Glucocorticoid receptors and mineralocorticoid receptors bind with glucocorticoid and mineralocorticoid steroid hormones. The two steroid hormones studied in Thornton’s work are cortisol and aldosterone. Cortisol activates the glucocorticoid receptor to regulate metabolism, inflammation, and immunity. Aldosterone activates the mineralocorticoid receptor to regulate electrolyte homeostasis of plasma sodium and potassium levels. Glucocorticoid receptors and mineralocorticoid receptors share common origin and Thornton’s work was to reconstruct ancestral versions of these proteins along their evolutionary path and test their properties experimentally.

Modern mineralocorticoid receptors can be activated by both aldosterone and cortisol but modern glucocorticoid receptors are activated only by cortisol in bony vertebrates. So in their evolution GRs developed an insensitivity to aldosterone.

The evolutionary trajectory is as follows. There are versions of MR and GR extant in tetrapods, teleosts (fish), and elasmobranchs (sharks). GRs and MRs trace back to a common protein from 450 million years ago, the ancestral corticoid receptor (AncCR). The ancestral corticoid receptor is thought to have been activated by deoxycorticosterone (DOC), the ligand for MRs in extant fish.

Phylogeny tells us that the ancestral corticoid receptor gave rise to GR and MR in a gene-duplication event. Interestingly enough this was before aldosterone had even evolved. In tetrapods and teleosts, modern GR is only sensitive to cortisol; it is insensitive to aldosterone.

Thornston and his team reconstructed the ancestral corticoid receptor (AncCR) and found that it is sensitive to DOC, cortisol, and aldosterone. Most phylogenetic analysis revealed that precisely two mutations, amino acid substitutions, resulted in the glucocorticoid receptor phenotype: aldosterone insensitivity and cortisol sensitivity. These amino acid substitutions are S106P, from serine to proline at site 106, and L111Q, from leucine to glutamine at site 111. Thornston synthesized these different proteins to observe their properties. The protein with just the L111Q mutation did not bind to any of the ligands: DOC, cortisol, or aldosterone. So it is unlikely that the L111Q mutation would have occurred first. The S106P mutation reduces aldosterone and cortisol sensitivity but it remains highly DOC-sensitive. With both the S106P and L111Q mutations in series aldosterone sensitivity is reduced even further but cortisol sensitivity is restored to levels characteristic of extant GRs. A mutational path beginning with S106P followed by L111Q thus converts the ancestor to the modern GR phenotype by functional intermediate steps and is the most likely evolutionary scenario.

Michael Behe has commented that this is an example of a loss of function whereas his challenge to evolutionary biology is to demonstrate how complex structures evolved in the first place. That’s a fair point. Still, this is a good example of the kind of molecular precision we can get in our reconstruction of evolutionary processes. This does seem to show, down to the molecular level, how these receptors evolved. And that increases our knowledge. We know more about the evolution of these proteins than we did before. That’s valuable. We can learn a lot more in the future using these methods and applying them to other examples. 

One of the things I like about this kind of research is that it not only shows what evolutionary paths are possible but also which ones are not. Another one of Thornton’s papers worth checking out is An epistatic ratchet constrains the direction of glucocorticoid receptor evolution, published in Nature in 2009, co-authored by Jamie Bridgham and Eric Ortlund. The basic idea is that in certain cases once a protein acquires a new function “the evolutionary path by which this protein acquired its new function soon became inaccessible to reverse exploration”. In other words, certain evolutionary processes are not reversible. This is similar to Dollo’s Law of Irreversibility, proposed in 1893: “an organism never returns exactly to a former state, even if it finds itself placed in conditions of existence identical to those in which it has previously lived … it always keeps some trace of the intermediate stages through which it has passed.” In their 2009 paper Harms and Thornton and  state: “We predict that future investigations, like ours, will support a molecular version of Dollo’s law: as evolution proceeds, shifts in protein structure-function relations become increasingly difficult to reverse whenever those shifts have complex architectures, such as requiring conformational changes or epistatically interacting substitutions.”

This is really important. It’s important to understand that evolution can’t just do anything. Nature imposes constraints both physiologically and biochemically. I think in some popular conceptions we imagine that “life finds a way” and that evolution is so robust that organisms will evolve whatever traits they need to fit their environments. But very often they don’t, and they go extinct. And even when they do, their evolved traits aren’t necessarily perfect. Necessity or utility can’t push evolution beyond natural constraints. A good book on the subject of physiological constraints on evolution is Alex Bezzerides’s 2021 book Evolution Gone Wrong: The Curious Reasons Why Our Bodies Work (Or Don’t). Our anatomy doesn’t always make the most sense. It’s possible to imagine more efficient ways we could be put together. But our evolutionary history imposes constraints that don’t leave all options open, no matter how advantageous they would be. And the same goes for biochemistry. The repertoire of proteins and nucleic acids in the living world is determined by evolution. But the properties of proteins and nucleic acids are determined by the laws of physics and chemistry.

One way to think about this is with a protein sequence space. This is an abstract multidimensional space. Michael Harms and Joseph Thornton describe this in their 2013 paper.

“Sequence space is a spatial representation of all possible amino acid sequences and the mutational connections between them. Each sequence is a node, and each node is connected by edges to all neighbouring proteins that differ from it by just one amino acid. This space of sequences becomes a genotype–phenotype space when each node is assigned information about its functional or physical properties; this representation serves as a map of the total set of relations between sequence and those properties. As proteins evolve, they follow trajectories along edges through the genotype–phenotype space.”

What’s crucial to consider in this kind of model is that most nodes are non-functional states. This means that possible paths through sequence space will be highly constrained. Not just any path is possible. There may be some excellent nodes in the sequence space that would be perfect for a given environment. But if they’re not connected to an existing node via a path through functional states they’re not going to occur through evolution.

To conclude, it’s an exciting time for the evolutionary sciences. If you compare our understanding of the actual physical mechanisms for inheritance and evolution, down to the molecular level we are leaps and bounds ahead of where we were a century ago. Darwin and his associates had no way of knowing the kinds of things we know now about the structures of nucleic acids and proteins. This makes a big difference. It’s certainly not the case that we have it all figured out. That’s why I put evolutionary biology in the same class as neuroscience when it comes to what we understand compared to how much there is to understand. We’re learning more and more all the time just how much we don’t know. But that’s still progress. We are developing the tools to get very precise and detailed in what we can learn about evolution.