The Grammar of Meaning Systems:

Sign Systems, Symbolic Cognition, and Semiotics

Martin Irvine

Communication, Culture & Technology Program

Georgetown University

[Drafts of  book chapters in progress; for reference only.]

“A sign is something by knowing which we know something more”

“The whole universe is perfused with signs.” --C. S. Peirce

1.  Humans as the Symbolic Species:

Symbolic Cognition as the Core Human Operating System (OS Alpha)

This introduction provides a brief conceptual orientation and synthesis of current research on symbolic thought and sign systems with a minimum of technical vocabulary, which will be introduced in the context of specific arguments, research, and theory (see bibliography for backgrounds). For consistency, we will use the key concepts and terminology developed by C. S. Peirce and by contemporary linguists who approach language and symbolic thought as an ongoing generative process in rule-governed combinations of signs.

Semiotics (from the Greek word, semeion, sign, mark; Latin: signum) is the general discipline devoted to the study of signs and symbols as systems of expression, meaning, reasoning, and cultural memory in any kind of artefact (human-made thing). We’ll begin with a brief inventory of human symbolic functions and sign systems that we know about with reasonable, non-controversial certainty. Developing integrative, generalizable explanations that account for both sign-system specific features and functions (e.g., language and images as symbolic systems) and the rule-governed patterns that link complex and compound symbol structures (e.g., kinds of discourse, music genres, film/video and multimedia genres) in all their forms of technical mediation is the hardest problem for all the sciences and disciplines concerned with these questions.

Our approach will be building conceptual models for testable hypotheses, and see how far they can be developed for understanding how human communication, symbolic representation, and meaning-making work. This means we will be thinking with models (both discursive and formal) that are open to revision, knowledge-checking, and redefinition. There are many rival schools of thought on all topics in this field of study, and many approaches that treat a body of theory as a doctrine to be followed, rather than as tools to think with. We will not seek to follow or establish a doctrine, but develop the best and clearest tools for thought and models for meaning processes that we can assemble from an interrelated knowledge base across many disciplines.

Our Core Human Operating System: Symbolic Cognition (OS Alpha)

The “mind/brain-as/is computer” metaphor is now commonly used in philosophy, linguistics, cognitive science, and AI to model and explain how minds/brains process perceptions, information, language, and meaning. We’ll explore how this metaphor is used, but we can also reverse it:  the universal human capacity for symbolic cognition (in Peirce’s sense) is the core human operating system (let’s call it OS A, Alpha) that has enable all our sign systems and technologies for mediating them possible. This means there is a continuum in human symbolic capabilities (which include abstractions and combinatorial rules) that have enabled everything from writing and imposing symbolic forms on two-dimensional media to computing. As we’ll see, computation is a model and method for implementing automated symbolic processes in the designs for a physical system that can do interpretive sequences through steps in time. Everything we experience and interpret in culture and represent in media and computing technologies is, in various ways and levels, structures and artefacts dependent on, emerging from, “running” on, our core OS.

Human culture, social relations, and technologies are thus inseparable from our interrelated cluster of symbolic systems (structured perceptible-cognitive systems for meaning representation like language, music, and images) and the daily activation of symbolic functions in every technical form of media and communication (termed semiotic or symbolic resources).

The interdisciplinary study of human symbolic thought, expression, and representation in sign systems now provides a rich knowledge base with important findings.  Over the past fifty years, a rough consensus has emerged in many sciences and disciples for understanding human language, symbolic thought, social organization, forms of media, and technologies as co-constitutive (depending on and requiring each other). A leading hypothesis in cognitive science and human evolution states that language and symbolic cognition have been part of a human species-specific co-evolution directly connected to the  evolution of societies, tools, communication  media, and technologies. The full story of how and why humans developed symbol-dependent societies and cultures is an open question for ongoing research in many fields, but we have foundational research and testable theory for making new advances in knowledge on the central question of human meaning-making in symbolic structures and technologies for meaning.

An important principle for research in this field is widely recognized: how meaning happens -- how we make, communicate, or “intend” meaning -- when we use symbols, signs, and representations is unobservable (not available to either conscious awareness or neurological testing). We have to work from observable “inputs” and “outputs” -- the expressions, material media, social situations and uses, and technologies for representation and transmission -- to build out models for what must be necessary to account for what we can observe. (This is the scientific method of making inferences by induction, and an additional form of inference that Peirce called abduction, making creative leaps in explanatory hypotheses that can be looped back for testing and application to new cases.) Symbolic cognition and sign/symbol processes are like a black box with observable inputs and outputs, and we can use sign/symbol representations and expressions as interfaces to the whole system that makes them work the way we experience them to work. Signs and symbols, then, are not so much things to isolate and study, but are the physical-material interfaces being used by our OS Alpha, symbolic cognition.

We strain for adequate descriptive terms and conceptual metaphors to explain what happens in all our meaning-making processes -- all the ordinary, day-to-day thinking and expression in words, images, sounds, and artefacts, the meanings of which seem so automatic, transparent, immediate, and beyond measurement. We have to catch ourselves in the act of making meaning in signs and symbols using the very structures of meaning provided by symbolic thought and signs themselves. Sign systems and symbolic thought are thus fundamentally reflexive and self-reflexive (we can use them to refer back to themselves at different conceptual levels): we can always go “meta” and describe what goes on in any system of signs with sets of signs from the same system. A metalanguage is a “language”--a defined set of terms--about language. Metadata is data for describing data. We also use all kinds of complex intertextual and intermedial genres for making interpretations and commentaries that “translate” one form into another (like making a movie of a novel, writing a commentary on an artwork, or writing a news article or essay that assumes, and critiques, many other writings and sources of information). These ordinary observable and describable symbolic activities – the ability for abstraction, self-reference, reflexivity, and translation across symbolic systems – all provide important clues for what must be happening in the unobservable “black box” side of our symbolic capacity.

What Is Included in the Study of Sign Systems?

C. S. Peirce’s life-time project (and incomplete when he died in 1914) was working out a generalizable model for a unified view of all kinds of sign uses and symbolic artefacts:

[The term] "sign" [includes] every picture, diagram, natural cry, pointing finger, wink, knot in one's handkerchief, memory, dream, fancy, concept, indication, token, symptom, letter, numeral, word, sentence, chapter, book, library, and in short whatever, be it in the physical universe, be it in the world of thought, that, whether embodying an idea of any kind (and permit us throughout to use this term to cover purposes and feelings), or being connected with some existing object, or referring to future events through a general rule, causes something else, its interpreting sign, to be determined to a corresponding relation to the same idea, existing thing, or law….

...

It is necessary to insist upon the point for the reason that ideas cannot be communicated at all except through their physical effects. Our photographs, telephones, and wireless telegraphs, as well as the sum total of all the work that steam engines have ever done, are, in sober common sense and literal truth, the outcome of the general ideas that are expressed in the first book of the Novum Organum.^[1] (MS 774, 1904, EP 2.326).

Similarly, in 1909 he wrote that his semiotic method “considers Signs in general, a class which includes pictures, symptoms, words, sentences, books, libraries, signals, orders of command, microscopes, legislative representatives, musical concertos, performances of these" (MS 634:17-18, 1909). We will follow his lead and study the cumulative sign and symbol functions that have enabled the designs for our contemporary representational media (whether analog or digital), computation, software, and computer networks. In his own day, Peirce saw that all technical mediations of representational systems--photographs, sensor instruments, telegraph and telephone, calculating and logic machines (early proto-computers)-- are implementations of the physical-perceptible and meaning-correlating structures of all signs and symbols as they are used for thought, communication, and social identity. Since all signs and symbols (from spoken sounds to recorded media) require a physical-perceptible component, they are interfaces to the ideas that motivate them and to the system of signs in which they will be received and interpreted.

Symbolic Cognition and Material Sign Systems: The Human Advantage

Humans as cognitive beings with a symbolic faculty and multiple sign systems have major advantages over life forms that only sense and respond instinctually. (Other animal species have forms of communication, and at very basic level, sign, signal, and information processes pervade the physical universe, but we will bracket off these broader dimensions for now.) We not only perceive with our senses in brains adapted to immediate awareness of the world around us, we make mental relations between perceptions and thought, and generate further relations among thoughts connected in vast networks of collectively understood signs spanning many states of time. After acquiring language and experiencing symbolic structures in images, objects, and sounds in a culture (by the age of 4), we also somehow know that things can mean something beyond their materiality as things or the mere perception of what strikes our eyes, ears, and other sense organs. This ability is termed semiotic competence. We readily distinguish between those kinds of things to be understood as indicating, standing for, or signifying something else from those kinds things perceived as simply as objects.^[2] 

In developing our linguistic and symbolic competencies, we also discover that all the many forms of  “sign vehicles” or “signifiers” used in our society function as perceptible “meaning mediators” for encoding thoughts, intentions, beliefs, and values in ways communicable with others.  Entering language -- and crossing the symbolic threshold -- means entering a world of meanings that are always collective, intersubjective, public, and interpersonally shared and communicable. (We check in with others on our participation in meaning-making all the time in phrases like: “You know what I mean…?”, “In other words...”, “I don’t get it”, or even “Huh?” -- which all presuppose that meanings, to be “real,” are public and shared, not private.)

Technologies of Meaning: How We Mediate Symbolic Cognition

A second major advantage of symbolic cognition distributed through multiple sign systems is the ability -- now a cultural necessity -- to represent symbolic expression in material forms through technical mediation. This is the symbolic- cognitive “trick” that we in human societies have been doing for at least 50,000 years when we off-load and externalize collectively shared symbolic states onto artefacts (writing, images, representations, 3D forms) and systems of technical mediation (everything from drawing, printing, photography, and broadcast media to audio/video recordings, computer code and computer memory, digitization, and digital networks). By externalizing our symbolic functions into technical memory systems like writing, recordings, art works, computation, and the whole built environment, we are able to “store and forward” symbolic thought from one generation to many others, enabling a cumulative cultural “ratchet effect,” the cumulative development that often goes by the name of “progress.”

These well-established principles, widely accepted in anthropology, archaeology, and cognitive science,  allow us to understand many technologies as “stored value” memory systems, providing conceptual models for generating new combinations by others. Our current “technologies of meaning” have not come down to us as outcomes of inevitable, deterministic, or irresistible forces in linear, causal histories, but, rather,  as cumulative developments emerging from the stored memory of previous states of knowledge and symbolic expression, and developments in other technologies that make new combinations possible. As we will see, our current and newer cognitive technologies are cumulative summations of many prior technologies--their states, material base, and conditions for enabling the memory and transmission of culture, science, and social identity--that we take for granted but “cash out” every day in all our recent accelerating technical advancements in computation and digital media.

Further, all this symbolic meaning-making activity is a dynamic, living, emergent process, adapted to living in “real-time”, managing individual and social memory, and acquiring new knowledge communicable to others. The generative process in symbolic thought enables us to learn and form abstractions and generalized concepts in semantic networks organized in hierarchies of categories and nested layers of symbolically representable concepts.  Our symbolic repertoires allow us to connect present-time perceptions with cognitive information in memory into patterns of meaning or understanding, patterns expressible only in further sequences of signs. Digitization of all forms of media is thus a recent cumulative step in this ongoing symbolic state-shifting that we’ve been doing for over 50,000 years.

Symbolic Cognition and Symbols:
A Way to Manage Conceptual Abstraction, Memory, Time,  and Continuities

Signs and symbolic cognition have many important functions, and one often missed in the textbooks is the cognitive ability to manage living in time (representing and connecting past, present, and future in a lived continuum), and to conceptualize time as a function in many other human social processes (e.g., interacting with short and long-term memory in artefacts, and externalizing or “off-loading” memory and states of meaning in all forms of media). The living-in-time reality is central to C. S. Peirce’s model of symbolic activity, but was totally missed by Ferdinand de Saussure and structuralist thought in general. We use language, written signs, images, and artefacts for daily, “real-time” functions, and register them in material mediations as externalized memory for recall and symbolic reactivation.

These “design features” of symbolic cognition lead many specialists in the cognitive sciences to hypothesize that symbolic cognition, especially in language, enables us to maintain continuous, connected, present conscious states that are social and collective. Our symbolic capacities enable us to recursively project mental representations of meaning forward in our “stream of consciousness,” the continuous present that we manage in “working memory,” in ways that are intersubjective and socially embedded, not private and individual.  Understanding, explaining, accounting for, and describing all this ordinarily transparent, automatic, non-self-conscious symbolic activity is one of the main tasks of semiotics and many other branches of cognitive science.

The diagram below maps out what we know about the main features of human symbolic activity, though how these capabilities and functions are interconnected in core brain processes (on the biological-neurological level) and enable complex cultural cross-symbolic activity (on the collective, inter-personal, and distributed levels) remains one of the most difficult questions in all sciences and disciplines.

All of these symbolic, meaning-making functions are not private or specific to an individual brain or a personal network of cognition, but are fundamentally inter-individual, intersubjective, shared, public, and collective. Meanings and values learned and discovered in our sign systems are not “located” “in” anyone’s head or any-where, but are cognitive-social events that we activate and instantiate by using our “cognitive-symbolic resources.” As we will see, we can access distributed meaning resources in different situations of use, including all the languages we inhabit, other people’s expressions, memories, and communities of knowledge, writing and representational systems, artefacts, media representations, and our entire built environment. We also activate the meaning functions in these collective, mediated meaning resources as forms of distributed cognition and distributed agency, all the forms of interdependent social-cultural/material-technical structures in the human-built world.

For symbol systems to work as they do as intersubjective meaning generators, each kind of sign is based on, requires by necessity, a perceptible-material component (sometimes called a “sign vehicle,” a perceptible and intelligible component) recognizable by all members of a community using the sign system. Our collective symbolic cognition is necessarily structured in material, inter-perceptible forms (examples: the discrete sound units in a spoken language, visual elements in writing systems, image features, sound qualities and patterns in music, etc.). These combined cognitive-material sign systems function within, and as, distributed networks of social relations and mediations.

Because the meaning-making structures of our symbolic faculties are formed in many layers and distributed networks of agency, they are not usefully represented in containers, boxes, or static points like those pictured in older communication and information diagrams:

Meanings are not something transferred or transmitted from one location to another, or from one container (someone’s head) to another. Using any shared sign/symbol system involves making correlations between patterns recognized in perception and  the kinds of meanings possible in our learned collective meaning networks. (We’ll develop this central idea further with more background.)

“Networks of social mediation” are implemented in externalized cognition or “cognitive technologies” that embody our sign systems and make them public, open-ended intersubjective resources. The distributed system of symbolic functions form a structure with positions already in place for individuals to become “thinking subjects,” cognitive functions in which we as individuals learn, enact, implement, and instantiate the resources of our symbolic faculty.

To make this difficult concept intuitively clear, let’s use the example of personal pronouns in a language. Think through what happens when anyone learns a language and how to use the simple pronouns “I” or “you.” These words have no meaning outside a context of expression for language-using subjects: the meaning of “I” is simply “the person currently uttering the phrase in the first person subject position.” And we can flip “I” and “you” (and “we” and “they”) as alternating, reciprocal positions in dialogue and all forms of symbolic exchange. (Not getting too far ahead of ourselves, we should note that this is the fundamental structure of the dialogic principle in all living cultural meaning systems.) Our capacity for a sense of self among other selves, for being a subject of, and subject for, others is a direct function of our being in language, of our capacity to represent our thought and consciousness in the same symbolic forms as others. Our symbolic faculty and all of human symbolic systems and subsystems, from the simplest utterances to the sign functions in complex computer algorithms and scientific formulas, form a web and continuum of symbolic cognition and meaning making that defines being human and living in cultures and social groups of any kind.

The intersecting areas of research in the cognitive sciences, linguistics, semiotics, and communication are generally united by a consensus (described differently in the various schools of thought) that language and symbolic cognition are fundamentally inter-individual, intersubjective, and distributed through networks of social actions and organization. Any individual expression of meaning is always already embedded in a larger network system of relations which makes it possible.

2.  Key Concepts and Interdisciplinary Background

Over many years of research and philosophic inquiry (some spanning hundreds of years), many disciplines and sciences are now converging on ways to define the complexities of human symbolic activity. These fields provide us with conceptual terms, a knowledge base to learn from, research methods to compare, and research findings to evaluate.

All these forms of inquiry are motivated by finding ways to bring to conscious awareness the ordinarily inaccessible or unconscious processes and structures that we depend on for all daily meaning making. It is no surprise that research and theory domains with names like “symbol processing,” “parallel distributed processing,” “the semantic web,” “cognitive semantics and semiotics,” and “cognitive anthropology” are now the key words in hundreds of journal articles and books every year.

Open Questions for Research and Hypothesis Testing

• Is language the main, or sufficient, modelling system for all human symbolic activity? What model most usefully accounts for meaning making in non-linguistic sign systems (visual, auditory, spatial, mathematical, artefactual) and the complexity of combined and interconnected sign systems used in all daily activity?

• How do we model the meaning of minimal units of sign systems (words, sentences, images, sounds) with their dependent contexts, situations, social frames, genres, and the implicit prior knowledge in which meaning is created and understood?

• How do we model the parallel and interdependent meaning structures in complex artefacts like discursive arguments, a book genre, movies, TV shows, rich media websites, and buildings (just to name few)?

• How do we model the generative processes of meaning making in the ongoing understanding of expressions and the ability to create new expressions as outcomes of what we understand?

We can aggregate some some key findings and widely accepted concepts from several related fields in linguistics, semiotics, and cognitive science that enable us to start from a knowledge base for de-blackboxing human meaning-making, sign systems, and our symbolic faculties:

(1) The encodings structured in any symbolic system are not natural extensions of the physical properties of sign vehicles: for example, the sounds or written characters for the English word [baseball], have no natural connection to actual baseballs or “baseballness” as a concept.  Language and our other sign systems are grounded in learned, conventional meaning associations--the function of being a symbol. The meaning is not a property of a thing, but is in the meaning system of which it is a part. (This principle is one of the major foundations of modern linguistics and semiotics, from de Saussure to recent post-Chomsky cognitive linguistics.)

(2) All sign systems share the characteristics of being (1) acquired and learned in specific communities (that is, meanings are not naturally given in things or in the biological brain), (2) rule-governed (not random), and (3) collective and intersubjective (not private or individual).  

(3) The “standing for” function in all symbolic forms, the ability to associate something absent (meanings registered only in thought and collective thought) with something present (perceptible sign vehicles like spoken sounds, written characters, or images) is called a sign relation or sign function.

(4) Human thought, culture, and every kind of social interaction is grounded on meaning making in shared signs--language, images, abstract symbols in mathematics, and all the complex and mixed forms of culture (media, ideologies, artefacts, the human built environment, technologies).

(5) Meaning making is also always a time-sequenced, embedded activity or process. Through sign functions, we connect or mediate present thought to the past (memory) and to the future (anticipating, inferring).

(6) It’s more useful to talk about sign processes or sign functions rather than signs or symbols as things. Meaning making in signs is an activity, a process. We instantiate meanings (generate specific expression from abstract rules, codes, and concepts) through mediations, representations, and cognitive frameworks--in short, using signs and symbols to generate further signs and symbols in an ongoing interpretive and meaning-generative process.

3. The Cognitive Science Model and Problem of Observing Symbolic Cognition

How the neural substrates for the human symbolic faculty (or faculties) in the neural nets of the brain produce what we experience as meaning in sign systems remains profoundly unobservable and inaccessible to introspective thought (see graphic below). We have technology to measure and visually map brain activity and for defining regions in the brain where kinds of perception, motor control, and cognition occur, but we are still far from understanding the complexity of these neurological processes in our symbolic abilities, and how human brains/minds necessarily work collectively and intersubjectively in societies and cultures. But symbolic cognition is not a complete, unknowable black box. We have many years of research on the necessary structures of language, symbolic representations, human conceptual processes to draw from, and we have the rich evidence “hiding in plain sight” all around us for the effects of symbolic cognition and sign processes in all the forms of expression with technical mediation that now make up daily human life and culture.

However close we can get to describing the neuro-bio-cognitive bases of the symbolic faculties, we will also need to recognize that “meanings aren’t in our heads.” All the evidence is right around us all day long in countless simultaneous conversations, in all the symbolic artefacts of culture, in the media for inscription, encoding, and transmitting messages, in our built environment, and in the cognitive technologies of computation and digital media. Human meaning making in symbolic systems is fundamentally intersubjective and distributed beyond the biological limits of any brain. The profound mystery is how our individual brains simultaneously implement the same cognitive structures to enable language-based societies, collective meaning and cultural memory, and all the generative, combinatorial capabilities for discrete infinity in all our symbolic systems from speech to complex artefacts and technologies. Using a software analogy, we can say that meanings and symbolic cognition are distributed globally and installed locally.

The diagram below maps out a top level view of research problems and domains across the disciplines, and represents areas in a continuum of functions that are open to many approaches, theoretical models, and research programs. We will begin investigations into the areas bracketed underneath as the “semiosphere,” the domain where meaning making and symbolic processes are observable and commonly known in cultures.

Let’s consider some ordinary, daily, meaningful activities that are possible from all the interacting parallel systems in our symbolic activities. We have the capacity to do very fast, parallel, simultaneous processing of many kinds of represented information (in language, sounds, images, 3D space), a capacity that also makes the process seem transparent, automatic, and “natural” to anyone in any language community, society, or culture.

It’s difficult to take a few steps back to try to catch what happens in our most common everyday interactions. What makes the experience of any conversation, movie, TV show, software app, or website seem immediately intelligible and also open to unanticipated, new meanings? Many culturally encoded symbolic systems are happening simultaneously: we combine spoken language, written text, graphics, images, long narratives, conceptual framing and visual layout into coherent patterns of meaning. And none of the meanings we make are material properties of the perceptible sign vehicles we observe: we make the meaning associations on the fly from the internalized codes and conventions of our language and culture, and we readily generate new signs (verbal expressions, images, diagrams, mathematical equations, etc.) to express understood meanings to others and they to us. So, our symbolic faculties and specific sign functions like language don’t simply connect our individual thoughts and perceptions: sign functions mediate our cognitive links to others and enable any social member to conceive an other’s meaning and point of view. Tracing some common meaning making activities back to necessary symbolic faculties and sign systems quickly yields up a hidden dimension with major consequences: meaning making and all symbolic activity is grounded in, and is the ground of, intersubjective, collective, and shared cognition. Culture is thus the summation of what and how we  transmit symbolically encoded meanings.

An analogy from computer science also yields an insight about our ability to implement systematic links between material signals and abstract meaning. In modern computation, binary mathematics is the “bridge” or interface between math and logic (formal, conceptual symbolic systems) and electronics (physical states of energy in circuits). In modern digital computation, the on/off states in electric current are designed to represent yes/no, true/false sequences in logic (as 1 or 0, numerical symbols for the “bit” or binary unit). Sequences of bits (the numerical values mapped to electronic states) can be defined to encode any kind of symbol (letters, numbers, units of digitized sounds or images). Human symbolic faculties are also the “bridge” between mental states emerging from neural activity (individual and collective) that link us to the world and to others. Sign systems enable us to think abstractly (applying general concepts to multiple instances beyond one experience in time and place), express thoughts to others and they to us (in language and every form of communication media), invent and develop technologies (which require representable, communicable, conceptual models translatable to material form), and create all the complex, meaningful artefacts and media of culture (which require physical-material substrates structured with sign/symbol components).

The irreducible ground of all human thought and communication is always already there in the form of our thoughts and interactions themselves, but the ground is usually unobservable because we are always using symbolic faculties in almost all other cognition. And, of course, computation is an artefact of human symbolic cognition, and as such allows us to model many aspects of thought and perception that are not observable.

4.  Studying Sign Systems and Symbolic Cognition: Semiotics

Those who haven’t read around in the fields of linguistics, semiotics, media theory, and philosophy of language find semiotic terminology strange and difficult to dissociate from the common, everyday uses of the words. So, like in all hard conceptual and scientific work, we have to unlearn some things and define terms in more precise ways to learn new things. In talking about meaning structures as signs, or, rather, sign functions, symbolic functions or symbolic activity, we’re not talking about, or modeling problems on, things like street signs, logos, advertising, or everyday things that we often call “signs.” (Of course, street signs, logos, and computer icons are instances [tokens] of a type of sign function that we will study, but these static types are not useful as models for symbolic activity in general.)

The core question for semiotics is accounting for meaning generation in any symbolic system (from language to complex multimedia): how do we make meanings with perceptible things (sounds, visual shapes) such that what we understand is not caused by a physical property of things but supplied by intersubjective cognitive structures independent of any specific perceived expression? Further, in language and all forms of cultural expression, we’re born into systems of meaning already in place and re-learnable, systems in which we become cultural members with the ability to understand and create an infinite set of new expressions never personally experienced before.

We have a 2500-year-old tradition of thinking about how words and images mean something beyond material, perceptible marks of expression. How to best refine and define a vocabulary of description for all our observable and unobservable meaning process is an ongoing question. In ancient Greek philosophy, the ordinary Greek word adopted for philosophical description was semeion (mark, sign, symptom), from which we get the base term for modern approaches called semiotics and semiology (the inquiry into signs, sign processes, and the meaning-making function). C. S. Peirce termed his inquiry semeiotic (borrowing from the philosopher John Locke), and Ferdinand de Saussure used the term semiologie (French; English: semiology). The English word sign comes from the Latin word signum, which had a similar semantic field to that of the Greek word semeion, but was based on the root metaphor of a mark or seal. The English word symbol (also used in other modern languages) comes from the Greek words symbole and symbolon (by way of transliteration into Latin as symbolum). The Greek word was formed from the prefix syn/sym (with, together) and bole/bolon (something thrown), hence “something thrown together, something joined together.” It was the common Greek word for token, tally, mark, and anything used to indicate a socially agreed exchange (for example, a document used as a legal agreement, a ticket, etc.).^[3]  Symbol is a conceptual metaphor for the “bringing together” of a perceptible form and a mental meaning; hence, symbol in all of our the ordinary uses of the word. The extended metaphoric sense of the Greek verb symballein (literally, “throw together”) was already in use in the late classical world for “putting thoughts or words together,” conversation, and interpretation. The question for us now is how to work with these terms, and the related concepts that they try to capture, by establishing more precise definitions, and then expand and apply the terms for making useful descriptions and interpretations.

The Major Inherited Models of Sign Functions: Heuristics and Limitations

The main sources of sign-function models and theoretical terms in current use come from four main disciplinary sources:

How Semiotics Can Help Reveal the Core Human Operating System

Semiotics also helps us understand that any human-created symbolic system for representing abstractions with logical rules for combination and necessary meanings -- for example, all forms of mathematics, algebra, and algorithms -- are extensions of core symbolic cognitive capabilities. The methods for modeling complex bundles of high-level abstractions that can be implemented in technical and material forms are mirrored all around us in our technologies and in our built environment. For example, we find the model of a hierarchical, layered modular system in the computer network “protocol stack” of tiered interoperating functions that we use every day (the structure of the Internet and locally routed Ethernet connections), and in the nested hierarchically defined abstractions and embedded functions required in computer programming. Every technical system begins with symbolically representable abstractions (equations, diagrams, flow charts) capable of multiple instantiations or realizations in actual built stuff. Everything from natural language to the latest complex combinatorial technologies are based on generative principles that are both specific to a specific symbolic system and also shared among other systems that we use concurrently or in combination.

5. Ferdinand de Saussure and Structuralist Linguistics and Semiology

Students in the humanities and social sciences are often introduced to semiotic concepts from the writings of Ferdinand de Saussure, which provide a starting point but a very limiting tradition for conceptualizing how symbolic structure and meaning-making works across all sign and symbol systems. During his lifetime, de Saussure was known mainly as a specialist of Indo-European comparative and historical linguistics (a field that emerged from philology, the study of ancient languages and literature). His name is most famously known from the Course in General Linguistics (1916), which was published after his death from notes and reconstructions of his ideas by students and colleagues. He never completed a systematic treatise, and never thought he had an adequately complete model of language and the larger human sign system. De Saussure’s approach to language was structuralist, that is, viewing language as an internally constituted system of relations that form underlying structures always already in place when a language is learned in a language community. He correctly observed that any language is formally complete at any historical moment (the synchronic perspective, one slice of time with all internal states in sync): how a language works for a language community can’t be explained by references to the historical development of the language (the diachronic, through-time perspective). It was this systems view--the principle that explanations must come from understanding the interdependent relations within a language as an internally complete system at any point in time--that became a seminal, heuristic conceptual model for other fields considered in the “structuralist” and “poststructuralist” program in the second half of the 20th century, including anthropology (Levi-Strauss), the French approach to semiology (Barthes, Greimas), exposing ideological contradictions (Derrida’s deconstruction), and film and media studies (Metz and many others).

De Saussure’s primary insights were (1) describing language as a structured system based on arbitrary (that is, not naturally given) mappings between the speech sounds of a language and the mental/conceptual meanings of spoken words and sentences, and (2) differentiating actual, concrete instances of speech (utterances as we speak them; French term: parole, speech, spoken discourse) from the underlying rules or grammar understood by a language community (langue, “language” as a formal, necessary rule system) that determine how expressions can be formed.  For de Saussure, language forms a system of relational differences; that is, differentiations among the elements in the various layers of language with no independent, positive states outside the system. The function of a language component arises in its differential positions in an internally self-sufficient system. For example, in English phonology, we differentiate /bat/ from /pat/ in the minimal structured sound differences of the initial consonant sounds in the words, a sound-value differentiation which, of course, only has a value in English.

[I]n language there are only differences. Even more important: a difference generally implies positive terms between which the  difference is set up; but in language there are only differences without positive terms. Whether we take the signified or the signifier, language has neither ideas nor sounds that existed before the linguistic system, but only conceptual and phonic, differences that  have issued from the system.  (De Saussure, Course, II.4, “The Sign Considered in its Totality”)

De Saussure derived this general differential principle from phonology and phonemics: each language has a system of sounds, or sound-phoneme mappings, that form an internal differential system specific to that language. The speech sounds in English, for example, within a range of dialectal variation, are internally self-consistent within English: we readily differentiate the sounds of /map/ from /nap/ from differences between phoneme units that map onto different words. But the sound units /m/ and /n/ mean nothing as independent spoken sounds. They have a value only within a relational, differential system of sounds used in an actual language.

De Saussure also hypothesized that the differential and binary structure of language was systematic through all levels, including semantics and signification (an assumption developed into the philosophical project of deconstruction in Derrida’s Of Grammatology and other works). Most linguists view this assumption as wrong for semantics and all the levels of meaning generation possible in a language, and have developed much more detailed models of parallel architectures, conceptual frames, and semantic networks to explain meaning systems. The system of differentiations in language and other sign systems includes binary oppositions or differences; for example, black/white, masculine/feminine, natural/artificial, which are partially explicable as mutually entailed differences, but only as relations in a larger system of meanings. Our whole meaning system is a complex network with multiple kinds of relations and levels of abstract conceptualization involving an interplay of semantic levels (like dictionary meanings) and the cultural values and symbolic correspondences known in a culture (an encyclopedic level of meanings).

 De Saussure’s model of the linguistic sign was dyadic (two poles or layers), and was perplexed and confused by the linear communication and information model. The model has limited appeal today, but has provided the starting point of semiotic description in many fields.

The major limitations of de Saussure’s model are the very static, unitary, and context-free conception, which is incapable of explaining generativity and complex expression with combined sign systems (e.g., language in long discursive patterns, language combined with other sign systems like video/film, and all forms of multimedia).

Where de Saussure distinguishes between langue and parole (the underlying grammar and rules of a language vs. spoken and written expressions as used in specific instances of expression), Chomsky (in his early work) distinguished between “deep structures” and “surface structures,” and “competence” (internalized rules) vs. “performance” (speakers’ actual expressions with all their variations). These distinctions allow us go beyond the experiential data of language in instances of use to the underlying rules everyone shares in making new expressions and participating in a system of meanings.

The structuralist presupposition is a systems theory model, which is the important starting point, but by maintaining a model based on binaries, dyadic relations, and dichotomies whose relatedness was unexplained, the theory is inadequate for explaining what we need to explain. But like with all theories and methods, it’s important to understand what motivated the questions and how problems were framed for those trying to figure out systematic explanations. Saussure’s structuralism became the starting point for many earlier approaches in semiology (the usual French term) or semiotics (in the US and UK). In the first wave of structuralist work in the 1950s-1960s in anthropology, literature, and early cultural studies, we see an interest in outlining a formal method for analyzing any meaning system as a “second-order” language; that is, for semiological description to proceed and succeed, we must presuppose that many of the structural features of language also function in other “language-like” systems (for example, visual art, music, and film/video). A semiological description of a symbolic system presupposes that language-like structures (rule-governed procedures for combining units and mapping them onto socially shared meanings) are implemented in an internally complete system with different levels of operation analogous to linguistic levels and underlying principles for combining units into meaningful structures. The analogy with language as modeled by Saussure provided a basis for describing all the higher levels of meaning, like ideologies and cultural values, as differential positions within an internally closed system of binary relations. (This is not a full and sufficient model of the meaning process, but one dimension at the internal structural level of description.)

6.  Peirce and Semiosis: Meaning as a System of Symbolic Cognitive Activity

The important principles outlined above are usefully completed by merging them with C. S. Peirce’s model of the generative sign process and symbolic cognition that he termed semiosis. The structuralist approaches in the De Saussurean tradition provided conceptual advances, but they are limited by modeling meaning structures as atemporal, dyadic, and static abstractions. Peirce and recent advances now allow us to see meaning and symbolic activity as dynamic processes of intersubjective cognition that happened only in time and in the historical contexts of meaning generation in communities and societies.

C. S. Peirce (1839–1914) worked on the problem of sign processes, meaning, concept relations in formal logic, and human cognition over his whole career, and he continually revised his terminology and conceptual models for capturing what happens in understanding and expressing meaning (which has made his theory notoriously difficult). We won’t be concerned with following his terminology as doctrine, but as attempts to find more precise ways of describing the symbolic activity common to all forms of human thought and expression from language and creative works to mathematics and science. Most of the time, Peirce’s concepts are translatable into the more normative terms used in contemporary linguistics, cognitive science, computation and mathematics, and recent syntheses in semiotics, but he made some very important distinctions which we will preserve.

Peirce was motivated by abandoning both the Cartesian and empiricist starting points of accounting for meanings as private, mentally interior “contents” or states of mind derived from (caused by) sense perceptions. For Peirce, as for us, it’s difficult to maintain a consistent terminology. He uses the term “sign” often to mean the perceptible sign-vehicle like sound units or visual forms, as well as the name for a general category of cognitive activity. His formal definition of the term “symbol” as the general, abstractive, and rule-governed type of sign is the main concept we will focus on. His main discoveries, which at first sound cyclical or tautologous, can be summarized:

1  Human thought is based on signs in symbol systems which have a structure of material/perceptible and cognitive relations that unfold dynamically in human-experienced time as shared cognitive processes.

2  Meanings, learning, and knowledge are produced through symbolic- cognitive transformations (signs yielding interpretants expressible in further signs in unlimited and open-ended chains or networks).

3  The meanings of signs and symbols (any set of meaning tokens used and taken symbolically in a community) are not things but the processes of interpretation initiated and instantiated in the symbolic representations themselves as signs are used to create meanings.

4  The intersubjective social bonds of sign users, which are installed by symbolic cognition and language, precede individual understanding and individual instances of expressed meaning. Hence, all sign-instantiated meanings are dialogic.

Symbols Enable Meaning Inferences from Perceptibles to Cognizables

Peirce’s key insight--now corroborated in other sciences--is the discovery of the parallel processes in symbolic cognition for interpreting and generating intersubjective meaning made possible only by the mediating functions of signs. Signs enable us to associate or relate something perceptible to something conceptually additive, a cognizable that is more than, and other than, the perceptible component of a sign-vehicle. For example, meanings are not “in,” or properties of, the perceived sounds of language or music or the visual data of an image. For Peirce, meanings are chains of inferences from things physically given (the sign tokens we recognize) to cognitive responses, learned associations directing us from something physically present to something cognitively supplied (levels of meanings and other responses only symbol cognizers generate).

“[A] sign is something by knowing which we know something more…

[A]ll our thought and knowledge is by signs” (Peirce, CP 8.332).^[4]

Often-cited descriptions of Peirce’s model of the sign process:

That, since any thought, there must have been a thought, has its analogue in the fact that, since any past time, there must have been an infinite series of times. To say, therefore, that thought cannot happen in an instant, but requires a time, is but another way of saying that every thought must be interpreted in another, or that all thought is in signs. (CP5.254)

A sign, or representamen, is something which stands to somebody for something in some respect or capacity. It addresses somebody, that is, creates in the mind of that person an equivalent sign, or perhaps a more developed sign. That sign which it creates I call the interpretant of the first sign. The sign stands for something, its object. It stands for that object, not in all respects, but in reference to a sort of idea, which I have sometimes called the ground of the representamen. (Peirce, CP 2.228)

A sign stands for something to the idea which it produces, or modifies. Or, it is a vehicle conveying into the mind something from without. That for which it stands is called its object; that which it conveys, its meaning; and the idea to which it gives rise, its interpretant. The object of representation can be nothing but a representation of which the first representation is the interpretant. But an endless series of representations, each representing the one behind it, may be conceived to have an absolute object at its limit. The meaning of a representation can be nothing but a representation.... So there is an infinite regression here. Finally, the interpretant is nothing but another representation to which the torch of truth is handed along; and as representation, it has its interpretant again. Lo, another infinite series. (Peirce, CP 1.339)

C. S. Peirce’s Model of the Meaning Process:

One Moment in a Sign Relation Connecting Developments of Meaning

Peirce never provided a diagram of his concepts for sign relations, but they are often presented that way today. Capturing Peirce’s insights in a diagram or static model is difficult if not impossible, but we can try to make the components or coordinates his generative meaning-making model intuitively clear so that we can use the basic structure for analyzing meaning processes in specific symbolic systems (writing, images, graphics, music, code, etc.) and different forms of media.

Peirce consistently states (and shows) that what we perceive and understand as meaningful signs and symbols only happens in a triadic structure of correlations. The diagram above attempts to model a “meaning event” (center) when we use and understand signs. Physical-perceptible instances (Representamens) don’t and can’t have meanings or values “in” what is perceptible, and if signs and symbols stand for what they mean, what they are “about,” some third mediating agency, the Interpretant, must make the correlation.

Peirce’s writings show that he was working through several permutations of the terms “sign” and “representation.” Sometimes he uses the general term “sign” to mean the Representamen, that which does the representing function like perceptible marks on a page, sounds, artefacts (like thermometers), or image patterns.  Butu he also differentiates the perceptible units from the sign-process, the dynamic correlation. Note that in the statements above Peirce does not define the material-perceptible sign-vehicles (speech sounds, written words, images, artefacts, etc. as they appear to our physical sense perceptions) as signs in the cognitive process (these are more properly termed “signals,” which, as perceived, Peirce calls the representamen, the perceptible thing represented in thought). In his model, the sign isn’t a thing, but a relational process that elicits a meaning-bearing unit in a cognitive event: a sign is the conceptual-relational process or activity, which is cognitive and interpersonal. The symbolic function evoked by signs does not pre-exist their activation in a “standing-for” relation for someone. Meaning is what a symbolic-cognitive agent does with units of a sign system. For Peirce, the sign function (the cognitive role) and the sign process (what happens when the sign relations are activated and we understand meaning) are at the core of human consciousness, reason, logic, knowledge, learning, creativity, and social existence.

Peirce’s term object in his definitions of sign functions is often misunderstood: “That sign which it creates I call the interpretant of the first sign. The sign stands for something, its object. It stands for that object, not in all respects, but in reference to a sort of idea, which I have sometimes called the ground of the representamen” (CP 2.228). He did not mean simply real-world referents (though he classified some kinds of sign uses for referring to real things). Since he began the semiotic project as a way to understand logic, he recognized that signs and symbols are used for all kinds of concepts and imaginary things. An object in his terminology is a position in a cognitive relation -- a concept, idea, code, or mental rule (like an inference or an interpretation) -- which may or may not be prompted by, depend upon, or point to a real world referent. “By an object, I mean anything that we can think, i.e. anything we can talk about.”^[5] In this sense, the complex sets of meanings in fictional narratives, musical compositions, media artefacts, and everyday speech relate to “objects” of symbolic thought interpretable in further sequences of signs. In his terms, even the index, the kind of sign used to indicate an immediate real world thing (like the gesture of pointing at something or a physical description), becomes an indexical sign only in a cognitive event (i.e., a sign cognizer who forms the relation: indexical sign => something indicated), not because the real world of things (the perceptible-empirical world) imposes its reality or meaning on us as if by unmediated irresistible force.

Peirce triadic model of the sign function provides the core of a very useful explanatory model: the “standing for” (relational) function in any symbolic representation requires a cognitive agent, and our cognitive leap in making meaning takes the form of what Peirce termed an interpretant (the “meaning” we understand), which is conceivable, representable, and expressible only in further “more developed” signs (like moving up levels of abstraction to more generalizable concepts). Interpretants, as symbolic instantiations of understood or interpreted meanings, embody and initiate the meaning-making process again through sequences of time. In this model, meaning grows and develops in an emergent process (like ideas that develop by seeing inferences, implications, and moving through orders of abstraction), not by rearranging anything fixed. Peirce’s model -- as frustratingly incomplete as it is -- is actually a first map of the dynamic, emergent, and generative processes underlying all sign and symbol processes.

Semiosis: The Generative Meaning-Making Process

From this conceptual “Copernican Revolution” in thinking about cognitive-symbolic relations rather than signs as things perceived (i.e., already-formed objects), Peirce focused on the meaning-making process which he termed semiosis, a concept for symbolic meaning-productivity as an active, ongoing process in all thought, communication, representation, and cultural history. Note that his term for the sign-vehicle (parallel to Saussure’s notion of the signifier) was initially representamen, what is presented to cognition with a representational and potential relational function. In other words, the sign process begins with something perceived as interpretable or meaningful, that is, in a context and in a recognizable symbolic form for  conveying meaning.  (Peirce used the term ground to describe the recognizable context of sign systems. The whole context of a language in symbolic cognition is the “ground,” in Peirce’s terms, for using language as meaning tokens; that is, the sign-specific “code base,” as we would say today, of speech sounds activating semiosis, generated meaning. )

If we were to think with Peirce to critique de Saussure, he would say that that speech sounds per se as possible acoustic information are not the signs or signifiers, but become signs when mapped to phonemes and morphemes on up to all the grammar and semantics in a person’s language. (Although he does not draw this connection specifically, Ray Jackendoff implies this description in his “parallel architecture” model of language.) That is, the perceptible signals become part of sign functions when the perceptible data is recognized as standing-for something beyond a sense perception, that is, something cognitive, interpretable, understandable. One obvious direct proof of semiosis, meaning-making or meaning-being-made, is one person answering another in a linked sequence of further signs in ordinary dialogue and conversation, i.e., instantiating interpretants by continuing linked expressions over the time frame of a conversation.

But then how are the two cognitive events -- perception of a sign vehicle and cognition of a mental content -- connected? How do we make meaning?  The answer lies in the third relation, the interpretant, which underlies what we most commonly call “meaning.” Peirce went beyond a binary model (like de Saussure’s) because he was trying to describe the sign-function as productive of new meanings and understanding, the model for a logical process unfolding in chains of inferences to fuller or more complete meanings in a knowledge process. The relation of something perceived as a sign-vehicle (representamen) to the cognition of a sign is mediated by a third relation, the interpretant, a necessary relation internal to the sign process, the output of which is another sign or set of signs. The interpretant is not an “interpreter” in the sense of an individual person, though, of course, interpretation is a cognitive process that someone -- or some agent like a software interpreter -- activates. The interpretant is the underlying meaning-making relation all sign users instantiate. The interpretant is the key to meaning-making in providing a meaning-relation between a sign-vehicle (token) and a cognitive response. Further, the interpretant in the sign process is expressible only in an additional sign or signs, thus enabling our ongoing, dialogic meaning processes.

In his model of semiosis, Peirce zeroed in on the ongoing, generative nature of meaning systems, not simply a static descriptive model of semantics or logic. His breakthrough was seeing the meaning process as a nexus of relations and mediations that enable thought to be productive of new meanings. A sign isn’t a static object, and meanings aren’t isolated events: meanings from symbolic activity are part of a continuum of thought that link an individual’s cognition to shared concepts, to the experienced world, and to others. Meaning is an open triadic process with one element of the structure, the interpretant, always unfolding new meaning. Peirce’s model is a view of a dynamic process, not a static moment, and his work continues to be highly relevant to day. In this model, each moment of semiosis is a linked process, a mediation and transition, to prior and future moments with no necessary end point. (Meaning and understanding are purposive or goal-directed, but Peirce and others view meaning and knowledge as processes of unlimited development.) Beginning in the center of the diagram, labelled T0 (for the positing “Time of the present moment” or beginning ground time-state), we find that a sign understood (for example, a statement in a dialogue) has followed from a prior sign completion (T-1) and is capable of generating meaning states T+1 through T+n (cognitive meaning states that end with practical limits or when we reach a satisfying understanding). Everyday examples of this process are ordinary conversations, understanding sequences of expression that presuppose prior expression (e.g., TV shows), or interpretation of complex wholes like magazines and websites. In other words, the “outputs” of meaning-making activity through signs received as “inputs” are additional strings of signs that represent the meaning we have “activated” in the sign process. We frequently register our understanding of a prior set of signs by making new statements that include and presuppose the earlier ones, keeping conversations going, or writing about something we’ve read or seen.

This fundamental linked process of meaning-making led to the discovery of goal-directed “unlimited semiosis” -- the underlying generative function enabling chains and networks of expression and interpretation with unlimited productivity. This principle can be expressed in a general postulate:

The interpretation [meaning] of a set of signs

will usually take the form of additional sets of signs.

We say “usually” here, because Peirce recognized (along with most cognitive psychology today) that many interpretive responses take the form of emotions or actions (as in commands, requests, and artistic expression). In the largest, most general sense, the “meaning” of any symbolic form is the response we get. That we do respond as a result of perceiving and understanding a socially symbolic form (like an artwork or piece of music) is proof of the Interpretant correlation: an expression in perceptible form elicits a response it was generally intended to evoke. So in all symbolic systems, we find an interplay of conceptual, general, emotional, and action-producing responses that form a complex of meanings.

Peirce provides heuristic models that continue to pay off with dividends for our current multimediated era in which symbolic activity expands across a digital infosphere. Symbolic systems don’t just exist like a static inventory; they grow and develop over time and place: they come from somewhere, become transformed in real-time use in specific cultural contexts, and are going somewhere else to be received in other situations. We navigate a world of meaning every day, and the meanings we express about something always take the form of additional sequences of signs--further sets of words, visual information, code, algorithms, images, artworks, and interpretable data.

Interpretants, “Discrete Infinity”, and Pragmatic Limits

Peirce thought that the chains and networks of interpretants that form the expressions, debates, and cultural representations of a community (our actual instantiations of semiosis in material expressions) are driven by a need for meaning completion, the “more than” built-in to the sign process which always generates further meaning complexes. In his later description of the symbolic process, Peirce distinguished three phases of Interpretants: Immediate (our first impression or response that something perceived as symbolic has meaning), Dynamic (the meanings that emerge and develop with new associations in uses and contexts, like being in a dialogue or learning new concepts), and Final (the motivation for completed meaning experienced as future-directed unlimited possibility).  Meaning chains are thus open-ended or unlimited (like Chomsky's “discrete infinity”), but likewise bounded by practical limits in situated uses of communication or creative expression. As a pragmatic philosophy of meaning, Peirce understood that social groups of all kinds usually come to some kind of temporary closure, conclusion, or consensus about meanings (as in normative scientific knowledge, laws, any kind of group understanding), and that, on the individual level, all our thought is bounded by human limits of cognitive time and attention. But anything we express or represent symbolically is always open to further interpretation, dialogue, argument, and expansion in a social context of collective and intersubjective meanings.

7. Foundations of Cultural and Multimodal Semiotics:

Sign Systems and Parallel Architectures

Important questions motivating research:

Can we extrapolate from models of language to language-like models for other symbolic systems, and, if so, how? In other words, do the features of language, in the best models we currently have for them, “map onto” or correlate with the features of other symbolic systems (images, music, complex artefacts, software, digital media)?

Given that language is the primary human meaning-making system functioning with multiple, linked processes operating in parallel simultaneously, is there a Parallel Architecture within and across other (or all) symbolic systems?

7.1  The Project of  Cultural Semiotics

The main question of the semiotics of culture is the problem of meaning generation. What we shall call meaning generation is the ability both of culture as a whole and of its parts to put out, in the ‘output’, nontrivial new texts. New texts are the texts that emerge as results of irreversible processes..., i.e. texts that are unpredictable to a certain degree.^[6] (Yuri Lotman)

The starting point of a “cultural semiotics” is the insight turned explanatory-hypothesis that all of our symbolic systems and their implementations in media are structured like language; or, stated another way, language is the primary modeling system on which, or from which, our other sign systems are understood. The diverse field of semiotics from the 1960s to the present (including Anglo-American, French, and Eastern European-Russian schools of thought) has largely been devoted to building interpretive models for understanding “language-like” structures in all symbolic forms: narrative (in texts, images, and cinema), artworks and photography, music, film/video, multimedia, software, and human computer interfaces (HCI). The angels and devils are always in the details, and the goal of describing extensible models of the complexity in all meaning systems is still far from being achieved. This chapter introduces an approach that combines work in cultural semiotics and generative-cognitive linguistics that may get us closer to a research method for the central questions of human meaning making and the symbolic function in different forms of media and material implementation.

Important steps toward an inclusive cultural semiotics model were developed by Yuri Lotman (University of Tartu) in the 1960s-80s by expanding on the work of Russian and French theorists (especially Bakhtin and de Saussure).  Umberto Eco, also a major contributor to the field, succinctly sums up Lotman’s primary contribution to modeling semiotic systems:

Semiotic systems are models which explain the world in which we live (obviously, in explaining the world, they also construct it, and in this sense, … Lotman saw semiotics as a cognitive science). Among all these systems, language is the primary modelling system and we apprehend the world by means of the model which language offers. Myth, cultural rules, religion, the language of art and of science are secondary modelling systems. We must therefore also study these semiotic systems which, since they lead us to understand the world in a certain way, allow us to speak about it.^[7]

This general view is widely accepted across many disciplines, but precisely how “secondary modeling systems” are structured in relation to “the primary modeling system” (language) continues to be the open question for detailed research and complex modeling. With the recent inclusion of cognitive science approaches in the interdisciplinary fields of semiotics, the question language, meaning systems, and the a general human symbolic faculty is motivating important revisions in the way language and other symbolic systems are connected. The main difficulty in the problem of language-as-modeling-system lies in describing what being language-like means: the generalizable semiotic description of meaning systems presupposes an adequate description of language as the reference point. If we accept the working hypothesis that language is the primary modeling system, then the most important step is to describe a model of language which permits the kind of scalable and complex modeling we need for our symbolic systems. What model of language is both complex and generalizable enough to explain how complex artefacts and multimedia work as meaning systems, systems which may also include language within hybrid structure (e.g., actors’ speech in cinema, text in graphics and image systems)?

7.2 The Semiosphere: The Whole Inter-Symbolic Realm of Human Culture

Yuri Lotman, the Russian pioneer of social semiotics, termed the whole interconnected human cultural space of meaning-making the semiosphere, the zone of human social life in which meaning happens in an interrelated system. The concept of an interconnected sphere of all symbolic activity may seem too large, vague, and abstract, but it provides a valuable systems model for understanding forms of symbolic activity in social-historical contexts. A network and systems model allows us to investigate expression and meaning-making as ongoing processes in multiple material forms rather than as isolated or compartmentalized functions (language, discourse genres, images, sound and music, multimedia, the pan-digital media platforms, and the analog-digital continuum).

7.3  Why a Model of Language Matters

Much work in semiotics is often accused of being (merely) speculative, overly generalized or simplistic, and not developed as hypotheses that can be tested or enacted in real-world cases and with specific forms of media. For example, if we start with a reductive signifier/signified sign model, all we have is a truism with a huge gap in it. The binary sign model is not conceptually equipped to answer these questions: (1) how are signifiers are related to meanings, (2) how does the binary structure explain the meanings expressed and understood in all our complex artefacts and symbolic systems, (3) how are meanings in the signs and symbolic systems that we use always interpersonal, intersubjective, collective meaning, (3) how are signs combinable and combined in sequences and networks of meanings, (4) how are signs generated with unlimited productivity through finite means? Surely we need a semiotic model (a model of  “secondary modeling systems” involving images and sounds and material artefacts) equipped to deal with the generativity of meaning in combinatorial structures specific to a symbolic system (like images) equal to the specificity of the models developed for language. Again, beginning with an adequately explanatory model for language must be the starting point for inquiry into language-like symbolic systems, the “secondary modeling systems.” Theory, with its cluster of conceptual metaphors, descriptions, and hypotheses, is like software: what you load to think with structures what you can do and what outputs you get.

The Combinatorial System of Language -- Syntax, Lexicon, Semantics

Since we need to think carefully about the kind of model we want, and what a model needs (at least) to account for, let’s do a recap of the major points in the linguistics- semiotics convergence so far. We’ll begin with the working hypothesis that language is a combinatorial system of integrated structures in parallel architectures involving a rule-based configuration of “components” (which may not be a helpful conceptual metaphor, as we will see).

It would be very unsatisfying to explain language and symbolic systems as a simple summation of these “components” in some kind of linear, sequential form. Research in linguistics, cognitive sciences, and semiotics has reached a general consensus that we form (compose) and understand (“get”) meanings in simultaneous and parallel processes in present awareness and memory. The major challenge is describing the activity of these structured processes, as we experience them in real-time, with sufficient detail and complexity.

7.4  Ray Jackendoff’s Parallel Architecture of Language Model

Ray Jackendoff's model provides a bridge between linguistics and semiotics as disciplines with models and methods (diagrams from Jackendoff 2003 and 2007 with my annotations):

7.5  From Sign Processes to Networks of Meaning:

Intertextuality, Dialogism, Intermediality, Cultural Encyclopedia

Intertextuality/intermediality, dialogism, and semiosis form a cluster of concepts used to describe conditions of meaning structures already in place before we make sense of anything or make a new expression in any form or medium. Human meaning-making is distributed across many symbolic systems (e.g., language, images, sounds, artifacts, designed space), and we can merge the most useful work in several fields for an integrated view. Peirce’s investigations are usefully combined with those Mikhail Bakhtin, the Russian scholar who analyzed language, social consciousness, and symbolic activity as forms of dialogue, a structure that grounds meaning making all cultural contexts.

"Dialogism" and "intertextuality" and (now) “intermediality” are powerful heuristic concepts developed in semiotics and cultural theory: these ideas keep on giving, allowing new discoveries in domains never anticipated by the originators of the concepts (Mikhail Bakhtin, Yuri Lotman, Julia Kristeva, and elaborated by many others). (See first the outline of Bakhtin’s main statements about dialogism in the separate document.)

Bakhtin’s first major insight: every expression in a culture emerges in a living, intersubjective dialogue of speakers with identities in social and historical contexts. The “origin” of an expression--a “word” in speech or writing--is not in the self, but in the dialogic environment with others that is the condition of possibility for an utterance from a self to occur. In contemporary linguistics, the study of this living, intersubjective situatedness of speech is called “pragmatics,” and many sociolinguists now also begin from Bakhtin’s starting assumption, rather than Chomsky's “i-language” (the internalized grammatical rules accounting for individual grammatical competence).  In Bakhtin’s important insight, every expression from the “first-person” position--when we state or assume “I” in an expression--is necessarily infused with an “other’s” (or multiple “others’”) speech: all expression is simultaneously a response, an answer, a follow-up that is also directed toward others and further responses or answers. Expression forms a node in a collective process with a past, present, and future: expression came from somewhere and is headed somewhere else. The “dialogic condition,” not an individual statement, is thus the foundation of living discourse and culture.

All of our individual expressions necessarily assume and embed the expressions of others, either in an immediate context (like conversation) or in cultural genres that unfold in a larger environment, both contemporary (during our own time) and an inherited cultural memory (a cultural encyclopedia). The “meaning of an expression” is thus always a function of its social use arising when speakers, writers, or creators assume and reference prior expressions and prior contexts in the genres of their culture. If cultures are formed by their texts (organized systems of expression in artefactual forms like writings, artworks, musical compositions, architecture, media), then cultural expression is necessarily intertextual, based on networks of dialogic responses and anticipations through both real-time, lived meaning-making and across long time spans of cultural memory in symbolic continuums maintained by reinterpretation, negations, parodies, rewritings, re-expressions, appropriations, remixes,  and new combinations of received sign systems and media.

Furthermore, living cultures and language groups have many “dialects,” a mix "official" culture and cultural "others." Ideology works to create the illusion of "monological" culture, a unified totality without differences. But the illusion can only be maintained by the suppression, repression, or control of "other" ways of speaking.

[Dialogism] is a mixture of two social languages within the limits of a single utterance, an encounter, within the arena of an utterance, between two different linguistic consciousnesses, separated from one another by an epoch, by social differentiation, or by some other factor.  

(M.M. Bakhtin, The Dialogic Imagination. University of Texas Press, 1981: 358)

Intertextuality and dialogism disclose how all forms of expression presuppose a shared context of knowledge, prior forms of expression, and possible new moves in the ongoing dialogue. The theory of intertextuality now morphs into intermediality (interdependence, cross- mediation, presupposition of prior and contemporary media already pre-existing for the new to appear).

We are now living with “transmediation” and the promiscuity of the digital media object: we experience "the same" media fragment or “content” across multiple platforms, material forms, and devices: movie or video in a theater, on TV, DVD, web-video streaming, iPod or hand-held player. Images, brands, logos appear across all media: billboards (solid or LED), magazine ads, TV commercials, shopping mall displays, Web ads, Youtube commercials.

Today, cultural participation depends on accessing a shared, learned "library" or "encyclopedia" of cultural knowledge acquired from multiple media sources. We can extrapolate from Michel Foucault’s and Jacques Derrida’s discussion of the function of commentary and the supplement, and Yuri Lotman’s view of cultural generativity, the always ongoing (re)interpretations, new appropriations, new expressions that presuppose and incorporate the old.

Bakhtin’s Model of Dialogism

The model below maps out the dialogic, intermedial meaning system for written (text-based) artefacts with the larger intermedial system of culture. (Merging concepts from Bakhtin, Kristeva, Eco, and Foucault for modeling semiosis on a the text-based processes.) This is a macro-level model which requires much more detailed descriptions of the processes in each component.

The problem with most diagrams of meaning-making processes is that they are static and one-dimensional. What’s missing is time and the full complexity of lived cultural experience.

Our experience of meaning-making and cultural artefacts of all kinds is always intermedial, not simply intertextual. We process symbolic information across all media types: a movie or TV show uses music, video/film images, language (speech and text); we can “translate” a novel into a movie and a movie into a novel. Most media is always already multimedia, embedded and presupposing other media. All expression relies on a collective cultural encyclopedia of genres, types, categories, styles, cross-references, and intermedial symbols.

Intermediality: A Model for Multimodal Artefacts

7.6  The Cultural Encyclopedia:

Umberto Eco’s Model of the Cultural Encyclopedia as Generative in Meaning-Making

Umberto Eco elaborated the concepts of “cultural encyclopedia” and “dictionary” (expanded from the distinction between lexical and encyclopedic levels of meaning in linguistics) to account for how members of an interpretive community decode meanings in language, texts, images, or any sign system by rule-governed procedures for handling multiple levels of meaning.  Merging Eco and Jackendoff, we can say that we develop meanings in parallel processes that depend on access to a structured knowledge base organized in hierarchical conceptual levels and with symbolic cross-references like an encyclopedia. “Dictionary” levels of meaning, of course, are well-known in linguistics and other fields: definitions of lexemes (word units) are normally restricted by metalinguistic procedures for “context-free” meaning. (Look up a word in a dictionary. What do you get?) A dictionary, a lexicon list, cannot provide a map of all the potential meanings and uses of a work, especially metaphorical and context-specific meanings, because these are formed in the living discourses of cultures, which are encyclopedic. The conceptual model of an encyclopedia entails describing multiple overlapping and intersecting contexts (historical, social, political, ideological) in conceptual structures that form networks of meaning beyond a dictionary list.

Merging the concepts of lexicon or dictionary (discrete, “base-line” units of meaning) and encyclopedia (networks of meaning relationships, unifying abstract categories like genre and type) we can describe how members of a culture participate in meaning-making by greater or lesser access to, and competence with, a preexisting, constantly accruing, constantly reconfiguring body of words, symbols, concepts, discourses, artefacts, and events that circulate within a community.

As Eco describes the way we develop interpretations in reading texts, there is always variation in the reading or interpreting experience because of the way different interpreters/readers enact or instantiate a cultural object in their various interpretive communities:

When a text is produced not for a single addressee but for a community of readers – the author knows that he or she will be interpreted not according to his or her intentions but according to a complex strategy of interactions which also involve the readers, along with their competence in language as a social treasury.... I mean by social treasury not only a given language as a set of grammatical rules, but also the whole encyclopedia that the performances of that language have implemented, namely the cultural conventions that that language has produced and the very history of the previous interpretations of many texts, comprehending the text that the reader is in the course of reading. (Eco, 1992, pp. 67-68).

The idea of the “cultural encyclopedia” here is a regulative hypothesis or heuristic model for accounting for how anyone in a community gets the meanings that they do.  “Such a global representation [of an encyclopedia] is only a semiotic postulate, a regulative idea, and takes the format of a multidimensional network” (Eco, 1984, 68). For a full model of semiotics that follows Peirce’s insights, the encyclopedia is a useful description for how we relate interpretants (at many levels) to symbolic information to generate meanings. With concepts for levels of abstraction that describe how we form the meanings that we do, we have a way to explain the observable “outputs” of generative meaning that all participants in a meaning-event like a work of music, art or film seem to produce automatically and transparently. We begin understanding any cultural expression by selecting from our “local” dictionaries (the elementary “vocabularies” of sign units) and pass this perception on for accessing codes and relations in the cultural encyclopedia for which the perceptible sign and symbol elements are interfaces.

The higher order symbolic system of the encyclopedia also helps explain metaphor, which requires symbolic relations and knowledge beyond the dictionary level:

The success of a metaphor is a function of the sociocultural format of the interpreting subjects’ encyclopedia. In this perspective, metaphors are produced solely on the basis of a rich cultural framework, on the basis, that is, of a universe of content that is already organized into networks of interpretants, which decide (semiotically) upon the identities and differences of properties. (Eco, 1983, 254-55)

Our learned codes for associating signs and symbols with their cultural meanings are a function of this macro-cultural Encyclopedia. For language and texts, the various vocabularies, discourses, dialects, and the whole lexicon of a language (like English) form a cultural dictionary, which preexists any individual user, who has varying levels of access to, and familiarity with, the whole Dictionary. Language, discourse, narratives, and visual images are the memory machines of culture. Information, communication, and media technologies are externalized memory machines for transmitting culture (Debray, Lotman), and thus function as physical or material disseminators of the larger cultural encyclopedia. Selectivity and privileging of certain contents is a function of ideology and processes of hegemony (ways in which everyone buys into a dominant view, those in power co-opting those with lesser power). Ideology privileges certain contents of the shared Encyclopedia, and selectivity or hierarchizing of cultural knowledge discloses the social function of the Encyclopedia as culturally constructed, not given or natural.

For Eco, then, interpreting or understanding the meaning  of something is not a matter “correct” or “incorrect” interpretations, but rather an instance of an interpreter's competence in engaging the available cultural encyclopedia, the whole repertoire of symbolic resources available and known to a culture. In this view, a reading or interpretation is an enactment or performance of possibilities generated by interpreting something within a larger social framework supplied by the community’s encyclopedia. This model also helps account for the cultural imperative of creating new meanings through additional cultural works. The interpretation of a text/set of signs will always take the form of another text/set of signs.

The important point about these ideas is their emphasis on ongoing semiotic process (semiosis) and historical continuity at the social and cultural level, independent of any individual member of an interpretive community, who is born into a culture that is always already happening and always reconfiguring codes and symbolic associations in which individual reader/interpreters can intervene and develop new interpretations.

7.7 Toward a Generative Syntax and Semantics for Visual Culture Signs

How We “Call” the Encyclopedia in Meaning-Making: Segmentation and Combinatoriality

The generative and interpretive procedures for complex signs are only possible by rule-governed syntactic forms, initially abstract or empty schema in which infinite pieces of encyclopedic information can be inserted. Like ordinary conversation and all forms of discourse, a living, culturally grounded dialogic situation constrains the possible uses of meaning into an actual expression, governing what encyclopedic information will become instantiated or represented.

In Peirce’s model of semiosis (the generative, meaning-making process of signs) expanded by Eco, to interpret a sign means to define a position in a continuum of meaning possibilities, a position which serves as a node in relation to other nodes in a continuum defined from its total combination of meaningful elements or decomposable content (see Eco, 1984, 44). In other words, receiving and understanding a complex symbolic artefact means that an interpreter in a cultural community forms a node in a network of possibilities entailed by the kinds of symbolic functions used (language, images, music, material objects, etc.) and affiliations with known expressions and codes accessible in collective encyclopedia:

“The universe of semiosis, that is, the universe of human culture … is structured according to a network of interpretants...” (Eco, 1984, 83)

This approach provides a way to explain meaning-making processes as the way we use symbolic resources (all of our sign systems) in an ongoing continuum. Peirce, Bakhtin, and Eco converge on the question of modelling meaning in a social-cultural dialogic process with multiple levels of symbolic abstraction. In short, through our use of a cultural encyclopedia of symbolic relations in the dialogic structure. The encyclopedia, like all symbolic functions, is collective and intersubjective. Individuals will always have different levels of competence and access to the resources of the cultural encyclopedia, but various subsets or nodal connections are always the ground of any cultural expression or understanding.

“Segmentation” of content means recognizing the meaningful elements composed in the combinatorial process for an expression in its own genre. (We are not always aware or conscious of all the signifying elements and contexts of an expression or a work because these are usually assumed and quickly grasped.) For example, we can break down a music composition into its formal structure and its many layers of tone, phrasing, rhythms, instrumental sounds, and words (if it has lyrics), each meaningful element having its own history and cultural associations with connections to a larger encyclopedia of cultural resources for music genres.

Music (like all symbolic systems) depends upon a combinatorial system made up of universal structures (notes, scales, rhythm) and structures specific to music genres and cultural categories of musical styles. We understand the meanings of a work as a sum of the combined elements, but a sum that is more than a mathematical summation of the individual components we may be able to segment out for analysis. Similarly, in complex artefacts like art works, a painting, sculpture, or photograph can be broken down into many compositional elements, forms, codes, and processes--including the symbolic value of the materials used in the works--but we understand the compositions as a whole, as a sum of the combined symbolic functions.

We do this in understanding all meaning systems by parallel processing combined symbols into a momentary intersection of meanings, a node in a meaning system instantiated by the artist or writer, and re-instantiated in some form when we interpret or understand the work. We can “segment”  from multimodal and multimedia complex forms (artefacts), which after merging simultaneous results are more than the sum of meaning processes, more than a simple computational summation.

Basic Bibliography & Further Reading

Semiotics

Bal, Mieke. On Meaning-Making: Essays in Semiotics. Sonoma, CA: Polebridge Press, 1994.

Eco, Umberto. A Theory of Semiotics. Bloomington, IN: Indiana University Press, 1976.

———. Semiotics and the Philosophy of Language. Bloomington, IN: Indiana Univ. Press, 1984.

———. “Metaphor, Dictionary, and Encyclopedia.” New Literary History 15, no. 2 (Winter 1984): 255–271.

Barthes, Roland.   Image-Music-Text. Translated by Stephen Heath. New York, NY: Hill and Wang, 1978.

Kockelman, Paul. “The Semiotic Stance.” Semiotica 2005, no. 157 (November 2005): 233–304.

Liszka, James. A General Introduction to the Semiotic of Charles Sanders Peirce. Bloomington: Indiana University Press, 1996.

Lotman, Yuri M. Universe of the Mind: A Semiotic Theory of Culture. Translated by Ann Shukman. Bloomington, IN: Indiana University Press, 1990.

———. “On the Semiosphere.” Sign Systems Studies 33, no. 1 (2005): 205–229.

Lotman, Yuri M., and B. A. Uspensky. “On the Semiotic Mechanism of Culture.” New Literary History 9, no. 2 (Winter 1978): 211–232.

Nöth, Winfried. Handbook of Semiotics. Bloomington, IN: Indiana University Press, 1990.

Peirce, Charles S. The Essential Peirce: Selected Philosophical Writings (1867-1893). Edited by Nathan Houser and Christian J. W. Kloesel. Vol. 1. Bloomington, IN: Indiana University Press, 1992.

———. The Essential Peirce: Selected Philosophical Writings (1893-1913). Edited by Nathan Houser and Christian J. W. Kloesel. Vol. 2. Bloomington, IN: Indiana University Press, 1998.

Petrilli, Susan, and Augusto Ponzio. Semiotics Unbounded: Interpretive Routes Through the Open Network of Signs. Toronto: University of Toronto Press, 2005.

Sebeok, Thomas A., and Marcel Danesi. The Forms of Meaning: Modeling Systems Theory and Semiotic Analysis. Mouton De Gruyter, 1999.

Sebeok, Thomas. Signs: An Introduction To Semiotics. 2nd ed. Toronto: University of Toronto Press, 1994.

van Leeuwen, Theo. Introducing Social Semiotics: An Introductory Textbook. New York, NY: Routledge, 2004.

Linguistic Foundations

de Saussure, Ferdinand. Cours De Linguistique Générale. Paris: Payot, 1900.

———. Course in General Linguistics. Trans. Wade Baskin. New York: Philosophical Library, 1959.

Chomsky, Noam. Syntactic Structures. First edition, 1957; 2nd ed. Berlin and New York: Mouton De Gruyter, 2002.

———. Aspects of the Theory of Syntax. Cambridge, MA: MIT Press, 1965.

———. Language and Mind. 1st edition, 1968; 3rd. Cambridge University Press, 2006.

Jackendoff, Ray. Foundations of Language: Brain, Meaning, Grammar, Evolution. New York, NY: Oxford University Press, USA, 2003.

———. Language, Consciousness, Culture: Essays on Mental Structure. Cambridge, MA: MIT Press, 2007.

———. “The Parallel Architecture and Its Place in Cognitive Science.” In The Oxford Handbook of Linguistic Analysis, edited by Bernd Heine and Heiko Narrog, 643–668. New York, NY: Oxford University Press, 2009.

———. Meaning and the Lexicon: The Parallel Architecture 1975-2010. New York, NY: Oxford University Press, USA, 2010.

———. “What Is the Human Language Faculty? Two Views.” Language 87, no. 3 (September 2011): 586–624.

Useful Handbooks and Introductions: Linguistics and Semiotics

Cobley, Paul, ed. The Routledge Companion to Semiotics. New York, NY: Routledge, 2009.

Gonzalez-Marquez, Monica, et al., eds. Methods in Cognitive Linguistics. Amsterdam: John Benjamins, 2007.

Heine, Bernd, and Heiko Narrog, eds. The Oxford Handbook of Linguistic Analysis. New York, NY: Oxford University Press, USA, 2012.

Noth, Winfried. Handbook of Semiotics. Bloomington, IN: Indiana University Press, 1990.

Radford, Andrew, et al., eds. Linguistics: An Introduction. 2nd ed. Cambridge, UK: Cambridge University Press, 2009.

Cognitive Science Backgrounds

Clark, Andy. Natural-Born Cyborgs: Minds, Technologies, and the Future of Human Intelligence. New York, NY: Oxford University Press, USA, 2003.

*———. Supersizing the Mind: Embodiment, Action, and Cognitive Extension. New York, NY: Oxford University Press, USA, 2008.

*Deacon, Terrence W. The Symbolic Species: The Co-evolution of Language and the Brain. New York, NY: W. W. Norton & Company, 1997.

Dehaene, Stanislas. Reading in the Brain: The Science and Evolution of a Human Invention. Viking, 2009.

Donald, Merlin. Origins of the Modern Mind: Three Stages in the Evolution of Culture and Cognition. Cambridge, MA: Harvard University Press, 1991.

Dror, Itiel E., and Stevan Robert Harnad. Cognition Distributed: How Cognitive Technology Extends Our Minds. John Benjamins Publishing, 2008.

———. “Offloading Cognition Onto Cognitive Technology.” In Cognition Distributed: How Cognitive Technology Extends Our Minds, 1–23. Amsterdam and Philadelphia: John Benjamins Publishing, 2008.

Hollan, James, Edwin Hutchins, and David Kirsh. “Distributed Cognition: Toward a New Foundation for Human-Computer Interaction Research.” ACM Transactions, Computer-Human Interaction 7, no. 2 (June 2000): 174–196.

*Hutchins, Edwin. Cognition in the Wild. Cambridge, MA: MIT Press, 1996.

*Pinker, Steven. How the Mind Works. New York, NY: W. W. Norton & Company, 1997.

———. “Language as an Adaptation to the Cognitive Niche.” In Language Evolution, edited by Morten H. Christiansen and Simon Kirby, 16–37. New York, NY: Oxford University Press, USA, 2003.

*———. The Language Instinct: How the Mind Creates Language. New York, NY: William Morrow & Company, 1994.

Tomasello, Michael. Origins of Human Communication. Cambridge, MA: The MIT Press, 2008.

Journals

Sign Systems Studies

Semiotica

The American Journal of Semiotics

Signs

Zeitschrift für Semiotik

Applied Semiotics / Sémiotique appliquée