Semiotic approaches to computing

How we understand computing is related to how we view computing technology. The concept of computers as semiotic machines in new media theory and informatics offers a rich perspective on the intersection of computation, semiotics, and sociocultural meaning-making. A semiotic approach to computing was developed by scholars such as Peter Bogh Andersen [1986], Mihai Nadin [2007], Lev Manovich [2002], and Frieder Nake [2016, 2001]. It positions the computer as a unique technology due to its role in processing, generating, interpreting, and transforming signs within societal and individual contexts.

Driven by formalised logic and rules, aligning with Charles Sanders Peirce’s triadic model of semiosis (sign, object, interpretant), Mihai Nadin [2007] defined that “computers are semiotic machines driven by semiotic engines” [53, p. 95]. Nadin stresses their equivalence to automata, capable of simulating sign processes through structured input-output operations, while acknowledging their limitations in replicating the holistic, context-sensitive nature of human semiotic activity.

Frieder Nake builds on this by contrasting humans as semiotic animals – capable of reflective sign use – with machines as semiotic machines bound to operational, rule-based sign manipulation. Nake has coined the term ‘algorithmic sign’¹ which he understands as a dual entity comprising a visible surface (analog presentation) and a manipulable subface (digital representation). This duality reflects the computer’s role in processing semiotic content, meaning that the surface is the perceptible output (such as icons, images, text), while the subface consists of algorithmic rules and data that generate and transform it. Unlike traditional signs, algorithmic signs are tied to formalized procedures, stripping away context to enable computational operations. Importantly, algorithmic signs are not static but generated through recursive, rule-based processes. While humans interpret signs fluidly, computers process them reductively via syntactic and algorithmic transformations [55, 56].³

This perspective has impact on conceptualising human-computer interaction, and understanding the computer as a powerful media device oscillating between abstraction and embodiment/materialization [57]. To illustrate this, Nake introduces the poetic term "Zeichenhaut" (’skin made of signs’) to describe the process by which all matters must develop a skin consisting of signs to be processed by a computer. Furthermore, this concept emphasizes the circulating and relational character of signs, which are derived from the world but also have formative effects upon it. The use of "Haut" (skin) is particularly noteworthy, as it suggests a tangible connection between signs and the physical world, highlighting the materiality of semiotic processes. For anything to become an object of computation further steps beyond semiotisation, namely formalisation and algorithmisation, are needed. While these steps of abstraction can be viewed as reductive, Nake emphasizes that reduction is only one aspect of abstraction. As demonstrated in his notion of the algorithmic sign, computers are also productive. Lev Manovich’s theories of database logic and modularity in digital media resonate with this concept, as computational semiotics underpins the fragmentation and recombination of cultural symbols in algorithmic environments [54].

With generative AI technology², this productiveness, modularity and the role of data gains a new quality. The concept of computers as semiotic machines reconfigures our understanding of digital technologies as infrastructures for encoding sociocultural meaning. It bridges the gap between mechanistic precision and the fluidity of human interpretation in mediated contexts. This understanding illuminates how new media leverages computational semiotics to automate creative and communicative tasks, from algorithmic art to data-driven narratives. By recognizing the layers of abstraction, transformation and materialization involved in computer-based semiosis, we gain insight into both the limitations and potentials of digital media and generative AI. ⁴

Furthermore, a semiotic approach to computing is tied to how sociotechnical aspects can be situated in computer science. Heidi Schelhowe [58] argues that fundamental differences in the understanding of the computer as either an information-processing or data-processing machine are decisive for an inclusion or exclusion of the social. Schelhowe follows Nake’s et al. [1992] perspective on computers as semiotic machines and the duality of the algorithmic sign. She points out that computers process data:

“Only through the addition of people, of human activity, human work, can the signs acquire meaning, semantics, i.e. become information [...] The fact, which has been increasingly discussed in the theory of science in recent decades, that social conditions find their way into the construction of technology and that the technical construct in turn changes social reality, finds its immanent confirmation here. [...] By becoming aware of its social implications, computer science is opening up to social discourse, which for a long time could only take place outside the engineering disciplines. This also opens up the possibility of discussing the effects of gender relations on the theory and application of computer science within the discipline itself.” [58, p. 54] ⁵

Expanding on the inclusion and exclusion of the social, an artificial separation of ‘the technical’ vs. ‘the social’ in relation to computer science is especially common in Germany. The (west-) german origin (hi)story (geschichtliche Gewordenheit) of computing and notably its strong localisation in engineering and mathematics had implications for the scientific self-image as well as the external perception of the discipline that are still relevant today.⁶ Here, computer science – as a whole, or at least at its core – is still frequently perceived today as an objective, rational, logical and hegemonic, command-control based discipline and therefore fundamentally separate from the social, creative and positivistic (the ‘other’). Opening computer science to other disciplines is possible, but is often only conceded by adding ‘the other’, for example via so-called hyphenated disciplines (e.g. media-informatics, socio-informatics), and rarely via an integral anchoring and embedding in computer science itself.

Due to the historical interconnectedness of German computer science, engineering and mathematics, gendered encodings and especially constructions of masculinity of the latter, disciplines have also been incorporated, if partially transformed, into the former.⁷ The artificial separation and exclusion of the social is thus fundamentally also a gendered one, in which computer science includes (white, able-bodied/minded) men ‘qua definition’, but excludes, devalues or even hides e.g. non-male people (as ‘the other’).⁸ Exemplary, women are still often regarded as an exception (Ausnahmefrauen) or their womanhood/femininity is argued away or even denied. On an intersectional level, other – often non-Western – countries with a high quantity of female computer scientist are ignored or their participation is legitimised through argumentation grounded in (anti-muslim) racism. If women are included, they tend to be i.a. white and abled, and in general fitting to normative Western beliefs. ‘Special’, gendered spaces in computer science aiming to increase female participation are being created on biologistic, essentialist assumptions (e.g. by offering (more) social and creative courses such as media design informatics or initiatives explicitly catered to assumed ‘female interests’), and sometimes they are even being used to achieve an illusion of inclusion without actually having to include women in the assumed ‘real’ (core of the) discipline.

All this, again, furthers separation and exclusion not only of the social, but also of specific groups of people, and reinforces existing conditions and normative patterns, be it in relation to approaches, theory, knowledge, participation or outcomes. This results in the lack of awareness not only of differences in understanding of computing and how it came to be, and its social constructions and implications, but also of its inherent social-ness and its importance for the future of computing. This perspective coincides with the findings of feminist critiques of the natural sciences, which succinctly emphasize that a positivist understanding of science, as is traditionally common to the natural and technical sciences, derives its validity precisely from a supposed independence from social references [60].

In contrast to such naturalising and dichotomous approach, a semiotic approach to computing highlights meaning-making aspects of the discipline and computing practice. As David M. Berry [2011] highlights, there are a multitude of entanglements between computers and our lives, with computational technologies in turn also mediating their relationships with each other through software. Spun like webs, a continuous formalizing, translating, mediating, and interpreting of signs takes place by human and non-human participants. In these processes, software - but also by extension processes of informatics in general - turns into a knot tying together the material and ephemeral components [61]. For computing to not lose sight of these entanglements, such a feminist material-semiotic approach to the algorithmic sign might allow to draw a wider net when examining computational technologies.

¹ Sign = A relation not a thing; thereby it's never natural and always human-made… [56]

² Regarding (Generative) AI Technologies and its influence on the triad of object, sign and interpretant:
In 2001, Nake states: ‘The interpretation of the computer is the precise and repeatable execution of a calculable function. We would be dissatisfied if it were otherwise. The algorithmic sign is thus a sign that is determined by a simultaneous process of interpretation and determination.’ [56, p. 5, translated by DeepL.com] in case of ML and GenAI: the computer’s interpretation becomes undetermined and the “task” of the human extends from interpreting the sign to also interpreting the computing itself and being aware of its partial (non-)interpretability → Thus the crisis and current dissatisfaction?

³ Similarly, David M. Berry [2011] highlights the need for translation for computer mediation to take place, as they are specific temporal durations and symbolic sets of discrete data to represent reality. After context is removed through this encoding process, reassembling, filtering, and transforming can take place endlessly.

⁴ Additionally, and also in relation to ² , with the translation of information in context for computational and algorithmic purposes in ML and GenAI, a new (computational) way of a understanding reality emerges, and the output of these models in turn presents a new type of knowledge and new method to control reality [61]. Accordingly, the continued use of of generative AI in various domains also shapes a new conception of reality based on signs tied to formalised procedures.

⁵ This in turn highlights the importance of an entanglement of computing and social perspectives or a softening off the edges between engineering and SSH, instead of strictly dividing the examination of computing machines into development on the one side, and critique on the other. Additionally, this has an impact on knowledge production in non-computational disciplines, as computational technologies are not merely another tool or method in their repertoire, but fundamentally impact the knowledge production and research processes.

⁶ Discourse on the scientific categorisation of computing happened/s on national and international levels throughout its history. In Germany, the technical-mathematical situatedness ultimately prevailed.

⁷ As well as social conditions, constructions and beliefs in general.

⁸ E.g. through gender essentialist beliefs of men as logical and rational beings, as well as women as social, nurturing, illogical et cetera.