Literacy Based on Conceptual Graphs: An Argument for a Semiotic Digital Humanities
This paper proposes Charles Peirce’s system of existential graphs (EG) as a theoretical foundation for the digital humanities (DH), arguing that such an approach is particularly insightful for research on education and literacy. While acknowledging its relevance, we challenge the discursive, glottocentric (language-centered) modelling theory on which DH is founded (McCarty 2005), in favour of a semiotic non-glottocentric view. McCarty’s seminal proposal for DH is based on the observation that any computing system depends on a model of the (empirical) world. This expanded the scope of humanities research, particularly as afforded by digital methods and the entailed capacity for computing. However, it still supposed a discursive and relativist notion of model, as inspired from poststructuralism. This obstructs newly found DH from taking the humanities out of the persisting crisis of the last decades (Bérubé, Nelson 1995, Nussbaum 2010, Jay 2014, Martinelli 2016, Cobley 2017) because, we consider, by (infinitely) questioning the reliability of evidence as relative to language, glottocentrism, and implicit epistemological relativism, fails to acknowledge the relation between theoretical models and social progress/change.
More than questioning the borders of literature, this paper questions the limits of language-centered modelling theories based on a notion of meaning as verbalizable, circumscribed by notions of text and discourse. Uprooting the notion of literacy (competences required for surviving and thriving in society) from traditional conceptions of reading and writing as linear, monomodal and linguistic allows its expansion to fit more comprehensively the multimodal affordances of new media. Commonly the concern of the humanities, literacy has been thought to consist in competences of interacting with texts (reading, writing, arithmetic). This implied that education has the role of equipping citizens with a mastery of the established formal, symbolic codes of culture, art and research. It thus supposes that a fixed set of skills (mostly stemming from the liberal arts) both suffices and is mandatory in modern societies. Given the affordances of digital media, the new plethora of competences that gives access to sustainable and thriving lifestyles is inexhaustible by lists of fixed skills. Thus, instead of grounding the notion of literacy in the philological concept of text, we argue for grounding it on the more encompassing notion of model, as the specific scope of DH. For this, we bridge two recent areas of research: (1) the uptake of Peirce’s semiotics in DH modelling (Ciula, Eide 2017, Ciula, Marras 2016, 2019) and (2) the vast semiotic research on multimodality (Kress, van Leeuwen 2001, Kress 2010), particularly vis-à-vis education (Kress 2003, Lacković 2018).
The recent uptake of Peirce’s semiotics in DH has the rationale of developing a pragmatic modelling theory, tailored for digital methods by its reliance on schematic signs (icons, diagrams) as instrumental for modelling, in contrast to formal symbols, often understood as supposing verbalization. Schematic signs represent their object on account of similarity or, generally, structural resemblances, not requiring convention or higher-level formalization in their constitution. The proposal of such a DH modelling theory (Ciula, Eide 2017) relies on two recent uses of Peirce’s semiotics, namely Kralemann and Lattmann’s (2013) semiotic model of iconic modelling and (2) Elleström’s (2013) iconicity theory. These developments follow the recent iconic turn (Boehm, Mitchell 2009, Moxey 2008), which renewed interest for Peirce’s semiotics, given the central role that schematic signs play in his logic (Pitarainen 2006, Stjernfelt 2007). The iconic turn consists in the idea that meaning is primarily evoked by schematic, non-linguistic signs. While the language turn (Rorty 1967) considered language, instead of (Cartesian) ideas, as the vehicle of epistemology, the iconic turn moves away from both idealism and glottocentrism by a notion of knowledge as primarily schematic and, thus, broader than strictly analytical. This supposes switching from language-based to phenomenological, embodied epistemology, whereby cognitive schemata are responsible for apprehension and for the development of complex, symbolic systems. A quintessential claim in Peirce’s semiotics for the iconic turn and for its adoption in DH is that only icons (meaning stemming from shared qualities) can be used as predicates (CP 2.278). This explains why iconicity is central in modelling: an iconic relation between an object and its model warrants that the model preserves certain qualitative features of the object that render it pragmatically operational. If the iconic relation in modelling is ignored, the model can be so abstract in relation to its (presumed) object that it becomes a predicate of something else, of an abstract idea about the object, rather than of the object. Put simply, modelling that is not based on iconicity can result in talking about something else than what it is supposed to address.
Another highly salient but little discussed aspect of Peirce’s semiotics is, in contemporary terminology, multimodality. Peirce disagreed with the mentalistic psychology of his time (Stjernfelt 2014: 14-15, 44), arguing that similarities can be observed across representational modes (CP 7.21-7.48, Cristalli 2017: 38). The discovery of cross-modal icons (resources) can lead to useful predicates (competences) for modelling. Particularly, Peirce deemed the educational programme of modernity responsible for the rigid conception of knowledge and learning as proceeding in linear textuality (CP 1.312). His concept of iconicity affords multimodality in meaning-making, by acknowledging the non-conventionality of cross-modal translations. For example, once perceived, similarities between colours and music can be copulated into one organizing iconic sign such as a synaesthetic representation, which can increase efficiency in modelling. This is highly relevant for digitalization, as digital media exponentially enhanced representation modalities.
The visual representation of data in technology extends from the front-end user interface, through different libraries of programmatic logic, to the machine code itself. The action of sending data over fibre optic cables as pulses of light, or the magnetizing of the hard drive as a simple “on” or “off” (1 or 0), leads to the question of how literate does someone need to be to coherently digitize their actions? Within social media networks and the broader network of the online connectivity of the cloud, users very rarely operate by typing out commands. Instead, such actions are visual and schematic, resulting in a process where users will present themselves – and curate/construct their identity (Georges 2009) – as a collection of images, write using picture-like emoji, and issue instructions to computers using the visual pointer on a button that represents a line of code, which itself represents the dichotomous 1/0 operation of hardware. Based on Peirce’s EGs, the logic and ontology by which a computer processes user queries have been developed as a visual workflow of conceptual graphs (CG) to symbolically represent the data exchange within semantic networks (Sowa 2000). Thus, as digital literacy becomes increasingly pertinent, semiotics can provide an approapriate educational philosophy for contexts where modelling involves cross-modal translations between symbolic and diagrammatic representations.
Multimodality (Kress, van Leeuwen 2001) began, as an area of research, with a criticism of the classic linguistics hypotheses of arbitrariness and double articulation. Traditionally, linguistics, language turn philosophy and discursive theory are based on the supposition that meaning is a conventional (arbitrary) articulation of form and content (Saussure 1916). This view was sedimented by Martinet’s phonological theory, which, in the context of the language turn, advanced a view of language as based on phoneme combinatorics that are arbitrary in relation to represented objects. As digitalization implies that “design and principles of composition move into the foreground” (Bezemer, Kress 2008: 166) in detriment of monomodal, linear text, traditional linguistics and semiotics based on the double articulation hypothesis fall short in comprehending new social dynamics and corresponding new literacies and emerging industries. The technology of natural-language-processing and voice synthesis opens the argument further, with programmes forming combinations of phonemes based on user-inputted visual representation (e.g. the VOCALOID software synthesizer by YAMAHA), or statistical extrapolating the most common combination of such relations (Johnson 2009).
The multimodality concept marked a turn from linguistics to semiotics, as the latter affords a concept of meaning more comprehensive than merely verbalizable content. This framework led to extensive applications in communication (Kress 2010) and education (Kress 2003, Bezemer, Kress 2008). While Peirce’s semiotics is compatible with the modelling affordances of digital media, only recently (Lacković 2018) has it been considered in the multimodality framework. The novelty of the present paper consists in bridging these two approaches in view of Peirce’s rationale of EGs, namely that a semiotic system should afford “a method (1) as simple as possible ([…] with as small a number of arbitrary conventions as possible), for representing propositions (2) as iconically, or diagrammatically and (3) as analytically as possible.” (CP 4.561) This idea is explicitly adopted in DH modelling via Elleström’s proposal that “[a] model should be understood as a clearly outlined cognitive scheme that is both described with the aid of language, and depicted as a diagram.” (2018: 270) This non-glottocentric semiotics, which we develop by bridging the frameworks of DH modelling and multimodality, acknowledges that interpretation is inherently multimodal. Thus, it frames meaning, the central concern of the humanities, as specifically fit for digital contexts, localizing the key to digital literacy in the capacity of discovering iconic relations (inherent in relations formulated symbolically).
References
Bérubé, M. & Nelson, C. (eds.) (1995). Higher Education under Fire: Politics, Economics, and the Crisis of the Humanities. New York, London: Routledge.
Boehm, G. & Mitchell W. J. T. (2009). Pictorial versus Iconic Turn: Two Letters. – Culture, Theory and Critique.
Ciula, A. & Eide, Ø. (2017). Modelling in digital humanities: signs in context. – Digital Scholarship in the Humanities, 32 (1): i33–i46.
Ciula, A. & Marras, C. (2016). Circling around text and language: towards “pragmatic modelling”. Digital Humanities. – DHQ: Digital Humanities Quarterly, 10 (3): 17–29.
Ciula, A. & Marras, C. (2019). Exploring a Semiotic Conceptualisation of Modelling in Digital Humanities Practices. In: Olteanu, A., Stables, A. & Borţun, D. (eds.). Meanings & Co.: The Interdisciplinarity of Communication, Semiotics and Multimodality. Cham: Springer, 33–52.
Cobley, P. (2017). What the humanities are for – a semiotic perspective. In: Bankov, K. & Cobley, P. (eds.), Semiotics and its masters. Vol.1, 3-23. Berlin: De Gruyter Mouton.
Elleström, L. (2013). Spatiotemporal aspects of iconicity. In: Elleström L., Fischer O. & Ljungberg C. (eds.), Iconic Investigations, 95–117. Amsterdam: John Benjamins Pub.
Elleström L (2018) A medium-centered model of communication. Semiotica, 224: 269-293.
Georges, F. (2009). Représentation de soi et identité numérique: Une approche sémiotique et quantitative de l’emprise culturelle du web 2.0. – Réseaux, 154(2): 165–193.
Jay, P. (2014). The Humanities “Crisis” and the Future of Literary Studies. New York: Palgrave Macmillan.
Johnson, M. (2009). How the statistical revolution changes (computational) linguistics. Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics, 3–11.
Kralemann B. & Lattmann C. (2013). Models as icons: modeling models in the semiotic framework of Peirce’s theory of signs. Synthese, 190(16): 3397–3420.
Kress, G., van Leeuwen T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold.
Kress, G. (2003). Literacy in the New Media Age. London: Routledge.
Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. London: Routledge.
Lacković N. (2018). Analysing videos in educational research: an “Inquiry Graphics” approach for multimodal, Peircean semiotic coding of video data. Video Journal of Education and Pedagogy, 3(6).
McCarty, W. (2005). Humanities Computing. New York: Palgrave Macmillan.
Moxey, K. (2008). Visual Studies and the Iconic Turn. Journal of Visual Culture, 7(2): 131-146.
Nussbaum, M. C. (2010). Not for Profit – Why Democracy Needs the Humanities. Princeton: Princeton University Press.
[CP=] Peirce, C. S. (1931–1935, 1958). The Collected Papers of Charles Sanders Peirce. Eds. Hartshorne, C. Weiss, P.; Burks, A.W. Cambridge: Belknap.
Pietarinen, A.-V. (2006). Signs of Logic: Peircean Themes on the Philosophy of Language, Games, and Communication. Dordrecht: Springer.
Sowa, J. F. 2000 [1999]. Knowledge Representation: Logical, Philosophical, and Computational Foundations. Pacific Grove: Brooke Cole Publishing Co.
Stjernfelt, F. (2007). Diagrammatology. An Investigation on the Borderlines of Phenomenology, Ontology and Semiotics. Dordrecht: Springer.
Stjernfelt, F. (2014). Natural Propositions: The Actuality of Peirce’s Doctrine of Dicisigns. Boston: Docent Press.