Capturing, Tracing, Transferring the aural flux Convergence, interactivity, in situ composition and emergent-instruments in […]S[…]S[…]S[…]

Mauricio Arturo Meza Ruiz – [email protected]
  University of Reims Champagne-Ardenne/Conservatory of Reims/Césaré-CNCM

Introduction

The hint-corner stone of a body of work

All of the instrumental composition works of my first period² are hand-made pieces that I wrote, each on its own time, in one stroke from beginning to end, allowing very scarce intervention of technology in the process. Although, as a part of my creative engagement, I have always been actively questioning the possibilities and artistic outputs of technology, willing to find my way into the digital realm and how to achieve its formal integration into my compositional process, which concentrates primarily in the composition of chamber music. As a consequence, I undertook into the creation of another body of work, parallel to my instrumental composition output, focusing primarily on the exploration of the possibilities of interactivity, as it is conceived in the digital domain. From the tension produced by this double bind frame of work, raised a question that became the hint-corner stone of the reflections nurturing the work I have been engaged in lately: what is a musical culture and how musical imagination appears and individuates itself ?³ Assuming that every musical culture emerges and takes-shape throughout a historical and temporal complex process, in which a large variety of terms, conditions, factors and entities enter into systemic interaction, one of the purposes of this project is to create a body of work reflective on the poetics of sowing-and-harvesting. Such a body of work shall be constituted by different types of artistic output, going from instrumental and mixed compositions, to sound installations, interactive multimedia devices and furthermore, a radio-opera.

A protocol for […]S[…]S[…]S[…]

Moreover, what interested me about digital technologies, just as much as interactivity, is the broad range of possibilities they offer in terms of the creation and formalisation of convergences⁴, either between the materials of art themselves, as well as between the diversity of aesthetic operative behaviours⁵ art itself implies. As a consequence, the first challenge and the first task to be carried out in this context, is the constitution of an artistic research and experimentation protocol, following the idea of ??an aurallity emerging from the interactivity⁶ among an ensemble of inter-linked entities, in the centre of which is a dialogical relationship with sound⁷. From the interaction with sound and visual flows generated in real time, and making usage of generative-evolutionary large temporal-arc processes, as well as the constitution of databases, such a protocol should as well be comprised as a poetic play ground for the observation of different kinds of correlations between aural perception, the proprioceptive system, natural and artificial memory, and the cultural and natural environments. In order to explore some of the many possibilities suggested by these premises, I have been engaged in the production of a large-scale, long-term, and interdisciplinary body of work I’ve entitled […]S[…]S[…]S[…] (a.k.a Seed / Simiente / Semance, a.k.a 3S, a.k.a Tres-es series, a.k.a The Trip-less series). Particularly, since 2011, I’ve had the opportunity to create an ensemble of interactive multimedia devices⁸ all of which follow, partially or totally, the structure provided by the same protocol.

A compositional operative regime for in situ composition

In a good part of the interactive devices conforming the […]S[…]S[…]S[…] project, a series on which I’ve been working since 2005, a metaphor of aural collective memory is created on the first part of the process. Subsequently, this same process generates the conditions for a performance to take place, in which the action unfolds as a new endogenous process that I call in situ composition. When the convergence of this processes reaches its climax, the performative situation is enlarged towards an event that is much like an electroacoustic mixed-music concert in which one can experience the manifestations of an entity whose qualities and characteristics tend to determine a musical and conceptual object that I have been calling since 2012, emergent-instrument⁹. The theoretical frame at the foundations of this body of work relies on what I consider as a compositional operative regime based on the interaction among three core notions, which are: capture, trace and transfer. This operative regime on itself is completely dependent of a collective and individual interactive relationship with sound that I conceive as a dialogical relationship with aural matter, that is: live performance with sound, mediated by digital technologies. In this article, I will take-on the subject of in situ composition, the methodology it implies and some of it’s outputs by introducing the notions of capture, trace and transfer in the context of a general description of one of my interactive devices, Doppelgänger_Oblivion: Timeline (Dieburg, 2012)¹⁰. Starting by underlining the aspects that make of it a space and a time of interactivity and convergence, my exposé will point at some prospective aspects of the emergent-instrument while at the same time offer a description of some of the functions implemented in the software used in my interactive devices for the purpose of in situ composition.

Converging processes in Doppelgänger_Oblivion: Timeline

Doppelgänger_Oblivion: Timeline, a title that could be translated as “abolition of the double in the line-of-time”, is a device¹¹ that is first presented as an interactive multimedia installation, then, in the course of a circular process¹², it progressively transforms itself into a space in which an ensemble of entities, situations and activities get involved, with their respective spaces and regulators, in a relationship towards the generation and the evolution of the symbolic content, and audio and visual output of the system itself. Thus, an interactive multimedia installation, a composition workshop and a space for musical performance converge in the same space and time of exposition of the device. In the last instances of the process, as a result of the recurrence of this inter-activated convergences, emerges a new and autonomous entity, whose structural, material and behavioural characteristics, and whose aesthesic qualities of sound, visual, spatial and symbolic manifestations suggest, in the manner of a musical instrument, a framework of reference for the composition of a particular kind of electroacoustic mixed music, where an interpreter interacts with the composed and emergent qualities of such an entity. Figure 2 illustrates the convergence of modes, situations and processes as they occur in the time of exposition of the device.

First, by tracking the behaviours of the visitors, the installation mode enables a process of capture, which in turn leads to the creation of a database — a metaphor of collective aural memory. Afterwards, this same process generates the conditions for the emergence of the in situ composition process. Convergent with the first one, this process engages the composer as a performer within the installation. As it allows the composer to explore the database and to use its content for creative purposes, this phase in turn, is at the crossroads of the capture and transfer operative realms. When the convergence of this iterative situations and processes reaches its climax, the performance is enlarged towards a particular event that invites a musician to perform a score in an interactive situation involving the system as an instrument. Much like an electroacoustic mixed music concert, this phase allows the visitors to experience the manifestations of the emergent-instrument, an entity whose qualities and characteristics are endogenic and endemic in relation to the ecosystem represented by the device.

Capture process: Interactive multimedia installation mode

Interactive sound space

In the interactive multimedia installation mode of Doppelgänger_Oblivion: Timeline¹³ [Figure 3], a continuous flow of sound and light inhabits the space. The visitors are invited to wander in, while in the course of their visit, they may discover, or not, that their walk influences the behaviour of the flow of sound and light. Sound is generated in real-time by means of additive synthesis, controlled by a programmed looped structure that goes progressively from the projection of a single sinusoidal wave, all the way up into a wall of sound¹⁴, passing through successive stages, from the constitution of a harmonic spectrum, to an inharmonic one, until it reaches a state of relatively high saturation and then, back forward progressively into the projection of a single sinusoidal wave. On the other hand, the sound generated by additive synthesis is transformed in real-time throughout its evolution, with a combination of treatments like distortion, delay, reverb and granulation. This allows for the walk of the visitors to influence the resulting sound on its behaviour and its evolution, both at the level of its macro-temporal and micro-temporal articulations, resulting in a modulation of the overall behaviour of the additive synthesis cycle, from one instantiation to another of the capture-trace-transfer cycle. As for the light flux [Figure 1], it is generated by means of a video projection of synthetic images, which in turn evolve in correlation with the overall sound process¹⁵.

From capture to timeline and generative aural and visual outputs

When a visitor enters the installation, a video camera detects his / her position and then, by means of a program written with Max¹⁶, the speed and the direction of his or her walking are calculated. With this information captured by the system, a database is created, assigning to each visitor a data-structure¹⁷. There after, from a specific data-structure and according to its pertinence in relation to a set of rules, the program generates a timeline corresponding to the behaviour of a particular visitor. This timeline is then on the one hand, stored in memory for a later use while on the other hand, it is compared and filtered in real-time by the program with respect to a previously written master-timeline. From this operation of analysis, filtering and concatenation, the software determines the evolutionary course of the multiple parameters¹⁸ that come into play in the generation of the sound and visual flow projected by the installation [Figure 4].

Engaging the visitor towards attentive listening

In this first phase of the process, as the visitors are not allowed to learn that their movements are being recorded by the system, the purpose to record their behaviours in order to stock them and create a data base is therefore not intended to be part of the visitors first aesthetic appreciation of Doppelgänger_Oblivion: Timeline. Moreover, this implies that the interactive installation mode of the device has to stand on its own as a unique, and necessarily interesting, aesthetic experience for each visitor¹⁹, appealing to his or her engagement into the transformation of the aural and visual environment. Thus, at this stage, the visitor is expected to assume a critical position towards what he or she hears and sees, and from that point on, being able to express, through her / his ambulation, an index of sensitive preferences. The protocol at the foundations of this development implies that these sensitive preferences are, a priori, captured by the system, that is, the time spent in the installation by the visitors, as well as the shape resulting form his or her overall displacement, provide an index of intensities that can be correlated to a coefficient of attentive listening.

Trace process: Ecosystemic performance mode

From awareness to participation

The protocol implies the assumption, by the visitors, of a greater responsibility for their actions, as they become more aware of what the interactive situation has to offer. Hence, their status evolves from that of occasional, casual, lambda visitors to that of performers, thus taking part in the overall creative process proposed by the ecosystemic performance context offered by the device. From here, the performance mode of Doppelgänger_Oblivion allows to consider each of the participants, including the machines, as potential performers that progressively contribute, through their interactions, to the establishment of an ensemble of conditions that lead to the emergence of an artificial musical culture, proper to Doppelgänger_Oblivion as an ecosystem. Figure 5 illustrates the relationship amidst the performers in the ecosystem.

Building a database, an in situ composition task

As foretold, after some time running as an installation, the system builds a database from the information provided by the visitors through their behaviours [See figures 6 and 7]. This allows Doppelgänger_Oblivion to initiate a new process in which the installation becomes the background for a performative situation. In this one, a composer, or a sound artist, is invited to perform the writing of a composition in the situation offered by the installation, that is, to intervene assuming her / his own role as a composer in a performance within the same space-time of the installation. So, as the installation mode is still running²¹, the functions of an interactive multimedia installation converge with those of a composition workshop²² that allows, on its own means and terms, to create a composition from the different materials generated within the same interactive installation. Thus, the data structures conforming the database should be considered as essential factors in the sonority of the instrument-in-process-of-emergency, however, the composer-in-the-situation has authority to include or discard, with respect to his or her in situ composition project, the sonority implied by these blocks of information, that is, one can choose, for example, to use them as pre-written whole sections to be inserted in the score and / or using them by giving preference to other structural or aesthesic of their qualities and potentials, some of which the composer should ideally discover during the in situ composition phase.

In an out of the system: towards a dialogue between more than two

worlds

Ideally, as suggested before, the output of such a performance of in situ composition should be in resonance with the interactive, symbolic, aural and visual qualities of the ecosystem the device represents itself. Thus, the composer encounters with the challenge to find a way to preserve the stylistic elements of his or her own compositional language, while at the same time, sustain the aesthetic qualities prefigured by the system. Nonetheless, the protocol not only allows, but demands the projection of the device’s resonance far beyond the frontiers of the ecosystem. This is the case, for example, of my second string quartet, a forth coming piece on which I have been working since 2015 and where, in order to generate the materials for this particular work, I make usage of the Doppelgänger_Oblivion: Timeline, and the [L’emprise de Panurge]:::[Du feu de Toungouska], as well as the Un theatre des opérations²³ data bases. [See figure 10].

Composition tool for retrieving the “future-past”

In order to make use of the information generated by the system for in situ compositional intent, the same software implemented to allow Doppelgänger_Oblivion to run as an interactive installation-performance, allows access to the data captured during the first phase of the process, opening the way for the application of different methods in terms of how to interrogate the database for creative purposes. Recall and retrieve, graphic generation, reverse play, as well as temporal compression and dilatation, figure among the functions hereafter described and already implemented in the software. Still, this functions should be understood not as common-core composition digital tools-of-the-trade that one can find here and there, among the many existant software developments, but rather as endogenous and emergent qualities, bearing potential functionalities, that the system itself brings about, all the way from the process of its conception, up until its mise en route, realisation and current development.

Recall / retrieve function

As every trace left by the visitors in the system is correlated to the dynamic and parametric control of real-time transformation of the audio-visual projection, this function affords with the possibility of seeing and listening to a particular state in which the aural and visual flow was behaving at some specific instant of its evolution. By allowing a perceptual analysis of the aural and visual properties of a particular instance in the history of the ecosystem, that is, allowing to re-run the parameters involved in the transformation of the flow of sound and light between a moment ‘a’ and a moment ‘b’ of its gradual development, this function provides the composer with the means to evaluate and classify, from a sensitive perspective, the traces left in by the visitors-performers, in the memory of the ecosystem.

Graphic generation

By implementing the possibility to render a graphic representation of the ambulations of the visitors in a two-dimensional rectangular space, where axis ‘x’ represents the duration of the movement and axis ‘y’ corresponds to its direction, it becomes possible in this way to operate a transfer of the spatial and temporal indices generated by the displacement of the body in the installation towards other spatial and gestural realities, such as the one represented in the instrumental domain. Figure 8 shows the graphic visualisation generated from the trace left by visitor “user211” ; the software allows for a modulation of the rate of graphic-visual resolution.

In the first instantiation of Doppelgänger_Oblivion, a score for solo trombone and emergent-instrument²⁴ was written and performed ²⁵. In this one, the lines that describe the displacement of the visitors-performers are interpreted by the trombonist either as modulations to be carried out with the pressure of the lips, or as glissandos to be played with the instrument’s slide bar. Other articulations were inserted in the score, all of them related to the technical possibilities of the trombone [Figure 9]. As for my second string quartet, on its current state of development, the data corresponding to the displacement of the body of the visitors-performers is being transferred to the movement of the left hand across the fingerboard [Figure 10].

Reverse play, and temporal compression and dilatation

Temporal compression and dilatation, as well as reverse-play functions are implemented in order to allow adaptation to the constraints implied by the usage of traditional instruments and by the same token, afford with a more tangible flexibility of articulation between the digital and the analogical domains of the ecosystem, as is the case, for example, when dealing with breathing with the trombone. In Doppelgänger_Oblivion: Timeline, each of the aural and visual frames corresponding to the data structures chosen by the composer-performer for the trombone part were projected alongside the instrumental execution, with some of them in reverse play and / or compressed mode. As for the temporality proper to each memory-trajectory data structure chosen for the performance, operations of temporal compression and / or temporal dilation were applied, in such a way that the duration of each projected trace is relative to that of the instrumental gesture. Some other traces were projected in their original form and duration, intercalated with the parts of the trombone.

Transferring the flux: a projection towards the embodiment of the emergent-instrument

As it has been stated before, in the last stage of the process, the convergence between the modalities of installation and performance entails the possibility of the integration of a musician as an interpreter, so that the device manifests, on the one hand, the qualities of an aural projection space suited for the interpretation of the same score generated during the preceding stages, and on the other hand, the qualities of an electronic instrument whose behaviour and sonic, visual and spatial attributes are intimately linked, both in an ontogenetic and phylogenetic way, to the whole generative, transformational and evolutive process of the ecosystem. By train of consequence this implies, ideally, the generation of new dialogical conditions within every cycle of capture-trace-transfer processes, that which in turn results, with every recurrence, in the composition and the interpretation of a piece in which the instrumentalist inhabits, in the course of the performance of the score, an aural space that in turn is, an integral and emergent part of the system.

Towards music composition as collectively programming the individual

The type of electroacoustic mixed music emerging from the process described here implies a particular approach to composition. In situ composition, in this context, needs first of all to be established from the perspective of a formal approach to digital convergence and interactivity, this in turn, traduces into the conception of a dynamic space in which a capture of information is performed. The data captured here is that of the interactivity happening in the device as an installation. My assumption is that this kind of interactivity, proper to the digital realm, should be conceived as the pursuit of a dialogical relationship with sound, such a relationship being articulated by the dynamism of the body and mediated by digital code. The next step is to interrogate the outputs of the capture processes in order to determine what might have a function of material for the means of composition. Here, the composer-performer, in the manner of an archeologist observes, during the time of convergence between the installation and performance modes, some of the data structures generated by the system, and thus determines their relevance for a composition project to be made within the context offered by the device itself: a performative situation, in addition to a dialogical one. Finally, a transfer of the structures captured in the domain of interactivity to the domain of musical performance is operated. Furthermore, the idea of an instrumental entity emerging from the convergence taking place in this ecosystemic environment implies the inclusion of the aural and visual materials generated in the installation-performance phase as part of the in situ composition and consequently, in situ interpretation projects. This in turn allows me to touch on the subject of the status of the digital trace, as it is comprised in this context, by proposing a preliminary two-pronged definition of it. Firstly, the trace should be comprised as code emerging from a composed and mediated aural interactive situation — a programmed dialogical relationship with digital sound — that has transferable qualities and properties such as an index given by a date of beginning, a shape, a temporality of its own, and correlativity to historical sound parameters and visual and symbolic outputs. And secondly, as being the point of convergence, both at a structural and an operational level, among the capture and transfer realms. Hence, the digital trace, having a particular status concerning the articulation of the different modes of activities and participation in the ecosystem, the whole process implied by the device moves towards collectively writing the program of the emergent-instrument.

Conclusion

The protocol at the basis of this development can be synthesised by the operative conjunction: capture-trace-transfer, where capture is to be understood as the operation of exchange, recollection and symbiotic inter-conditioning ; trace-tracing is the operation of rendering what is pertinent for the purposes of in situ composition ; transfer is the operation of re-enacting what has been given in the course of capture and trace processes or, allowing myself an expression inspired by the philosophy of Jacques Derrida: re-actualising the system by lending a presence to the ghosts of the visitors. In the last instance of the process described here, in situ interpretation, as the last stage of the operation of transfer, concerns the musician, both as performer and interpreter, as well as the machine, as it mediates the whole process within itself, suggesting the involvement of an agent-code whose status of steward of the interactivity, and withholding inherent aesthesic qualities, just as much as operative potentialities, suggest in turn a certain amount of autonomy. Moreover, in a forthcoming development, the system should be able to regulate itself in order to stabilise its audio-aesthetic output, and hence be able to render an aural proposition of its own. Capture, trace, transfer and moreover, emergency and individuation, as well as digital convergence, all of it in the context of my artistic project, suggest a compositional approach on the convertibility bearing among structures and operations, that is, an allagmatic²⁷ approach to music creation. As the core question at the foundations of this project is that of the emergence of music as culture and imaginary, the file rouge of the collective performance taking place in the installation is that of the birth of music and the dramaturgy that such an event could imply. All of what should be deepen in the subsequent developments of the “Trip less: abolition of the double in the line-of-time”.

Bibliography

Agamben, Giorgio. Qu’est ce qu’un dispositif. Paris: Éditions Payot Rivages, 2014.

Ashby, W. Ross. An introduction to cybernetics. Londres: Chapman & Hall Ltd., 1957.

Barbaud, Pierre. Initiation à la composition musicale automatique. Paris: Dunod, 1965.

Barkati, Karim : “Notice de a2m-009c”, December 2008 (online): http://karim.barkati.online.fr/Universite/Doctorat/Logiciels/ApplicationsMacOS/a2m/ (visited 23 March 2017).

Boissiere, Jean-Louis. La relation comme forme. L’interactivité en art. Genève: Éditions Mamco, 2004.

Bossis, Bruno. “Ecriture instrumentale, écriture de l’instrument.” In Composer au xxie siècle. Pratiques, philosophies, langages et analyses. Edited by Sophie Stévance, 119-135. Paris: Vrin, 2010.

Criton, Pascale, and Jean-Marc Chouvel, ed. Gilles Deleuze. La pansée-musique. Paris: CDMC, 2015.

Cowell, Henry. New musical resources. New York: Cambridge University Press, 1996, (first edition 1930).

Chouvel, Jean-Marc. Analyse musicale: Sémiologie et cognition des formes temporelles. Paris: L’Harmattan, 2006.

Chouvel, Jean-Marc, and Xavier Hascher, ed. Esthétique et Cognition. Paris: Publications de la Sorbonne, 2013.

Couchot, Edmond, and Norbert Hilare. L’art numérique. Paris: Flammarion, 2003.

Couchot, Edmond. La nature de l’art. Ce que les sciences cognitives nous re´ve`lent sur le plaisir esthe´tique. Paris: Hermann E´diteurs, 2012.

Couchot, Edmond. “Médias et immédias.” In Art et Communication. Edited by Robert Allezaud, 101-106. Paris: Éditions Osiris,1986 http://www.multimedialab.be/doc/citations/edmond_couchot_medias.pdf (visited 23 March 2017)

Deleuze, Gilles, and Felix Guatari. Capitalisme et Schizophrenie. Tome 2: Mille plateaux. Paris: Editions de Minuit, 1980.

Derrida, Jacques. De la grammatologie. Paris: Editions de Minuit, 1967.

Derrida, Jacques. Trace et archive, image et art. Bry-sur-Marne: INA Éditions, 2014.

Di Scipio, Agostino. “ «Sound is the interface»: from interactive to ecosystemic signal processing.” Organised Sound Vol 8 No. 3 (2003): 269-277.

Estrada, Julio, “Le continuum in musique: structure et ouvertures in composition, ses derivations esthétiques.” In Ästhetik und Komposition. Aktualität der zur Darmstädter Ferienkursearbeit. Edited by Borio, G. and Mosch, U., 50-65. Mainz: Schott, 1994.

Gardiner, Michael E. “Le défi dialogique de Bakhtine aux sciences sociales.” Slavica Occitania 25 (2007): 67-87.

Leman, Marc. Embodied Music Cognition and Mediation Technology. Boston, Massachusetts: MIT Press, 2008.

Merleau-Ponty, Maurice. Phe´nome´nologie de la perception. Paris: E´ditions Gallimard, 1945.

Maturana, Humberto, and Francisco Varela. Autonomie et connaissance: Essai sur le vivant. Paris: Editions du Seuil, 1989.

Meza, Mauricio, and Frédéric Dufeu. “Dopelga¨nger_Oblivion: Timeline. An interactive multimedia installation for collective in situ composition.” Proceedings of the international conference The global composition. Conference on Sound, Media, and the Environment. Dieburg: 2012, pp. 459-461.

Rancière, Jacques, “The Emancipated Spectator” In Artforum (2007): 271–281.

Schaeffer, Jean-Marie. Adieu à l’esthétique. Paris: Presses Universitaires de France, 2000.

Sebbah, François-David, “Traces numériques: plus ou moins de fantôme(s) ?”, In Le sujet digital, edited by Larsonneur, Claire, Regnauld, Arnaud, Cassou-Noguès, Pierre, Touiza, Sara, 114-126. Paris: Les presses du réel, 2015.

Sha, Shin, Wei. Poiesis and Enchantment in Topological Matter. Boston, Massachusetts: MIT Press, 2013.

Simondon, Gilbert. Du mode d’existence des objets techniques. Paris: Éditions Aubier, 1958.

Simondon, Gilbert. L’individu et sa genèse physico-biologique. Paris: Editions Jérôme Millon, 1998.

Stockhausen, Karlheinz. “Wie die Zeit vergeht.” Die Reihe n°3 (1957). Translated to french by Christian Meyer: “Comment passe le temps”, Contrechamps n°9 (1988): 26-65.

Varela, Francisco, and Evan Thompson, and Eleanor Rosch. L’inscription corporelle de l’esprit. Sciences cognitives et expérience humaine. Paris: Seuil, 1993.

Weissman, Joseph, Fractal Ontology blog https://fractalontology.wordpress.com/2007/11/28/a-short-list-of-gilbert-simondons-vocabulary/#more-342 (visited 23 March 2017)

Xenakis, Iannis. Musiques Formelles. Paris: La Revue Musicale n°253-254, Editions Richard-Masse, 1963.

Notes