PDCON:Conference/A Pd framework for the Xth Sense: enabling computers to sense human kinetic behaviour: Difference between revisions

Revision as of 16:36, 13 July 2011

== A Pd framework for the Xth Sense: enabling computers to sense human kinetic behaviour==

Author: Marco Donnarumma

The Xth Sense is an interactive system for the biophysical generation and control of music. It makes use of muscle sounds1 produced by a performer as both raw sonic material and control data. Presently the Xth Sense (XS) technology consists of low-cost, wearable biosensors and a Pure Data2 based application for capture, analysis, real time processing and playback of human muscle sounds. The technical implementation of the XS biosensors has been recently illustrated in [2]. This paper describes the design of the XS software; it is a program that enables a computer to “listen” to the MMG signals transduced by the XS biosensors, to understand the performance main features, and therefore to interact with the performer. After a brief introduction on the nature of the interaction fostered by the XS technology, I focus on the framework main features such as: the XS library, a tabbed dynamic interface (TDI), a MMG features extraction unit, and a graph-on-parent3 (GOP) routing system for dynamic mapping of gesture to sound.

[2] M. Donnarumma: “Xth Sense: researching muscle sounds for an experimental paradigm of musical performance” Proceedings of the Linux Audio Conference (LAC'11), 2011.

[1]

@@ Line 1: / Line 1: @@
-== Researching muscle sounds for an experimental paradigm of musical performance ==
+== A Pd framework for the Xth Sense:
+enabling computers to sense human kinetic behaviour==
 Author: Marco Donnarumma
-Biosensing musical technologies use biological signals of a human subject to control music. One of the earliest applications can be identified in Alvin Lucier's Music for Solo Performer (1965). Alpha waves generated when the performer enters a peculiar mind state are transduced into electrical signals used to vibrate percussion instruments. Over the past thirty years biosensing technologies have been comprehensively studied and presently notable biophysical-only music performances1 are being implemented at SARC by a research group lead by the main contributor to the Bio Muse project Ben Knapp.
+The Xth Sense is an interactive system for the biophysical generation and control of music. It makes use of muscle sounds1 produced by a performer as both raw sonic material and control data. Presently the Xth Sense (XS) technology consists of low-cost, wearable biosensors and a Pure Data2 based application for capture, analysis, real time processing and playback of human muscle sounds. The technical implementation of the XS biosensors has been recently illustrated in [2].
+This paper describes the design of the XS software; it is a program that enables a computer to “listen” to the MMG signals transduced by the XS biosensors, to understand the performance main features, and therefore to interact with the performer. After a brief introduction on the nature of the interaction fostered by the XS technology, I focus on the framework main features such as: the XS library, a tabbed dynamic interface (TDI), a MMG features extraction unit, and a graph-on-parent3 (GOP) routing system for dynamic mapping of gesture to sound.
-Whereas biological motion and movement and music are arising topics of interest in neuroscience research, the biologic body is being studied by music researchers as a mean to control virtual instruments. Although such approach has informed gestural control of music, I argue that it overlooks the expressive capabilities of biological sounds produced by the body. They are inaudible but may retain a meaningful vocabulary of intimate interactions with the musicians' actions.
-To what extent could biologic sounds be employed musically? In which ways could the performer's perceptual experience be affected? How could such experimental paradigm motivate an original perspective on musical performance?
-This paper seeks to outline methods underlying the development of the Xth Sense project, an ongoing research which investigates exploratory applications of biophysical sound design for musical performance and responsive milieux. Presently the Xth Sense technology consists of a low-cost, biosensing wearable hardware device and a [[Pure Data]] based application for capture, analysis and real time processing of human muscle sounds.
-Firstly, an exploratory study of the acoustics of body sounds, namely muscle sounds is illustrated. I describe the development of an audio synthesis model for muscle sounds which offered a deeper understanding of the body sound matter and provided the ground for further experimentations in signal processing and composition in Pd. Then follows a description of the development and design of the Xth Sense - a wearable hardware sensor device for capturing biological body sounds. Eventually I describe the development of the DSP modular application, focusing on its main features such as: a MMG features extraction methods, a GOP routing system for dynamic mapping of gesture to sound, real time audio processing models and GUI utilities, which were developed to provide a fast prototyping environment.
+[2] M. Donnarumma: “Xth Sense: researching muscle sounds for an experimental paradigm of musical performance” Proceedings of the Linux Audio Conference (LAC'11), 2011.
-In conclusion, I elaborate on the implementation of the Xth Sense framework in the composition of Music for Flesh II, a solo musical performance; which realization process prompted a better understanding of muscle sounds capabilities for biophysical generation and control of music, thus providing further prospects for the inquiry.
+[http://res.marcodonnarumma.com]