Design Challenge 8

Interaction with Virtual Musical Agents :
posted by Fredrik Olofsson and Mark d'Inverno

Summary

We are interested in building a music installation based on agent technology.  Users will be able to manipulate agents in a 2-dimensional ALife world to produce different performances. Essentially this will be done using the outline of the shadows that users can cast on a 2D display.

Visitors (or users) will stand in front of a projection screen. They are captured and tracked by a webcamera. On the screen they will see a rough outline of themselves plus representations (as dots) of musical agents. The outlines of the visitors - limbs and torsos - define boundaries for worlds with different musical functions, and a different environment with different properties,  within the main 2D virtual world. The set of environments defines a  conventional artificial life world that makes up the whole screen. There is also a sound
system so that the visitors can hear the sounds the agents produce.

The musical agents live and evolve in - or outside of these outlined worlds given by the shadows cast by the limbs and torso of. By moving around and take different positions the visitor can, in real-time, interact with the agents. Agents can be caught, caressed and nursed inside of an outlined world and will behave differently and make different sounds depending on which world they live in. A world could be thought of as a musical part or function e.g. in a pattern world agents make short rhythmical sounds and in an ambient world the sounds produced are long and soothing. The agents know how to move, mate, eat food, make sound etc and their genome is made up of learned behaviours and a set of musical phrases and sounds. The sounds they make gets more complex as the agent grows and learn new behaviours. The musical phrases adapts to the environment i.e. which outlined world the agent is currently placed in.

The outlined worlds provide food for the agents, mating possibilities and relative safety. But to really gain advantages, agents will have to migrate to other worlds. There they will be unique sounding and thereby get more energy, more food and mate more frequently. The visitor helps agents to migrate by making a connection to other outlined worlds nearby the chosen agents. So two visitors holding hands will provide agents with a bridge connecting two worlds. Ideally the visitors selects a few pleasantly sounding agents and make them migrate, transferring back and forth between worlds by interacting with other visitors and get their favourite agents and their offsprings to visit and learn from multiple worlds. After travelling through worlds, picking up different elements of music and finding lots of food, the visitor will have breed his/her own family of unique sounding agents.

Implementing the above system we will use and extend the AWorld agent framework we recently developed. This framework is written in SuperCollider (Smalltalk variant). The sound will be synthesised in real-time also using SuperCollider, as well as system state monitoring, saving and logging. For webcamera analysis, blobtracking and visuals we will use Processing (Java variant). Communication between the different parts of the system will use the network protocol OpenSound Control (OSC). For developing rules, tuning the system and other experiments, we would like to try to code rules and behaviours on-the-fly. Our framework has features for exchanging the act&sense methods for all or some agents while the simulation is running.