Design Challenge 4

Challenge Details

This design challenge is to explore the possibilities for real-time interaction with
artificial life worlds. Conventional artificial life experiments are usually conducted
within highly specific metaphorical confines. These confines result from the
assumptions necessary to develop and test models that can be verified and validated. 

As an example, in Cellular Automata style simulations, the conceptual ‘universe’ is reduced to a finite, two-dimensional lattice, and the physics of this universe to simple discrete, local, state-based interactions. In the case of ‘sugarscape’ models, simple, discrete, virtual agents roam similar two-dimensional lattices searching for food, avoiding obstacles, mating and interacting, etc. This is the conventional artificial life world (CALW) – necessarily separate and apart from the real world – the mainstay for most of the Alife research community. Interaction, if it exists, is usually limited to direct model parameter control. Real-world interaction rarely considered as being part of the actual model itself.
 
Artistic works, on the other hand, have often sought interaction between the real and synthesised worlds. In my own artwork, Eden, for example, a symbiotic relationship develops between people experiencing the artificial life world, and the creatures that inhabit it. The presence and movement of people within the installation space create conditions to favour the growth of resources within the virtual space. These resources
are needed by the virtual agents in order to survive. Creatures can make and hear
sound, and this process is sonified as part of the work, meaning people within the
installation space can hear musical sound generated by the virtual creatures. Since
‘interesting’ sounds are normally likely to keep people interested in the work, the
longer people stay, the more resources are generated for the creatures. Over time, the
creatures learn that by making ‘interesting’ sounds, they improve their chances of
survival. The system works due to evolution. It is not that the creatures have any
explicit knowledge of what makes an interesting sound, or that there are people in the
installation space. It is only that the creatures that make sound have a greater chance
of survival, because food usually becomes more abundant. Survival also means time
to pass their genes (hence their knowledge – Eden uses Lamarckian evolution) to their
offspring. Thus, over time, the population adapts to its dynamic environment. 
 
     

Eden layout from above (left) and side (right), showing screens and
sensors. Eden uses eight infrared (IR) proximity sensors, placed close to the projection screens. The sensors measure the distance of the closest object. Software interprets the incoming data to provide presence and motion data over eight separate areas of the Eden world. While this configuration works reasonably well, there are some difficulties:

• Processing the IR data can be difficult and time consuming. For example, the
sensors have a large amount of noise, particularly with long cable runs so both
hardware and software noise filtering is applied.

• Motion detection is inferred in a single axis only: lateral motion is difficult to
infer.

• The sensor range is limited to approximately 1.2m, making it impossible to
detect the presence of people further away than this distance. People often
congregate around the outer periphery of the work, out of sensor range.

• The sensor detection area is a reasonably narrow, forward facing cone;
meaning areas of the installation space may be undetectable by the sensors.

• Sensor information in total is more global than local.
 
I would like to propose a system, somewhat similar in intent to Eden, but different in
the mode of interaction and function. The goal of the challenge is one of developing
new reactive relationships between people and autonomous systems. Eventually, this
might lead to new forms of human-machine interaction with a variety of applications. 

Layout

First, the basic physical layout is described (see Figure 2). A series of translucent
screens form a long continuous plane within the installation space. Projectors display
images generated via real-time simulation. As the screens are translucent the images
displayed are visible from either side of the screen and it is also possible to see
through the screen (hence the images projected appear to float in space).
 
A series of inexpensive video cameras are mounted at the top of each screen pointing
out into the space. Two cameras are required for each screen, pointing in opposite
directions, hence giving coverage on both sides of the screen. This is shown below.

Vision-based Interaction
 
In addition to detecting visible light, most modern digital video cameras are capable
of detecting light from the IR component of the electromagnetic spectrum (‘night
mode’). Using this feature enables interactive detection at low light levels, something
common in constructed environments and installation spaces. Most ‘web cams’ have
sufficient resolution and direct computer connection, so are well suited to this
application.
 
As discussed in the previous section, a series of these cameras track and project  images of people as ‘phantom’ mirror reflections on a translucent screen. The  translucent screen freely mixes projected, reflected and transmitted light, merging the
spatial boundaries between real and virtual spaces.
 
The system digitises images of people on either side of the screen and uses this
information as input to a cellular Alife world. Basic image processing methods allow
the system to distinguish between static objects and people moving about the space.
Further processing may allow the detection of movement or regions of interest (e.g.
region coding of specific body parts such as arms, legs, head, etc.).
 
A possible extension may be to use stereoscopic projection, placing the projected
cellular images in three-dimensional space, rather than two. 

Agent Learning & Interaction

Agents in the virtual world should be able to sense and react to the flow of image information fed into their environment. This raises some open possibilities regarding how, what and why.

The design issues can be divided into two broad categories:

i. Aesthetic: how can we design the system to explore the aesthetics of
interaction? The system already has some pre-set visual and physical aesthetic
constraints (use of translucent screens; image, figure and camera placement,
and so on). What effect will the aesthetics have on the way people approach
and react to the system?

ii. Procedural: How does the presence and movement of people with the space
effect the virtual environment? In Eden people’s interest in the sound the
creatures were making influenced the food resources for the creatures. This
lead to creatures evolving interesting sound-generation capabilities implicitly.
In this case there are other possibilities, but we’d still like ‘interesting’
behaviour to emerge through the interaction.
 
These design issues lead to questions regarding the properties and construction of
agents and their environment. For example:

• What agent sensors are appropriate?

• What agent actions are appropriate?

• What is the ‘physics’ of the virtual universe and how does this relate to
the real space?

• How does camera data translate into this virtual physics?

• What learning and evolutionary schemes are appropriate for real-time
learning and evolution in such a system?

• What kind of agent behaviours and human interactions should we expect to judge the above choices successful?
 
In the figure below, the digitised reflection appears to be behind the screen 

 

In the image below we see a corresponding cellular image.