Zen and the Art of VJ'ing
A paper presented by,
Adem Jaffers
www.amjaffers.com
amjaffers@gmail.com
               for    
F r e e  P l a y  2 0 0 5
Independent game developers conference
presented by,
Next  Wave Ferstival & ACMI       
15th July 2005 (3:30-4:30pm)

2005(C)copyright control

ZEN AND THE ART OF VJ'ING

W E L C O M E :

GREETINGS  -
Welcome ladies, gentlemen and gaming lovers alike. 

My name is Adem Jaffers, and today I'll be giving you a presentation titled 'Zen and the Art of VJ'ing', a discussion on techniques used in custom mapping for VJ performance purposes.

OVERVIEW -
Now, before I plunge into this. I'd like inform you that  the format for this talk will begin firstly by navigating you briefly through my past involvement with VJ'ing, or more broadly, real time performance.

Providing you first with some historical background that may give you a sense of where I'm aesthetically and conceptually coming from in regards to the development of the map VJ-iSOtope144; which I will be presenting to you a bit later on. 

I'll then move the talk into the present state of iSOtope; and on into the future possibilities I have in mind! Ending by opening up the floor for discussion.

P A S T :

EYE SPY -
In the late 80's I was working in Melbourne as a Photographic Survealance Investigator. I was armed with silicon enhanced night viewing gear, bugs, 35ml SLR, full swag of lenses and a B&W darkroom at my disposal. 

I cant help but admit at this stage that I wanted to be 007 as a kid :)

This is where I first learnt about using lenses. I began utilising the many boring hours involved in stakeouts by photographing scenes within eye shot of my vehicle.

Being able to zoom in and out on targets quickly using long lenses became somewhat of a skill I had acquired.

PARTY TIME -
At the tale end of the 80's I was introduced to the Amiga 500 computer. With its impressive graphic/audio capabilities and DPaint animation software; I soon put down the camera and began my long career of sitting in a different dark room, playing flight sims and animating with 32 color palette cycling.

The electronic music scene was well established in underground circles, and a ground swell was occurring in Melbourne that was becoming an exciting cyberdelic sub-cultural wave of new ideas and possibilities. Imparticularly in the arena of video art. 

This era was captured by the documentary Cyberdelia, which is currently online at ACMI's Memory Grid installation.

In the very early 90's I began to create video tapes for the few parties that could afford the expensive cost of hiring video projectors. 

Then moving into operating Intelligent lighting and soon after that, animated color Laser projection systems.

With some events it was a privaledge to be able to incorporate the 3 mediums; Video, Lighting and Laser together into totem like displays. 

The idea of totally submersing people in an environment of intense lighting conditions was extremely appealing; as was the real time nature of pulsing lights to music in real time.

CYBERTHON -
Overlapping this early period of luminous decadence was the birth of the Cyberthon events. Involving VJ's, DJ's, bands, performers and artist alike. These marathon collaborative jams lasted anywhere upto 15hours plus.

Over a period of 5 years, this eclectic group of artists would converge on an empty warehouse or TV studio with their audio and visual technologies to produce a series of yearly Computer-Art based television broadcasts, during Channel-31's test transmissions.

These events achieved some extremely interesting outcomes on many levels. One event imparticular called 'Cyberthon-IV:EPIC Omnicast', held in 1994 as part of the Experimenta festival, saw the inclusion of a BBS for viewers to upload images for inclusion in the show. 

EPIC also included a bi-directional simulcast on RRRs Radio show called Tranzmission, where audio transmitted from Channel-31 was simulcasted on RRR. 

Then in reverse, the audio transmitted by DJ's at RRR was simulcasted on Channel-31, where 5 VJ's using as many computers mixed up visuals to the music.

The audio and visual signal was also routed onto RMIT's MBone for webcasting to Universities around the world. 

As far as I'm aware to date, this was the worlds first webcasted live VJ event.

The final event in 1995, an 8x1hr tv-series called Cyberthon-V:Spectrum, was an  experiment with interactive TV. A gutted out arcade game top containing an Amiga-1200 was situated at the Lounge cafe in Melbourne. 

It was connected directly to a genlocked Amiga-3000 at the RMITV studio via modem. Allowing punters to script together a sequence of animations, then post the script to RMITV for cued playback over the airwaves. 

SELLING OUT -
In 1996 I began a career in compositing & animation in a boutique production company called Unreal-Pictures. 

At the time we were using Beta versions of 3dsMax and Digital-Fusion; and in the pursuing years accessing funky tools such as facial and motion capture systems, and Scene tracking software.

Creating animation and visual-effects for TV series, film, adverts, docos, game and web based projects. 

POD -
In the late 90's, inspired by the Cyberthons and Dance parties, came the development of the PODS. 

Two old hospital linen trolleys converted into computer racks. Containing an Amiga cd32, 500, 1200 & 3000 in each unit. 

One Amiga had a DMX card enabling control of various lighting fixtures. 

Both PODs interconnected with parallel and serial IP for networking. And Genlocked in-line for layering of animation over a VHS or camera video source. And finally Midi for sequencing of audio, animation and lights.

Containing over 200 patch points, the POD systems were an attempt to create a mobile and contained VJ performance system that enabled greater real time and scripting control.

REALITY ENGINE -
At the turn of the millennium, came the development of the Reality Engine. A video feedback instrument created from a lego robotics kit. 

A color pinhole camera was pointed directly into a 4inch color LCD panel, obscured partially by a spinning paper disk with holes cutout, mounted onto motorised lego cradles.

A small LOGO program written in the Dacta software, enabled the precise Z-movement and infinite XY-spin of the camera cradle; and hence scriptable repeatability of video feedback. 

The Reality Engine worked well in conjunction with the animation layers delivered by the PODs.

UNREAL TOURNAMENT -
In 2001 I came across the nouns EPIC and Unreal once more, this time in a different guise, and a very exciting one at that.

I was living with a hardcore sys-admin who'd downloaded Unreal Tournament. It was the first time I'd 'really' played a computer game in over 10 years.

Soon I began playing online under the nik dOs on Jetstream & AusUt's CTF servers. Competing against rogue clan members dawning the occasional wall hack; And every now and then an aim bot who was boosting their UT-stats. 

Instagib, a weapons mod that became an official fixture of the UT arsenal, was the weapon of choice. As the name implies, it is a instant kill pulse weapon that spawned its own sub-culture. Giving rise to its own rogue forum, server and league.

A group of 6 of us separated by thousands of kilometers then formed a clan called SSC - or Sumo Space Chimps; and began participating in the online clan wars.

Then came the launch of Internodes bookable clan server, which included mainly custom maps with several popular stock ut maps. 

Playing levels that resembled, ice-skating rinks and bathrooms complete with dunny and bum holes for spawning players. 

This period allowed me to see beyond the classic space-station, slum city, jungle, or military base levels that seem to be the norm in the official released pool of content. 

SSC and SKC clan members started to organise small IG mod based LAN parties hosting custom maps and UT favorites. 

It was great way to atleast meet each other physically and see the faces of people we were killing online.

So my introduction to the wild and wacky world of modding was now complete. 

AUTOTAUNT -
As a way of exploring the use of game maps to create a machinama video. I first created a music track that incorporated most of the UT in game sounds. 

Then along with the SSC clan members we logged onto the map Lava-Giant hosted on a home-based game server. We acted out the killing scenes based on a loose script. One of the clan members had a handle on spawning weapons & monsters from Unreal Gold.

The entire performance was server side demo-recced. Then processed out to avi's using movieUnreal, which enabled control of frame rate outputs, creating smooth bullet-time like sequences at a scalable resolution and frame rate. 

It was a succesfull experiment in creating a video online, syncronising the efforts of many players remotely. 

The video,  called Autotaunt, can be download from Fileplanet.

UNREALED - 
After the excitement of making Autotaunt, and being engrossed in clan wars on the nodes custom map server, the time to investigate the workings of the Unreal Map Editor had come. 

I was excited by UnrealEds method of Constructive Solid Geometry modeling, its dynamic lighting and texturing capabilities. 

After some weeks of toying around with sky boxes, zones, spawning and lighting I felt ready to tackle a simple, but playable map.

CTF-ISOTOPE144 -
Enter CTF-iSOtope144! A custom map initially designed and created in 2002 for IG Capture the Flag specific clannies. The map was picked up by Internode and XGE-Sports game servers for rotation, with mixed results. 

Comments such as 'what drug were you on when you made it dOs', or 'thats the trippiest map I ever saw, takes me way back to the 70's man', 

These comments, and others inspired me to keep on developing, and inturn post iSOtope onto Nalicity and Fileplanet. 

Even though iSOtope was initially an IG specific map, I'd always maintained the idea of developing it further for VJ purposes. 
Which is just what came next!

P R E S E N T :

VJ-ISOTOPE144 -
So now allow me to present to you VJ-iSOtope144, the visual performance version as it stands today. I'd like to take you separately through the various features incorporated in the map.

Then hopefully finish off by showing you how it all works together in a real time environment.

MODEL, TEXTURE, LIGHT -
The tool set available in UnrealED is quite sophisticated, even after coming from the advance modeling, animation, lighting and texturing of 3dsMax.

Building iSOtope was a newish process for me. Using 256 color palettes and restricted pixel, geometry and grid unit dimensions to multiples of 2; for optimal results.

Knowing that its possible to do most of the modeling and texturing in Max and then just import, I chose to use UnrealEds texture and modeling tools exclusively. Mainly so that I could use iSOtope as an exercise to get under the hood of the editor. 

Theres something nostalgically appealing in a minimalist sense  in  about working with primitives.

iSOtope is constructed with Cubes, Cylinders, Cones, Pyramids, Gengons and Spheres. Likewise, the textures are made up mostly of lines, circles, gradients and squares. Using solid color schemes of RGB, Rainbow spectrum, B&W and Gray Scale.

Only one example of animated texture maps appears in the level. Mainly because at the time of developing iSOtope I was using a 32meg video card, so screen lag became a major problem. I now have a 128meg video card!

A solution to the animation rule came in the form of Unreals procedural texturing. Which are animated algorithmically so they have infinite, non-repeating animation.  They also have the advantage of taking up very little disk space (on the order of 4K) and they only occupy as much memory as a single, non-animating texture of the same size.

Using procedural textures enabled me to include some more organic movement to iSOtope. UV-panning of textures was also explored as it gave seemingly static texture a little more life. As did adding coronas.

One of the simplest but most interesting features is a Black and White 1x2 pixel texture that is used frequently throughout iSOtope. 

When tiled, as done in the 2Pixel-room, it can deliver stunning opto-kinetics. A mind-fuck is one way to describe the effect.

Another powerful method to achieve more organic movement in the map came about quite by accident, in the form of 'The Dreaded "Hall Of Mirror" Effect' - or technically known as BSP holes.

Binary Space Partition holes are propagated geometry errors in a map. They allow the Black Space Grid, or the Void as I call it, to appear in the map during game play. As an industry standard they are ugly, to be avoided at all costs, and in some instances, a mystery as to why they appear.

But for me, the edge dweller, BSP's were a godsend. My first encounter allowed me see how the area revealing the Void would not be redrawn, and hence leave the previous frames of texture information behind in a motion like trail.

To simply put it another way - it was feedback. So based on that knowledge I purposefully applied transparent textures to the surfaces that separated the void from the inner game playing area. The results for me were incredibly fluid.

Lighting was the last bastion of exploration. Even though memory constraints are a major headache here, I still managed to use a fair bit of strobing, flickering and pulsing lighting types, with interference pattern effects applied. 

Most other surfaces use flat lighting or Unlit texture properties for  memory efficiency.

CUSTOM BINDS -
Now we come to key-binds, the ability to assign UnrealScript console commands directly to the keyboard. This is an exciting area that gave iSOtope the feel of a more complex performance instrument. 

Used in combinations, binds open up a rich palette of performance possibilities.

CAMERA -
The first bind being 'FOV'. With this command I was able to assign keys to vary the cameras field of view angle. So that in real time, a VJ can switch from an extremely long to wide fish eye lens quickly, bouncing the camera view to the rhythm of the music, creating at times an undulating and pulsating like effect.

ROTATE & MOVE - 
The 'Axis Turn' command enabled the bot, and inturn the camera view to rotate around in either left or right directions at speeds assigned to the bind. This provides a great effect as the rotation changes from gentle ambient spinning to a mind numbing gyration. The osscilating effect can be quite hypnotic.

Other standard binds were move 'forward, backward, walk, strafe, crouch and Jump'. Which basically enabled the obvious left, right, forward, backward, up & down movement of the camera view. These binds really give added finessing control in combination with other binds.

Several exciting binds that dramatically added functionality to iSOtope were 'Fly, Ghost and God-mode'. The Fly bind assigned allows the VJ to do just that - Fly! 

'Ghost' is just like Fly except that it allows the VJ to pass through walls, into another room or out into the Void. More on this later.

With out the bind 'God-mode', allot of iSOtopes functionality would not be possible. It renders the bot un-killable. That means as you pass through walls, lava or underwater (as ghost) the bot wont die. 

This means that the visual effects associated with passing through these zones (such as ripples, fire etc) can be included in the camera output for extended periods of time without having to re-spawn.

A very very cool bind is 'Slomo'. Changing the values in this command changes the overall clock speed of all the actors in the map. 

So weapon fire, texture animation speeds, other bots and just about all timings of everything in the map, can be varied at the touch of a key. Creating slomo bullet-time like effects or at the other extreme keystone comic like fast paced motion. 

This is a very powerful command when used in combination with 'Axis Turn' and 'FOV', as it effects all the commands speeds relative to the 'Slomo' clock speed set, thus expanding the variable ranges of all the binds.

Speaking of bullet time, the 'playersonly-pause' command allows the VJ to pause all action and actors in the map, except there own bot. 

Giving the VJ the ability to fly around objects frozen in space, which can be released to the bounds of time, at will.

The 'Allammo' command will become apparent in a couple minutes, but for now know that it allows replenishment of weapons ammo without having to run around and pick up ammo-packs. 

I mean, who has time to look for ammo-packs when there is art to be made.

WEPS AS ART -
I have constructed a zone called the Loading-room. A single space that has all UT's weapons lined up in a row for quick pickup. 

Once weps are acquired the VJ can move into another room constructed specifically for the performance use of weps. The Black-room! Which is simply that, a matte-black unlit room which allows clear visibility of weapon fire.

The object here is to use weapons fire in conjunction with the various binds to create looks such as; frozen glow trails, kalaidascopic sprays of green globules or bullet-time like explosions. 

Using the sniper rifles zoom capabilities also achieves added crash-zoom effects. The redeemers missile cam also allows dramatic camera action.

In other paler textured areas of iSOtope, walls can be used to imprint ammo impact decals. The decals dissapear after a short period of time. Used in particular ways decals can create a primitive mono-tone graffiti like effect.

CUTTING MECHANISMS
The equivalent to cutting between video sources in iSOtope can be achieved in a number of ways.

Making use of well placed Teleporters allows the VJ to pass from one room to another instantly, regardless of the distance between rooms. And with some nice little materialization effects to boot.

Another similar method is to use the Translocator weapon, which allows a VJ to place a beacon in a zone that is to be entered at a specific time later on. Or inturn it can be used rapidly to achieve faster cuts.

A series of rooms built like high rise commission buildings, but oriented laterally is The Grid; which includes the 2Pixel, Epilepsy and Loading rooms among others.

Quite simply, whilst in Ghost mode the VJ can fly straight through these rooms both vertically and horizontally. The walls themselves become the cutting mechanism.

When more than one player (or bot) is in the space, the VJ can use the 'Viewplayer' bind to switch between different player camera views. Thus creating another cutting mechanism. One player can be in the Ambient-room, whilst another is in the Epilepsy-room, then cut at an appropriate moment in music time.

P H U T U R E :
So this takes us to the end of the current manifestation of iSOtope, and takes us into an area I like to be in; the future. A place where my imagination has free license. 

With UT2004 already nearly a couple years old and UT2007 already announced and hiring. Its time to think about some of the things that can be achieved with the next version of iSOtope.

Whilst I do that, on screen will appear iSOtope in action.

VJ CLANS -
A natural progression for iSOtope would be to have it loaded onto a dedicated VJ-server. Where VJ clans can log in and jam together. 

I imagine a space where a group of VJ's perform together in a synchronized way. Possibly like aerial acrobatics or synchronized swimming.

SPECTATE - 
UT2004 now incorporates UnrealTV, the ability to stream game play to a thousand spectators; Which means in the case of iSOtope, a VJ-clan performance can be seen by quite a few viewers. 

Users on Cable or Dial-up can be delivered entertainment in a scalable resolution, in high or low detail. Best of all, recorded at the client or server end for later playback and manipulation.

TEXTURE -
Another function that can be implemented is the ability to change all the textures assigned to surfaces locally or globally. Enabling different themes to be called up as the music changes from one style to another.

BOTS -
Currently bots look pretty much bipedal, be it humanoid, alien or robotic. 

I'd like to see the next iSOtope manifestation have a collection of bots that look more orb like; Shapes that fit in with the look of the map.

UT is a 1st person shooter, but with the 'botview' command it can be switched to 3rd person easily. Which means you would see your own bot as part of your own visual experience. 

Other players or spectators would also see you as a weird orb like device emitting light pulses to the music.

LIGHT - 
Just like a lighting desk, addition of a series of binds that enable all facets of the lighting, locally and globally to be changed and sequenced, would also add a whole new dimension to iSOtope. 

WEPS -
To take the Weapons as art idea further. The next version would see weapons that look less like weapons and more like abstract organic shapes, that deliver a variety of interchangeable pulses and lighting effects. 

Also altering the kind and color of decals they leave behind on walls. Expanding the graffiti dimension.

AUDIO -
Weapons also generate sounds when fired. As does ammunition when in flight or when a target is hit. These weapons could be modular, enabling VJ's to easily associate various looped sounds that are triggered as weapons are fired to the rhythm of the music.

Music can also be delivered as a compressed stream. As UT2004 has now incorporated support for Ogg Vorbis. Which means that now DJ's can participate in the performances; Providing music for the VJ's to work to, and for the audience to enjoy.

Sounds can be assigned to almost any Actor or Zone. Which means that as a VJ moves through a map, sounds can be triggered by the default action of their movement.

Finally, UT allows binoral VoIP  comms and Text to Speech'. Which means that it can be used by VJ and DJ's to sync together. The comms can also be used to deliver extra layers of sounds to compliment the Streaming, Zone, Actor and Weapons audio.

T H E  E N D:
So I think I'll stop here, as the ideas for this kind of VJ environment are so open ended that I could talk about it for hours; And I haven't even begun to talk about turning it into an actual game, with scoring and progressive levels and so forth.

Therefore I will take this opportunity to thank you - the audience - for being here. 

To Alex for her support and for letting me jump onboard this gig so late in the day. 

And Michelle for her tech support.

To the Next Wave, ACMI crew and sponsors for putting on this uber cool event. 

And finally to the FedSQ posse for this triangulated space to hold it.

Also, a transcript of this talk is available from my web site.

I will now leave you with this sample  taken from a DVD that I'm about to release with the Psy-harmonics record label.

The music is by Shaolin Wooden Men. The actual DVD is 40 minutes long, and was performed in one take, no edits whatsoever.

So please enjoy, and thank you!

I will now open the floor for any questions or comments.