This is the text of a talk I gave at Electrofringe some years ago. Since then many of the links may have gone stale, but the message is just as valid.
Fiction that takes place in a 3d virtual world is a really cool concept. The audience doesn't just sit out the front and passively watch, but can wander around and through the action to view it from any angle.
Some people are already exploring it right now:
• Some 3d games use a story to provide some of the motivation for the player. Unfortunately this is usually only in the form of a 3d rendered prologue.
• Many Manga cartoons already use 3d worlds and characters instead of painstakingly drawn cartoon frames. This is currently rendered as pretend cartoons where the 3d rendering engine -- typically Lightwave3d -- gives the objects outlines and flattens the shading, but it would not be a giant step to make them into 3d worlds that could be explored while a story takes place inside them. (Incidentally, if there are any comics artists here this may be a useful way of making comics -- I don't know if anybody already does this.)
• VRML Dream was a live performance of Shakespeare's A Midsummers Night Dream over the net. The audience viewed it as invisible ghosts. We could wander freely, take on the viewpoint of one of the actors, or use the director's "camera" view.
• Alan Taylor http://www.wolfnet.com/~yoame has put some Raven legends up on the net which are done in 3d. They are not interactive but they are the beginning of work in this direction.
• Paul Hoffman http://www.cognetics.com has done some very cool work. He was largely responsible for the IrishSpace project which was an enormous 3d story that you could explore and interact with. He now has made another 3d world and put it up on the net that is more a kind of puzzle than a story. I hoped to be able to show you that but I stupidly left the address at home.
• Richard Kapuaala http://www.kahunanui.com has written some neat 3d stories. One is a detective story of sorts, where you have to solve the crime.
• Protozoa http://www.protozoa.com
have made a whole lot of 3d cartoons. My favorite are the Dilbert cartoons.
They are straightforward stories but allow you to use different viewpoints
and move around the action.
There are several advantages in writing fiction for Computer-based VR:
...OK, just a brief diversion for a moment on what VR is now, and what it will become
At the moment most computer VR takes the form of a window into the other world on the screen of your monitor. This can be surprisingly seductive. I have memories of walking around in places that I have only experienced thru a computer monitor, but that is not really surprising -- you have all watched a film and been sucked into the picture. Don't picture VR as people using strange helmets and gloves; more as a window on a world.
In the very near future, devices which paint the image directly onto your retina will be generally available (they have been under development for some time). That may be the last of the external computer devices for VR, unless there is more development on CAVEs, which are Holodeck-like devices. Some time in the next 10 or 20 years I expect we will interface our brains directly with computers. You will lie down and activate mental switches that link the computer outside your skull directly with your brain there is no need for a physical link -- it will be done without sticking wires into you. The brain will switch into sleep mode where your senses are cut off (as you do every night), and will interact with a generated experience inside your mind.
... but at the moment we make do with an image on a screen.
....OK, back to the main topic of VR fiction
There are a few different kinds of VR fiction.
From the audience's perspective it breaks down to 2 main areas:
Both these kinds can be either:
So VR fiction actually is not just one thing, but a spectrum of possible forms. At one end is the film-like way of doing things, where there is a single set story-line and the viewer is able to either wander around in the actions like a ghost or has their viewpoint moved to where the director wants. At the other extreme the writer simply sets up the world and lets it run after populating it with artificially intelligent bots and artificially alive creatures that the viewer interacts with -- the world changing in ways that nobody can predict... even changing while you are away from it.
Writing a branching story is quite difficult -- has anybody here ever read one of the "Choose Your Own Adventure" stories? You have to write much more material than you expect will be used.
One question that was raised at vrml-lit (a mailing list on VR fiction
I am on) was what point is there in writing a story where the author gives
up control to the viewer over the ending of the story?
Three answers resulted:
A friend described to me a linear, but interactive play she saw. Here is what she told me:
a long time ago in New York I went to a theater production called, Tony and Tina's wedding You, the audience, are part of the drama at this wedding. Wedding members sit at your reception table, dance with you, give you the dirt on the maid of honor and the best man, etc, the "drunken priest" asked me for a cigarette while one was already in his hand, that kind of thing... i went twice to inter-ACT with different characters. Each time I had a different experience, yet within the context of the WHOLE, the play was exactly the same. The actors played the same lines but with different folks. For ME, however, the play was different each time because of my different experiences with the cast.
Making even a simple, linear story in VR is harder than would at first appear:
In film and written works you can easily compress time...
For example in a written story you might say that a character picked up the keys, exited the house and drove to their friend's place
In film you might show a close-up of the keys as the character's hand
picks them up...
then a quick cut of the character exiting their front door
...a dollying shot of them running down the front step to the car
...a quick closeup of the keys being put into the ignition and being turned
...a shot from inside the car behind the driver showing the view out the front window as the car swings out into traffic
...a couple of shots of the car whizzing along roads
...finally the car pulling into the kerb toward the camera at the friend's place.
A journey that may have taken 20 minutes is reduced to a few seconds
Doing something like this in VR poses a number of problems. You can't have the story character simply leave the house, because the viewer may have wandered into another room to look around. And you can't simply cut from one scene to another because you don't know what the viewer was looking at at the time. This requires a whole new visual language, and we need to work out new ways to guide the audience's attention.
[Here I did a double-take, appearing to have seen something puzzling off to the side, & looking quizzically at it -- then I checked to see how many audience members looked to see what I was looking at. Unfortunately I stuffed up a bit and it didn't work well. Bummer. It worked well when I did it more smoothly at home.] You can solve some of the problems of being unable to do cuts in VR fiction by using tricks based on aspects of human psychology (such as empathy) to redirect the users' attention the way a cut or pan is used in film. Stage magicians do this kind of thing all the time.
A sudden noise will direct the viewer's attention... you can use a phone to get the viewer's attention and the story character could leave the room after the phone call.
In a darkened area, a light will draw attention, particularly if it moves, so you can use that for catching the audience's attention.
But cuts and fades are still a problem, because I can't see many easy ways of relating them to what the viewer is seeing at the time.
One possibility is that a ghostly viewer could have limited freedom. They could wander freely most of the time but at certain points in the story their viewpoint is forced to a certain position so that they must see and hear a pivotal part of the plot. This may be the most usable method of changing scenes or introducing dramatic story points.
The story character picks up her purse and unnoticed by her, the all-important note is pushed off the bench and falls behind the refrigerator... but the viewer sees because their point of view was pulled over to behind the fridge to see the note wafting down... then after that they are released and allowed to direct their own viewpoint once again.
This would allow most cinematic devices by constraining the viewer only at certain times, however it could end up feeling intrusive and contrived unless very cunningly managed.
Another way of making sure your viewer sees the right things is by triggering events by some particular action. For example: they could answer a ringing phone, and when the phone conversation is complete the scene could fade out to another. Or by walking into a particular spot they trigger a particular sequence to occur. This would mean that you would have to abandon telling the story in a time-constrained fashion though.
The earlier film example of compressing a long car journey into 20 seconds
could perhaps be done in VR like this:
The story character can get a phone call which they don't answer immediately. Letting it ring for a while gives time for the viewer to become curious and come back to the room if they have wandered elsewhere. Then when she answers the phone she can be angry and yell at the person at the other end. This ensures that we have the viewers' attention. Being dramatic like this won't be necessary in the future when we have an understood set of signals that viewers will easily understand, but at the moment the art is in its infancy and we will probably need some degree of overkill to get our point across. Remember though, that loud is not always the best to way to get attention.[Some time back I was in a room crowded with people all talking, and two people started to whisper. Everybody in the room stopped to look and listen. The two ceased and looked self-consciously around the room. Everybody realised what had happened and broke into laughter.] Anyway back to the example... the character now asks something like "What? It is there now?! OK, this I gotta see! I am coming right now!". She hangs up and exits the door, runs down the stairs to the car looks in her pocket for the keys (giving the viewer time to catch up), unlocks it and gets in. Now she delays by putting on her seatbelt, finding the car key, perhaps looking at the street directory or finding the address written on a piece of paper, or waiting for a break in the traffic. This gives time for the viewer to get in the car. After a while the car pulls out into the traffic and drives off. Some way down the road the car car fades out and fades in near the destination. To the viewer inside the car this would be accepted as a cinematic trick to compress time because not only the vision but sound would have faded. The car would now pull in to the curb and the driver would switch off the engine and exit the vehicle. The viewer would follow, curious to find out the reason for the journey.
A lot of what we see in film and TV is actually a kind of shorthand. In the early days of films none of the current range of cinematic tricks and tools were used -- the language hadn't been developed. However, gradually some experimenters and risk-takers developed some useful tools and effects, those effects were named, and people were able to pass on the knowledge. We need to do this with VR fiction to develop a whole new language, building on those used in film, cartoons, comics, and books. It is going to be an interesting journey. You have the chance to do something absolutely pivotal in a totally new artform.
VR fiction is at a very early stage. We are just beginning to learn how it may be done. I am convinced it will eventually displace film as the biggest mega-buck earner on the planet, but a small number of pioneering souls will need to explore the territory first.
Will some of you be the names that future kids will learn about in school?
Will you be the legendary writers of this new form of entertainment?
I am willing to bet on it.