This is a modified form of an original document written in 2001 (I cleaned it up for the publication and explained a few ideas better), it was the result of standing around waiting for a train in Utrecht in The Netherlands and wondering "How would you build The Matrix"?...
One day I began wondering can we make "The Matrix"? What would such a system be capable of? what could we do with it? and most importantly how would we build such a system?
The ADEX - Alternative Digital Environmental eXperience
Ever seen The Matrix, The Lawnmower Man? or Star Treks Holodeck? These are show examples of people interacting with computers in ways we simply can't do right now. The Operating Systems, Interfaces and other technologies involved simply don't exist - or do they?
The following document describes a proposal for the ADEX. Firstly a description of what it would do and then how it would do it and the parts which make up the system.
What is the ADEX
The ADEX (pronounced "A DEX") would be an immersive environment. Instead of sitting in front of a PC staring at a display it would ideally use some form of Virtual Reality type display so you would find yourself inside a 3 dimensional generated world. You would also wear parts or all of a suit which would produce tactile feedback - when your virtual hand touches something you would feel it.
The ADEX would use these and other techniques to represent a "world". You and others would be able to interact with each other and with objects within the world. A group of friends could play virtual football using a virtual ball in a world.
There would be multiple worlds in an ADEX system, these worlds could be anything you want, games art, entertainment, information, education and of course the big seller would be sex.
A world could be public or private or could be accessed only after paying to enter. You could of course create your own world in much the same way as you create your own web page.
ADEX would not represent the computer (or computers) it is running on. The user sees the worlds only. There should however be some way of uploading and downloading data and items to your local PC so a set of drawers could appear for you put things into and take things out of. There would also be some form of virtual wallet for you to pay for things.
You could record and playback actions within a world.
You would not be able to tell the real physical characteristics of a person you encounter in a world. You could change size, height, colour and any aspect of your appearance - the only possible give-away would be your voice - if it was being transmitted.
There would be advantages and disadvantages to this however and it may be possible to implement techniques to prevent modifying of appearances in certain circumstances.
When direct brain connections become possible severely disabled people could interact in ways currently impossible in the real world.
Some potential examples of ADEX worlds:
As with any new technology the real uses will likely become obvious after the technology becomes widespread.
Meetings would become very different:
Existing applications will also benefit from the system:
The ADEX is a system you can immerse yourself in, platform, device and location independent. A distributed system which displays abstract data in a manner depending on the local systems capabilities.
ADEX will at it's core consist of worlds. 3 or more dimensional representations of real or imaginary worlds. How you interact with these will be dependant on yourself and the capabilities of the interface you shall use. The state of the art today would be a virtual-Reality type display combined with some form of suit (think of The Lawnmower man). To the user the experience would be something like being in "The Matrix" or a Star Trek Holodeck.
Other interfaces could be a normal PC display or a mobile device, the display in these cases would be very different but even PDAs are powerful enough to run basic 3D apps these days.
Handling all of these would be a considerable challenge in itself so the worlds would not be concerned with them. It would be up to the interface to interpret the data in the worlds.
In order to create a realistic environment each object within it has obey certain rules, that is they will have behaviours. Behaviours should (when wanted) follow the laws of physics, i.e. objects should behave as they would in the real world. When hit an object should make a specific sound governed by the materials' resonant frequency. Some other properties which should be modelled are:
The ADEX system involves simulating millions of objects within a world all of which interact and have states which constantly need to be updated. This will require the server interacting with the objects - lists and indices will be required to keep them in order and prioritise them. An OS could use these objects as threads and will need to switch between them and update them in real time - latency and switching time must be low while performance must be very high.
Unique 64 bit ID per user (ADEX-User), the world could log these and forwards messages when users change world.
Messages option for sending and receiving mail, spam and virus protection could be built in from the beginning.
Addressing will consist of multiple 64 bit numbers, this allows for large growth over the lifetime of the system without having to worry about changing the addressing system - limited addressing causes all sorts of problems when it changes so get it right first time!
e.g. IPv6 address + ADEXPort + ADEXWorld + ADEXObject +ADEXUser.
How would ADEX work?
ADEX would probably operate not unlike an online 3D game but would be a great deal more sophisticated.
The ADEX is a system which would consist of several components, some dedicated to managing the worlds and others for background infrastructure.
This controls the interface with the user, it's purpose is to take data from the ADEX client and present this to the user, it shall also take input from the user and present this to the ADEX Client. The ADEX-Viewer could be an Immersive User Interface but it would not be limited to this. The purpose of any user interface is to allow the user to interact with the system. The ADEX-Viewer could present a different face depending on whatever the user is using to communicate with the system, so if a user is accessing the system with a PDA, a display like a 3D game would be displayed for 3D interaction, 2D graphics would be displayed when appropriate.
Since interface types will vary in capability, the Viewer will be different for each device and will require and generate different amounts of data depending on it's capabilities - It would only ask for the data it can use and no more.
This acts as a gateway between the Viewer and Server (the Client is like a web browser and the Viewer like a video driver). Since the Server may be on a remote system it would hold a copy of the world which the Viewer uses to generate the displays. This copy of the world would be kept in sync by data from the server via the communications system.
The Server is responsible for hosting a world. It does not generate displays but uses the inputs from the clients and the behaviours contained in the parts to control the interaction of everything within the worlds. The results of these calculations will then be sent back to the clients via the communications layer.
This handles the communications between clients and worlds. Since a great deal of data has to be communicated this will compress (and possibly encrypt) the data and send it on it's way. This will most likely be via normal internet protocols.
This would allow users to find worlds to look at and allow the communication system to find out where data has to go (it should only need to do this once per session). It would act much like DNS system web servers use but I think the addressing scheme could be improved upon considerably. All large servers would also have an index.
A component closely connected to the server that allows scripting of parts. This would be used to make immersive movies.
Worlds would include many parts and these all have behaviours depending on what they're made from. Instead of attempting to download all of this you would only give basic descriptions and generate the full description from the ADEX Store. The Store contains information such as:
ADEX VM / Applications
Programs run inside a world, for platform independence these would run using some form of virtual machine (much as Java does).
ADEX Application Interfaces
Interfaces to programs which run outside ADEX, this would allow a user to utilise existing non-ADEX software. These could range from having a virtual screen and mouse to being able to move data to and from the ADEX world.
ADEX Operating System
ADEX is not an operating system in the strictest sense of the term but it will need to provide much of the functionality of a modern OS. On the other hand there is no reason why it should not be perfectly possible to host ADEX on an existing OS in which case ADEXOS shall handle the communication with the host OS. A hosted ADEX will be the easiest way to start creating ADEX but ultimately ADEX will require access to most of the OS functions and a native ADEXOS would be best suited to this because it could be designed for this specific purpose.
The Bandwidth Problem
Connections with other computers are invariably slow and low bandwidth. Consequently as much caching as possible should be used. This can be extended to the basic shapes and behaviours of objects in any given world.
In order to communicate ADEX clients and servers will need to send a great deal of data to one another with low latency. This data can be reduced in a number of ways:
1) The ADEX Stores
As with real life parts in worlds will be made out of specific materials with properties, these are stored in the ADEX stores. Every server and client will have a copy of the library so rather than sending a full description of an object and all it's properties, a more general description can be sent based on objects, materials and properties from the library. A full description of a human being will be highly complex and include a lot of data, a description of a "basic" human modified to look like someone will be a lot smaller and thus easier to transmit.
This may seem obvious but the web didn't have compression built in from the beginning, the ADEX should compress everything in the aim of saving bandwidth.
Object X is moving at a specific speed in a given direction, it is going to hit object Y. Rather than waiting for the objects to interact then requesting data for Y's reaction from the server the interaction could be predicted and faked.
Rather than trying to describe an objects using individual polygons it could be described as a set of curves and each system reconstruct the full polygon model.
If you move you arm your hand is also going to move. Rather than sending the movements in exact detail only the most important parts are sent and the rest of the movements can be calculated. If you can reduce the movement down to the movements of bones there is a lot less data to transmit.
If an object moves into a world with an image mapped onto it a low resolution version can be sent first so it can be seen then the data filled in. Movements of objects that are close should be sent first and only then more distant object's data.
If an object is out of view data only low resolution data could be sent or alternatively none at all. If a user is alone in a room and nobody is in the area the server could let the client handle everything removing the need for communication altogether, only simplified data would be sent back to the server periodically.
7) Distributed worlds
A large world with a large number of users will prove problematic in terms of both computing power and bandwidth. An obvious solution to this is to break up the world into different areas and host them on different physical systems. This does however pose the problem of how to handle interfaces between different the parts of the world and how this affects the users passing through them.
8) Customised Network protocols
Probably the most difficult part would be to use a customised network standard. WAP uses it's own network protocols which were were designed for low bandwidth connections so something similar could be used here.
Random Ideas and Notes:
This is a list I never got around to finishing off...
Copyright (c) Nicholas Blachford 2001 - 2004
Nicholas Blachford's Home page