Rauli Puuperä, Playsign Oy
Toni Alatalo, Playsign Oy & realXtend association
DRAFT Mar 1st 2012
This document reports the research and proof-of-concept implementations of a browser based realXtend client. We consider and compare different technical solutions. Main goal here is to see whether realXtend scenes / applications can be deployed so that they a) don’t require installation and b) work reliably for everyone (on all kinds of machines).
This research originally was made in a customer project where also voice was required, hence the focus on that here as well. Voice is a key element in many virtual world applications, and is a typical part of on-line gaming experience too.
Using the existing native C++ Ogre+Qt Tundra client implementation inside browsers is also possible, as a plugin (Active-X or NSAPI) or in Chrome as a Native Client (NaCl) component. That was outside the scope of the original research here, as in the classical form plugins require installation, but a Tundra plugin has been experimented independently (with NSAPI) by Jukka Jylänki of Ludocraft and results from that work could be either integrated to this document, or a combined analysis could be made in a new doc.
The main conclusion is that a browser based client can be implemented with reliably working technologies for 3d rendering and voice conferencing.
Some plugin is needed for voice in any case, as browsers do not currently allow reading the microphone input. Flash is very widely deployed and allows production quality voice conferencing functionality with a mature open source software suite: Big Blue Button.
Since late 2011 Flash is now also capable of advanced 3d rendering. An existing open source graphics engine, Away3d, was successfully tested.
Alternatively, also WebGL works for the graphics rendering. The same Flash based voice solution can be used also with WebGL rendering and the WebSocket networking which exist from before in the realXtend WebNaali project. WebRTC is a proposed new standard for microphone & camera input in browsers which may change this in the future -- it has just become available in Chrome developer betas in February 2012. WebGL is a strong alternative when a fully open source, with open spec and multiple implementations technology is wanted. Also WebGL works with hardware acceleration on Linux also. WebGL is also most straightforward to seamlessly integrate with other web page elements, as it is not a plugin but a part of the integrated browser functionality.
Flash is however best also for the 3D rendering if the goal is to reach the widest possible audience, including MS Internet Explorer users. It also has more automatic and possibly best performing software rendering for machines without suitable graphics cards. Currently Flash always resorts to software rendering on Linux, due to driver uncertainties. Also tight integration of the 3d scene with avatars and the voice conferencing functionality, for example speech indicators, is easiest in a wholly Flash based client application.
We put up a video that shows BBB and Away3D in action.
Live demo of that prototype is available on an Amazon EC2 instance: http://ec2-107-20-110-120.compute-1.amazonaws.com/ . The BBB connection there is real so chat works (voice is not configured on though) but this version doesn’t connect to Tundra, the movement with arrowkeys is local.
We also wrote a simple server component for realXtend Tundra which we used in testing the network capabilities of Flash Player. There are several options how we could implement network connections for this project (FlashNaali), including the same kNet UDP protocol which is used with native Tundra. There is no on-line demo of the client with connectivity now, we may setup one later.
For 3D graphics we have basically three different technologies to choose between. We can use:
We tested the existing version of WebNaali without a network connection using 15 animated avatars and a moderately complex scene. Loading (to memory) is slow and there is initial slowness, but eventually the animations run smoothly and the camera could be moved around without considerable lag. See running demo.
Basically GLGE is good if your target users use always the latest versions of Chrome or Firefox. Enabling WebGL in Safari is not hard but it might be an extra step that is too much for the most casual users. WebGL won’t work in IE anytime soon if ever.
WebGL Animation test with GLGE
Overall GLGE is a nice WebGL library that makes WebGL development more productive. The documentation is a bit lacking but we have established contact with the main developer who has been very helpful in the past.
Three.js is an interesting choice since it can fallback on 2D rendering if WebGL is not available. Sadly this doesn’t help with Internet Explorer issues since the drawable 2D canvas is only supported by the latest IE version 9 which has a small install base at the moment.
Flash or Adobe Flash is a multimedia library provided by Adobe. Since from version 11 it supports hardware accelerated 3D graphics called Stage3D. There are some libraries that are built on top of it, both open source and proprietary. Flash has a big install base but version 11 is rather new and might not be installed so widely as previous versions. If you have the FlashPlayer installed the updating process is pretty straight forward.
If no suitable 3D hardware acceleration is available Stage3D switches automatically to software rendering. Adobe maintans a list of graphics cards on which hardware rendering is enabled.
Away3d is an open-source flash 3d-rendering library. The new version (4.x) utilizes Stage3D and hence uses hardware rendering with suitable graphics cards.
Away3D utilizes its own mesh format called awd which is specially designed for transferring data over network connections. There is a awd export add-on for Blender.
A heavy animated character from realXtend’s 4avatars project, animated with software skinning in Away3d. Exported from Blender as md5. Demo on-line.
After some hardship we managed to get the blender awd exporter in Mac OS X. It seems that currently there is some problems with exporting models with an armature. However, Away3D supports also md5 meshes and animations. We managed to export one of the realXtend avatars in md5 format and got it running in Away3D. Animated av’s work using the both awd and md5 formats. The Blender AWD exporter doesn’t work correctly for skeletal animations (yet). For an optimized test, we used a skeletally animated baby polar bear, exported from Maya, from the Away3D repository in testing / demos. This skeleton is authored so that it works with hardware skinning (GPU deforms the animated mesh). The woman from realXtend 4Avatars project (in image above) exceeds both the bone amount limit (has multiple for each fingers and many for the face) and the number of vertices assigned per joint. So in this demo Away3d resorts to software skinning which gets heavy especially with multiple animated characters with this many bones.
The restrictions with Flash & Away3d for hardware skinning with skeletal animation are:
Dynamic lighting (shading) slows down the rendering for plain software rendering. But if we used baked textures with otherwise flat light performance should be good.
All in all Away3D is a mature open source project with good documentation and a good development environment.
Example of 15 animated polar bears with Away3D
Unity3D is a commercial game engine with which you can make games for multiple platforms such as IOS, Android, Nintendo Wii. There is also the possibility make a games (or 3d applications) in Flash with the version 3.5 developer preview. We exported a test object and scene in Collada from Blender and put it up for display with Flash using Unity.
The loading time was considerably longer than with Away3D, perhaps because it always loads a relatively large Actionscript version of the Unity3D runtime?
123FlashChat is a commercial flash voice/text chat software. You can host your own chat server or buy hosting. We downloaded their software and tested it with three connections in the same LAN. The sound quality was OK and latency was around 0.5 seconds. The voice chat room seemed to have a 12 account limit.
FlashComs is a similar company with 123FlashChat.
Red5 is a Java server software which can be used serve voice chat. For example 123FlashChat uses Red5 as their server technology.
Since there are solutions available which use red5 we saw no point in creating one ourselves.
Adobe Media server is Adobe’s own server which can be used for voice chat. Due to its closed and commercial nature we didn’t look it any further.
jAudioBridge is a Java based sip/rtp server that could be used for voice chat. We didn’t find any good clients for it so it seemed like a dead end.
Big Blue Button (BBB) is an open source project that uses a myriad of different open source components which offer a full-blown eLearning environment. It includes both server and client side components. The server is a complex system of different components. We can use telephony component for the voice chat. BBB also has a basic chat.
Hosting should be pretty easy since BBB offers an virtual machine image that has all the necessary components installed. We tested the BBB server by running the server image in our own machines and it worked. The public BBB demo is hosted at http://demo.bigbluebutton.org/
We also installed BBB to a Amazon EC2 cloud hosted server instance successfully and made a successful voice test using a m1.small type instance (which are not expensive).
Openmeetings is another open source project using the red5 server and an own Flash client. It is superficially similar with BigBlueButton, but there are major differences (e.g. the voice server used) -- a good brief comparison from a BBB developer is given in this January 2011 posting.
XXX insert treatment here XXX
As a quick proof of concept we wrote a simple TCP server module to Tundra to handle incoming connections from Flash client. The server is mostly based on the WebSocket server module written for WebNaali. This quick and dirty module was written as a part of this initial research and is not ready for production. Options for development are:
If the target is to ensure that the client works for everyone with current technology, without (extra) installing, the choice seems obvious: Flash has over 90% install base and it can be easily updated with few clicks to the latest version which can hardware accelerated 3D graphics. Away3D is a mature open source library that has the capabilities which are needed in a realXtend client. Skeletal animated characters work with hardware skinning within given, quite reasonable, limitations for the rigging.
It seems that for the voice and text chat are Big Blue Button is well suited. It is open source and both the client and server components have been used successfully even though extensive voice test have not been made.
For network side with Flash there are two alternatives, and for plain pluginfree browsers only a single way. That is: Flash can do normal UDP and TCP connections, so we could implement the currently used kNet in a Flash client. However that may not be most efficient way for the binary serialization, as the Flash runtime has zip compression built-in (in native code) which might be lighter to use than doing complex deserialization in Actionscript. With plain browser Websockets are the only way, and there an upcoming extension will enable zip-compression of the stream (by reusing the compression frame cross messages) with native code in browsers.
A crucial component considering performance is the 3D rendering. Not all computers have hardware accelerated 3D graphics. We ran the rendering demos with different systems ranging from a five year old laptop to modern desktop computers. The conclusion of the tests was that if you craft the scene with baked textures and low detail skeleton animations you can have good enough experience even with low-end laptops.
In addition to previous research work we considered the feasibility of implementing various virtual world functionality. It seems that most of the desired functions can be implemented with reasonable effort.
Most of the 3D libraries we considered in the research phase have some sort of video textures that can be used to display videos. This is the case for Away3D also. We didn’t test it but we found this convincing demo.
Embedded Web page display seems possible but tricky. As far as our research concluded there currently isn’t any practical way to render web page contents on a 3D object in Flash. However, since most of the time during the research phase of the project was focused in the core functionality we don’t have a definite answer whether it is possible or not. For WebGL there is a proposed extension to allow rendering web pages to GL textures.
A RSS reader can be implemented so that the news feed is shown with text rendering in the scene (on a display or not), instead of display it as a web page. Both GLGE and Away3D have good text rendering (something that is still missing from the Ogre using native Tundra, in fact). Code for a Flash based RSS reader is for example at: http://alaashaker.wordpress.com/2008/09/09/build-your-own-flash-rss-reader-tutorial-flash-actionscript-30/
For displaying web pages in the 3d scene, it s also possible to render them to images on the server side -- this approach has been used with the Opensim server and Second Life client before SL implemented ‘MOAP’ (media on a prim).
At the moment the application level Tundra compatibility of the FlashNaali prototype is minimal. The client receives just few messages and send entity actions to Avatar application at the server. The feasibility of any Tundra feature needs to be considered separately.
The proposed FlashNaali is a Tundra client. Server side Tundra functionalities run normally without modifications. However, current client side implementations of functionality for the native Tundra can not work unmodified in a browser based client, as the GUIs are implemented with Qt which is not available. Also the API for the 3d scene, which uses Ogre in the native client, is different.
 15 skeletally animated arctic friends in Flash Player 11 with open source away3d library