Multimedia Communications and Applications

1- Distributed Virtual Environment with Training Applications (DIVERSIONS)

Start and End Dates
January 1, 2001 – January 30, 2003

Principal Investigator

• Nicolas D. Georganas, Distinguished Professor, SITE, University of Ottawa

Co-Investigators

• Emil M. Petriu, Professor, SITE, University of Ottawa
• Eric Dubois, Professor, SITE, University of Ottawa
• Ognian Kabranov, PDF, SITE, University of Ottawa
• Dimitrios Makrakis, Professor, University of Ottawa
• Dorina C. Petriu, Professor, Department of Systems and Computer Engineering, Carleton University
• Pierre Boulanger, University of Alberta (formerly with NRC)
• Thom Whalen, Research Scientist, Network Systems, CRC
• Linda Jackman (formerly with Alcatel)

Summary:
A Distributed Virtual Environment (DVE) is a software system rendering a simulated virtual world. A DVE runs over a network of computers allowing people who use them to interact in "real time" and share the same virtual world, a shared sense of space, a shared sense of presence; and a shared sense of time. The computers and their users may be physically located in different places around the world. On each host computer "entities" communicate their changing state by sending "update messages". A Collaborative Virtual Environment (CVE) is a DVE with special emphasis placed on user collaboration allowing for example multiple users to jointly perform closely coupled tasks.  In a CVE, users are represented by "avatars" sort of computer-generated incarnations. Collaborative work performed within a CVE requires a broad range of networking, database and graphics capabilities. CVE applications include tele-surgery, tele-training, collaborative design/engineering, games and entertainment. Research under this project focused on the following objectives:

• Development of enabling technology and multimedia applications for Distributed Virtual Environments (DVE), to allow multiple globally-situated participants to collaborate in industrial training situations over IP networks connected to heterogeneous computing resources and large data stores.
• Investigative study of user perception and user needs to assist with the design of good DVE and Augmented Reality user interfaces;
• Study the performance of networking technology for these applications;
• Training of graduate students and other researchers;
• Development of a distributed demonstration test bed across NCIT nodes over the NCIT*net.

This research built on previous work done at the University of Ottawa which established foundations in 3D rendering based on Java 3D, industrial training interface, second view and video streaming and CANARIE Trial over IPv6 and CA*net II. 3D tele-learning application launched as life demo among three IPv6 islands (MCRlab, UWO and CRC) linked through CA*net II by OC3 connections.

2- Haptic Augmented Reality Multimedia for Networked Interactive Environments (HARMONIE)

Start and End Dates
September 1, 2002 – September 30, 2004

Principal Investigator

• Nicolas D. Georganas, Distinguished Professor, SITE, University of Ottawa

Co-Investigators

• Emil M. Petriu, Professor, SITE, University of Ottawa
• Eric Dubois, Professor, SITE, University of Ottawa
• Dimitrios Makrakis, Professor, SITE, University of Ottawa
• Dorina C. Petriu, Professor, Department of Systems and Computer Engineering, Carleton University
• Thom Whalen, Research Scientist, Network Systems, CRC
• Francis Bogsanyi, NOVINT/Algonquin, Visiting Researcher from Australia (Haptics)

Industry Partners
MPB Technologies, Handshake and FCS Control Systems

Summary:
At their root telecommunications technologies derive a significant amount of their value from the simple concept of extending the reach of the human sensory space. It is not a big jump to realize that the next sensory frontier will be that of touch. At the National Capital Institute of Telecommunication we believe that the timeline is short and that, if chosen well, services with a "touch" sensory component will have a high value both to the industries which use them and to the service providers who supply them. This is what Haptics is all about. The term "Tele-Haptics" is used here to indicate the use of haptic interfaces to a communications network as opposed to the direct control of a robot or game.

In tele-haptics, three generic service areas are education and dexterous skills training, product development (extension of the 3-D computer aided design market) and group working to scale: the "haptic workbench" allows small teams of experts to work together on a problem in an environment that would be difficult to get otherwise because of scale issues. Tele-haptics or the transmission of haptics applications across networks will put new demands on elements of the network. Issues of latency, stability and reliability are of significant importance and represent challenging research topics. 

The HARMONIE project started in September 2002 to address the topic of tele-haptics. Building on multimedia expertise at the University of Ottawa, HARMONIE research work focused on four primary vectors that correspond to pure and applied research in human-computer interaction and in network performance for tele-haptic communications.

• Applied research in HCI within the tele-haptics context that involved the construction of example applications and the evaluation of their worth. The HARMONIE project focused on haptic-enabled eCommerce and collaborative dexterous task training simulations.
• Pure research in tele-haptics HCI that included human physiological and psychophysical models for haptic sensory perception, and multi-sensory fusion, as well as comparing task performance in virtual environments with and without haptics. Extension of this work to produce models of acceptable limits related to network artifacts in tele-haptic applications, and degradation models corresponding to ideal, acceptable, and unacceptable network performance, from the human user’s perspective.
• Pure networking research for tele-haptics to progress and map (in concert with the HCI research) constraints from the user’s perspective to network requirements. It is expected to form a theoretical model of network performance requirements for different tele-haptic application scenarios.
• Applied tele-haptics network research to involve the deployment and testing of an eCommerce haptic applications and collaborative dexterous task training simulators over a variety of network configurations, including experiments over NCIT*Net, experiments between Ottawa and CSIRO in Canberra, Australia, and the evaluation of the theoretical network performance and HCI models.
• The research team of the multimedia DIVERSIONS essentially planned to run this project but with additions of experts in the Haptics technology. The team’s focus was on extending their multimedia networking expertise to cater for the stringent networking needs imposed by tele-haptics. Dr. Nicolas Georganas along with a distinguished team of researchers at the University of Ottawa, Carleton University, CRC and CSIRO in Australia conducted the research under this project.

3- Research in Tele-Haptics

Start and End Dates
September 1, 2002 – September 30, 2004

Principal Investigator

• Jack Treuhaft, Director, Applied R&D, Algonquin College (Since Retired)

Co-Investigators

• Francis Bogsanyi, Algonquin Visiting Researcher
• Peter Casey, Staff Member, Algonquin
• Gerald Paquette, Staff Member, Algonquin
• Stephen Ryan, Staff Member, Algonquin
• Jim Mikolaitis, Staff Member, Algonquin
• Bernard Duffin, Staff Member, Algonquin
• Frank Gushue, Staff Member, Algonquin

Summary:
Until recently, most virtual environments have relied on visual and acoustic feedback to create the illusion of reality. Haptic devices contribute the useful additional element of force feedback. New haptic devices can provide up to six degrees of force to assist in establishing realistic feedback to the user creating a truly multi-sensory virtual environment. Hapto-visual/acoustic environments involve the simultaneous rendering of a graphic virtual environment, sensing of the haptic stylus within the virtual workspace, and control of the force feedback provided to the user. Haptic workbenches thus involve significant computational activities to establish the virtual environment (which is typically dynamic), track and interpret the user’s movements, and provide appropriate feedback to the user. Additionally, haptic environments are frequently presented to the user as three-dimensional using a variety of visual stereo devices. The tasks of linking two such workbenches, both locally and over a distance are non-trivial.

At the time of this project, Haptic technology was underdeveloped and poorly exploited with relatively few applications. The simulations that have been in use were mostly in the medical discipline and tended to be relatively simplistic in nature.  An enormous potential existed to develop innovative haptic applications especially as educational and industrial training simulations. The ability to manipulate three-dimensional objects along with force and acoustic feedback over a broadband network presents further possibilities for innovation. Tele-haptic applications pose interesting challenges due to their need for low latency and high bandwidth connections.

An opportunity existed to explore the use of collaborative multi-sensory virtual environments to enhance existing educational and industrial training environments by developing training simulations for dexterous tasks. Multi-sensory virtual environments can also be used to haptically enhance virtual prototyping possibly by obviating the need to build real prototypes. The ability to create, distribute and manage these multi-sensory environments using broadband telecommunications networks was considered an essential element of success.

The main focus of this project was to explore the educational and industrial design applications of linked hapto-visual virtual environments using high fidelity 6 degrees of force feedback devices;

• Creating hapto-visual training simulations for dexterous tasks in educational and industrial design applications
• Creating linked hapto-visual interactions involving local simultaneous multi-user interaction
• Establishing collaborative hapto-visual interactions among multiple users over an advanced long distance network

Algonquin College already has a well-established Haptic infrastructure which will be used to directly support this research project. This infrastructure has been acquired through funding from the Ontario Innovation Trust and includes two six-degree-of-force (6DOF) haptic devices with precision positioning input and high fidelity force output from MPB Technologies, one Phantom desktop haptic device from SensAble to be used for portable hapto-visual applications and three-dimensional graphics environment, ReachIn 3D integrated hapto-visual scenegraph based API to create the three-dimensional environment using the emitter and shutter glasses and drive the graphics card, emitter and shutter glasses to help establish the three-dimensional environment from Stereographics, a “left” hand 3D mouse orientating device from Logicad3D, Generic mirror used to help create the stereoscopic view and two dual-processor workstations.

4- Enhanced Personalized Collaborative Information Systems

Start and End Dates
July 1, 2003 – November 30, 2004

Principal Investigator

• Abed El Saddik, Assistant Professor, SITE, University of Ottawa

Co-Investigators

• Mauricio Orozco, Graduate Student
• Pullola Sherisha, Graduate Student
• Anwar Hossain, Graduate Student
• Khalil M El-Khatib, Research Officer, National Research Council
• Jianming Ke, Graduate Student

Industry Partners
PEETA Consultants and NI Solutions

Summary:
Collaborative virtual environments support interactive knowledge development and integration amongst participants with diverse backgrounds. There was a need for a cost-effective easy to use solution that avoids complex downloads and complicated set-up procedures. At the time of the project there are many web-based collaborative environments, for example web-mail and text/audio/video chat rooms.

• The Research focus was to develop multipurpose collaborative environments that use 2D/3D objects and text/audio/video/streaming to facilitate secure personalized e-learning and audio/video conferencing. The environment was planned to be modular, flexible and easy to use to allow customization by end users. The purpose has been to revolutionize the way teams can work together and reduce issues associated with current teleconferencing technologies.
• The research focused on planning to develop tools for automatic indexing of multimedia content through use of appropriate metadata standards (IEE-LOM, MPEG 7), for support of shared cut-and-paste capability and for low cost support of video conferencing. A particular contribution was the design of a generic single sign on (SSO) mechanism. The user interface of the developed environment has been evaluated using usability engineering methodology.
• The research built on successful experiences at SITE developing web-based e-learning systems and tele-collaborative tools (e.g. JASMINE) as well as the expertise and collaborative tools of our partners (PEETA consulting with their collaborative 3D modeling engine and browser and NI Solutions with their Flash Communication Server that it is currently used by NCIT).

5- Tools for Analysis of Dynamic Behaviour of Complex Real-Time Software Systems

Start and End Dates
July 1, 2003 – November 30, 2004

Principal Investigator

• Timothy Lethbridge, Associate Professor, SITE, University of Ottawa

Co-Investigators

• Abdelwahab Hamou-Lhadj, Ph.D. Student
• Thomas Fletcher, Neutrino Development Group, QNX
• Eric Fu, M.Sc. Student

Industry Partners:
QNX

Summary:
Software systems, particularly real-time, can be some of the most complex engineered artifacts; software engineers must understand many different aspects of such systems before they can make changes that preserve reliability and other system attributes. There are many tools at the engineer’s disposal, the focus of this project was on tools  that permit the analysis of traces. Traces have historically been difficult to work with since they can be extremely large – often millions of lines long. Engineers have instead been forced to rely on static analysis, debuggers, profiling and other approaches. Traces, however, may be the only place where the manifestations of some problems may be found, therefore the team set to investigate ways to allow software engineers to somehow find and absorb information despite large trace size.