Infrared depth sensor DTK NUI development kit using an infrared depth sensor Price: Ask quatation Release: January, 2012 Simulation

Introduction

Currently the amount of development of NUI(Natural User Interface) as a new interface has been increasing. NUI is a new interface which allows the instinctive operation using voice and body. DTK(Depth Tool Kit) is a development kit using infrared depth sensor. It can recognize the user by the distance information from the infrared depth sensor and detect the body’s moving on 3D. Unlike the usual motion capture, one of the merits of this is that it doesn’t need to put a sensor or mark on the body.

Infrared depth sensor

The usable device for DTK is 3 of Xtion PRO, Xtion PRO LIVE, Kinect(TM) (Figure１). The technology of PrimeSense, Ltd. is used for the infrared depth sensor for each. Irradiating with infrared laser from one side of the lens, reflection light is read on the other side of the lens (Figure 2). The green lens is laser and the opposite side receives the light for the sensor of figure 1. The middle of the lens of Xtion PRO, Kinect(TM) is RGB camera. Only infrared depth sensor is mounted on Xtion PRO. Xtion PRO LIVE and Kinect(TM) are multifunctional devices with the other functions (Chart 1). XtionPRO Xtion PRO LIVE Kinect(TM) sensor Figure 1 Infrared depth sensor (Xtion PRO, Xtion PRO LIVE, Kinect(TM)) Figure 2 Pattern of infrared laser Function Xtion PRO Xtion PRO LIVE Kinect(TM) Infrared depth sensor 640X480(VGA)/30fps 320X240(QVGA)/60fps RGB Camera X 1280X1024 (SXGA) 640X480(VGA)/30fps 320X240(QVGA)/60fps AudioMicrophone X 2 4 Tilt Motor X X o Access System USB2.0 USB2.0 USB2.0 Power supplier USB bus power USB bus power AC adapter Chart 1 Comparison of features of infrared depth sensor DTK supports the functions of infrared depth sensor and RGB camera. The effective range of infrared depth sensor is Xtion PRO, Xtion PRO LIVE, and the nominal value is from 0.8m to 3.5m. The nominal value for Kinect(TM) is from 1.2m to 3.5m.

Overview of DTK

DTK is a tool kit which simplifies the use of infrared depth sensor. Recognition of users and even acquisition of 3D motion can be simply done by just sending the order as well as acquiring depth map and images of RGB camera (Figure 3). It also allows to recognize and operate several users at the same time and to be used as an input device immediately by using gesture interface and 2D menu creation function. Figure 3 Function of DTK(Depth ToolKit) It also allows to create totally different application using information of depth map and 3D motion data.

Using method of DTK

DTK can be used regardless of the language or platform (Figure 4). DTK is used as a class library for  C++ such as Visual Studio(R). After assuming it as a class, required member function is called and processing is executed. Application using an infrared sensor can be created by simply describing a processing which matches the event. In the other language or platform, it works as a server using UDP communication. Users can use the information of infrared depth sensor with communicating with DTK as a client program. DTK can be used regardless of the language or platform because it works as the defferent application. Figure 4 Using method of DTK Clients send the processing to be executed and necessary information to DTK after connecting with server on UDP. If necessary, DTK transfers the data to a client program. It’s also possible to let DTK display the depth map and RGB image as well as using it as a background program.

AirDriving &Gesture Interface

AirDriving & Gesture Interface is a handless driving system which combines inputting 3D motion and inputting gesture input (Figure 5). Running on the model of UC-win/Road is allowed using the information of hands and feet input from infrared depth sensor (Figure 6). During steering operation, move the both hands lile holding the real steering wheel. For the operation of axel and brake, move the right foot like stepping on the pedal. It can react to the insensible movement, so the smooth driving is possible. Figure 5 AirDriving (Tokyo Game Show 2011) Figure 6 AirDriving Operation Switching of drive/ back is conducted by gesture. Gesture is input by moving the right hand like figure 7 in the state the left hand is fixed. In AirDriving, gesture 1 is drive and gesture 2 is back. By assigning the other gestures to scenarios, operation from start of driving to the end without mouse or key board is possible. Figure 7 AirDriving Driving (1)／Back (2)

Example of use of infrared depth sensor in application

Starting with AirDriving, applications using infrared depth sensor such as teleoperation of robot is under development in FORUM8 (Table 2, 3). Application Use of infrared depth sensor A)Simulator AirDriving Hands-free driving B)Teleoperation RoboCar(R) Driving control AGUL AR.Drone Flying Control Robotic arm Motion Control SLAM 3D mapping Table 2 Applications using infrared depth sensor Function AirDriving RoboCar(R) AGUL AR.Drone Robotic arm SLAM Depth map o o o o o Image of RGB stream X X X X o User space o o o o X 3D Motion Capture o o o o X Gesture Interface X o o o X Table 3 Utility form of infrared depth sensor for each application RoboCar (R) is a 1/10 scale model car of ZMP Inc. Driving simulation synchronizing between UC-win/Road and RoboCar (R) using AirDriving and gesture interface is possible. AGUL AR.Drone(P9, Up&Coming vol. 95) is a quadrocopter of Parrot (Figure 8). It enables a stable flight and can be used for taking picture from the air. AGUL AR.Drone and robotic arm (Figure 9) can be controlled using 3D motion capture and とGesture Interfac. SLAM(Simultaneous Localization And Mapping) is a system which estimates the self-location and creates the environment map at the same time (Figure 10). By the conbination of distance information of infrared depth sensor and colour information of RGB camera, 3D point cloud model (environment map) is created in real time and a sequential space is generated comparing the obtained images (Self-location is estimated.). This is used for control of robot and 3D mapping. Figure 8 AGUL AR.Drone(Parrot SA.) Figure 9 Robotic arm AL5D(Lynxmotion, Inc.) Figure 10 3D scanning using SLAM DTK supports the development of these applications.

(Up&Coming '12 New Year Issue)

Up&Coming