Information from the sensor is sent from a dedicated server program developed using the OpenNI library, a common platform for 3D sensors, via UDP through each and every port to the depth map and skeleton tracking feature which are implemented to the plug-in, and to the client within which different gestures are visualized as different 'actions' such as Air Driving, changing virtual gear lever, etc.

Gestures from the sensor are interpreted in UC-win/Road, enabling completely hands-free control of a vehicle within the UC-win/Road environment.

This system allows the operation of UC-win/Road by moving the arms as operating the steering in front of Kinect. The positions of both arms which are detected by Kinect are converted into steering, acceleration, braking of vehicle.

What is Kinect Sensor?
Kinect Sensor (Hereinafter, Kinect) is used as a peripheral device of "Xbox360®"which is a home-use game machine sold by Microsoft and is the device which allows you to operate the system via physical behavior, gesture and voice without a controller. A camera and sensor are mounted on it and players are automatically recognized when they stand in front of the Kinect.
This technology is based on the depth camera technology developed by PrimeSense. The sensor includes a VGA camera, 3D scanner, 4 microphones, inclined motor, etc. The 3D scanner is composed of an infrared laser (whose frequency is a little lower than that of a red signal) that gives off several dots in a continuous pattern. Although the laser is not visible to the naked eye (as the wave length is about 780nm), it can be detected by an infrared camera(Figure 1a and 1b).

Figure 1a A night-vision image of Kinect		Figure 1b Expanded view with points projected on the wall

1. Output of the distance image
Kinect provides the distance image of 640 x 480 pixel (The distance which is the closest from the sensor is stored per pixel.).

2. User detection and tracking
Kinect allows to detect and track the human within the visual field of the sensor.

3. Skeleton detection and tracking
Kinect allows to detect and track the joint position of the human.

AirDriving Interface

4.Air driving
Kinect allows to recognize the steering operation with both arms and the acceleration / braking operation with the right foot and enables you to operate the driving simulation by gesturing (steering, acceleration, braking operation are applied in UC-win/Road).

Gesture driving simulation of UC-win/Road via Kinect Sensor and 6K Multi-Cluster Digital Signage system (Tokyo Game Show 2011)

• Neutral position
  The operation of acceleration and braking is performed based on the distance from Kinect to the arms. It can be operated by moving backward and forward from the neutral position.   
  
• Steering operation (Right-turning, Left-turning)
  The steering can be operated same as actual steering control. It supports for both of right-turning and left-turning.
    
  
• Acceleration operation
  It can be accelerated by moving the arms from the neutral position to the front. The further you move the arms, the larger the acceleration amount becomes.
  
  
• Braking operation
  The braking operation is performed by moving the arms from the neutral position to the back. The further you move the arms, the larger the braking amount becomes.
  
• Detection of steering angle and steering direction
  The right and left direction of the steering wheel and its steering angle are detected as an analog value from the positional relationship between the user's right and left fist. How much the steering wheel is turned is indicated by a row of bars that change color from green to red. The bigger the steering angle, the more red the bar will be.
  

  Travel in a straight line	Right turn : As you steer to the right, the red bars will increase in the right direction.	Left turn : As you steer to the left, the red bars will increase in the left direction.

  
  
• Distinguishing accelerator from brake
  Acceleration and braking can be distinguished by detecting the angles between the right toe opening right and left.
  

  Accelerator position	Braking position

  
  
• Detecting the amount of pressure on gas pedal and brake pedal
  Accelerator
  This is detected as an analog value from the amount of pressure the driver exerts on the accelerator pedal. In case of accelerator, the stack of bars will increase in the upward direction.
  
  Brake
  This is detected as an analog value from the amount of pressure the driver exerts on the brake pedal. In case of brake, the stack of bars will increase in the downward direction.
  
  

5. Virtual gear lever
Grasping behavior of objects and gear operation can be recognized (Moving forward and backward can be switched in UC-win/Road).

6. Gesture Interface
Up to 12 kinds of operation can be distinguished by combining the operation of right hand moving from right to left or up and down (Moving forward and backward can be switched in UC-win/Road.).