Users Report
Users Report / Vol.130
MHI Sensing Dept., Automotive Business Unit,
SenseTime Japan Ltd.

Committed to Uniting State-of-the-Art Computer Vision Technologies and Manufacturing Know-how of Japan
UC-win/Road DS Utilized for Development and Demonstration of Driver Monitoring System

SenseTime Japan Ltd.
Location: Nakagyo-ku, Kyoto
Services: Provision of Solutions Applying Computer Vision and Deep Learning Technologies

“We are expanding (our business through development, proposals, and response to reviews) by examining many kinds of onboard applications using AI (artificial intelligence) algorithms.”

Using state-of-the-art technologies of AI and computer vision (digital image processing), SenseTime Japan Ltd. provides solutions for areas of automobiles, Smart City, FA (factory automation), robotics, and so on. It was about three years ago that Mr. Masanori Sato, Marketing Manager of MHI Sensing Dept., Automotive Business Unit, SenseTime Japan Ltd. joined the Company, which was Originated in Multimedia Lab of the Chinese University of Hong Kong and had just established its Japanese subsidiary. Since then, he had been coordinating his knowledge obtained in the automobile industry he was working for with the major technologies that the Company had been fostering originally through the Chinese market. This is what Mr. Sato describes a part of their stance aiming to create new values from such processes.

Our user to pick up in this issue is Automotive Business Unit, which is responsible for development of application specialized on onboard products to their business expansion in SenseTime Japan Ltd. Special focus is put on MHI Sensing Dept., which places greater weight on development of the driver monitoring system (DMS) for detecting drivers’ situations by face authentication technologies.

Aiming at more realistically reproducing experimental environments indispensable for developing DMS, the Company introduced the driving simulator (DS) based on “UC-win/Road”,3D real-time VR of FORUM8 in 2018. At the same time, the Company has also been utilizing it as a tool to demonstrate its outcome effectively. In addition, towards research and development related with new human machine interface (HMI), he also mentions an idea of increased use of UC-win/Road in consideration of function extension in the future.

Aims and Development of Japanese Base by Company Originated in Hong Kong Laboratory

In 2001, Professor Tang Xiaoou, the Chinese University of Hong Kong established Multimedia Laboratory (MMLab) in the Univ. Later, through making researches on image recognition and deep learning etc. in the MMLab, they rapidly implemented a method to apply deep learning technology to computer vision. Following this, SenseTime Group Limited (China) was established in 2014.

Since then, along its business expansion, SenseTime Group has been increasing its bases other than its headquarters in Hong Kong step by step, in cities in China including Beijing, Shanghai, Shenzhen, Hangzhou, and Chengdu, as long as Singapore, Malaysia, Dubai etc. Each base has a function of technology development. As part of this, SenseTime Japan Ltd. was founded in 2016 at the aim of linking the world’s top-class computer vision technology using deep learning, which the Company boasts of, with Japanese technological capacity that has been leading the world for long time as “a great manufacturing country” (according to Shihong Lao, CEO of SenseTime Japan).

In starting up its Japanese Subsidiary, they developed its business with on-board equipment as its main service. This is partly due to its history of starting from joint study related with autonomous driving, and members having background of research and development in this area were gathered. Currently, the Company has about 60 employees at its activity bases of the headquarters (Kyoto), Kyoto Lab, Tokyo Office, Utsunomiya Office, and AI / Autonomous Driving Park (testing course in Ibaraki prefecture).

Driver monitoring system using the
face authentication technology
At a get together after the kickoff event
for the 4th anniversary of SenseTime Japan
Mr. Masanori Sato, Marketing Manager,
MHI Sensing Dept., Automotive Business Unit

Approach for Japanese Market Explored Using Technology of its Parent Company to the Fullest

While having high-level technological accumulation of SenseTime Group as a base, SenseTime Japan also places weight on its original techniques and know-how it develops and progresses in Japan.

The Company holds up the following points as its major technologies; 1) Face detection / authentication using original face recognition deep-learning algorithm, face image database and deep learning platform, 2) Detection / tracking of vehicles or pedestrians in the scenes where kinds of vehicles and people are mixed up in road environments, 3) Detection of traffic lanes, stop lines, traffic signs, and traffic signals in road environments through image recognition, 4) Real-time detection of self-position or posture / construction of 3D models of environment by the visual-inertial navigation SLAM (Simultaneous Localization and Mapping) System developed in-house, 5) Detection of human body posture, 6) Liveness check to recognize if the shot object is a human face, a photograph, or a deceptive mask, and 7) “SenseParrots”, which is an AI learning framework originally developed to assist large-scale deep learning. Solutions that utilize these technologies includes the following; 1) The automobile field such as autonomous driving, Advanced Driver Assistance System (ADAS), and development of vehicle electronic technologies; 2) Support for security measures, traffic detection / tracking in smart cities, and smart retail, and 3) Fields of FA and robotics.

Among them, technology development of basic algorithms of AI and its platforms, for example, are performed in the Chinese side (SenseTime Group). In marketing them in the Japanese market, they are customized by SenseTime Japan as needed to guarantee the required quality and provided as solutions, according to Ms. Chikayo Nishioka, PR Leader, Corporate Planning Div., Strategic Planning Dept. of the Company.

Technologies of SenseTime UC-win/Road DS utilized in Demonstration
of SenseDrive DMS

 Aiming at Developing DMS by Original Approach

MHI Sensing Dept., Automotive Business Unit we interviewed aims at developing and productization of a variety of on-board applications using AI algorithm, while another group in the same Unit is specifically addressing ADAS.

One of those that The Unit currently places emphasis on most is development of DMS. This is for checking and estimating the driver’s physical conditions in order to secure safety and comfort of the driver. In particular, at the level 3 of automated driving, it is so stipulated that “the driver can leave driving operation to the system, e.g. by letting go of the steering wheel” under a given condition. There, it is supposed to be required to have HMI with a “function to monitor the driver’s condition capable of taking over driving operation from the system, and to give warning” when needed. In other words, the function similar to DMS is required.

Concerning this, the Company utilized face recognition / authentication technology they have fostered through technology and product development within SenseTime Group. At present, they are promoting development of DMS capable of not only detecting the driver’s dangerous behavior or conditions that interfere with safe driving in the on-board environment, but also realizing the following functions with high accuracy and high speed; 1) personal authentication for security, 2) gesture recognition for HMI between the vehicle and the driver, and 3) attribute recognition and line of sight recognition of the driver. At the same time, they are also making efforts to create opportunity to appeal their technologies through exhibiting them at multiple events.

Action Interface System
Bus passenger management system that applies
driver monitoring system
Face Entry System

 Introduced UC-win/Road in Starting to Develop DMS

The Unit’s start of full-scale development of DMS dates back to about 3 years ago, when Mr. Sato, Manager was invited to the Company.
“(At that time, in the Unit) we were performing sensing while doing image processing in variable ways, using algorithm of AI for studies related with automated driving.”

On the other hand, the Company owns excellent materials such as of image recognition technologies that SenseTime Group has originally developed and accumulated. Since these technologies have already been commercialized in the Chinese market, they tried to find a way to spread their use also in Japan. As they have attracted attention from their potential customers through exhibition at events etc., he came upon the idea of expanding objects of developing applications including DMS, using face authentication technology that the Company was good at.

So, they consulted with the concerned people, saying “Some tool for simulation will be necessary to develop DMS. What kind of things shall we use for what purpose?” When he was working at his previous post in the automobile industry, he had been using UC-win/Road for simulation for sensing related with ADAS-related diverse devices for many years. He explained his experiences of reproducing real-vehicle environment, repeating evaluation experiments, leading them to development. Then it was decided to adopt a similar approach.

Following this, the Company introduced UC-win/Road in the late 2018. They started to build up a mechanism to detect phenomena such as sleepiness or looking aside by DMS while the test subject plays traveling simulation by operating DS that reflects VR of traveling environment close to the actual situation created using by UC-win/Road.

At that time, since they were also preparing for exhibition of “Automotive World” (Tokyo) to be held in Jan. in the following year of 2019, they met the demand of “being in time for the exhibition in any case” (Ms. Nishioka). It was aimed to systemize it not only to assist development but also to use for exhibition. Since then, this type of DS has been actively used for multiple events the Company participated. In addition to those for permanent exhibition in the showroom to be established in the Tokyo Office, 2 units of DS were prepared and exhibited in Automotive World in January 2020.

Exhibition in ”Automotive Engineering Exposition 2019 Nagoya”

 Evolving Needs, Further Functional Extension UC-win/Road Expected

If there had been no UC-win/Road DS, we would have to perform travel experiment using an actual vehicle. Therefore, many problems were assumed in consideration of time, manpower, and cost etc. to spend there. In this sense, the DS is not only easy to use but also easy to rewrite data and switch day and night or weather. It is also capable of reproducing a variety of experiment environments set to see the driver’s condition. Mr. Sato mentions advantages of using UC-win/Road as a simulator for development, in particular, in that UC-win/Road can be used not only as a single item but also in connection with other diverse kinds of things.

Paying attention to future function extension of UC-win/Road inconsideration of new HMI-related research and development, he has been mostly checking the sensible part of the drivers so far with DMS. He expects that in the future, using the technology of UC-win/Road, not only vehicles and roads but also the complicated interior of vehicles and even facial expressions of drivers will be represented, enabling to reflect the change in driving environments more realistically.

“(Partly because customers’ needs are greatly evolving,) it will be very favorable if it will become possible to represent diverse elements from the vehicle’s exterior to interior seamlessly.”

(Written by Takashi Ikeno)
(Up&Coming '20 Summer issue)