WO2022211140A1 - Edge-deep-learning-based vehicle safe driving system using vehicle driving state information - Google Patents

Edge-deep-learning-based vehicle safe driving system using vehicle driving state information Download PDF

Info

Publication number
WO2022211140A1
WO2022211140A1 PCT/KR2021/003886 KR2021003886W WO2022211140A1 WO 2022211140 A1 WO2022211140 A1 WO 2022211140A1 KR 2021003886 W KR2021003886 W KR 2021003886W WO 2022211140 A1 WO2022211140 A1 WO 2022211140A1
Authority
WO
WIPO (PCT)
Prior art keywords
vehicle
neural network
network model
state information
training
Prior art date
Application number
PCT/KR2021/003886
Other languages
French (fr)
Korean (ko)
Inventor
김진복
이인섭
이윤희
홍현철
강지민
Original Assignee
주식회사 리트빅
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 주식회사 리트빅 filed Critical 주식회사 리트빅
Publication of WO2022211140A1 publication Critical patent/WO2022211140A1/en

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/02—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to ambient conditions
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the present invention generally relates to a vehicle safe driving system for technically assisting safe driving of a vehicle.
  • the present invention forms a neural network model by deep learning training on vehicle driving state information, including driver state information, vehicle environment state information, and vehicle driving state information, in an edge deep learning structure, and identifies safe driving threat factors through this neural network model It relates to an edge deep learning-based safe vehicle driving system that can increase the effectiveness of safe driving by removing the cause that affects the driver during vehicle driving and causes the threat of safe driving.
  • a Driver Monitoring System is being used to technically assist the safe driving of a vehicle. If a surveillance camera is installed in front of the driver inside the vehicle, and the driver's condition is monitored from the captured image of the camera, when the driver's actions that threaten safe driving, such as straying forward, drowsiness, drowsiness, smoking, and phone calls, are recognized. A warning message is provided through an alarm, etc.
  • the driver's face region and facial feature points are extracted from the camera-shot image, and the driver's gaze direction, pupil state, drowsiness, smoking, etc. are determined based on these feature points.
  • the identification method is mainly used. Accordingly, in the existing DMS, attention has been focused on extracting feature points from a camera-captured image and identifying the driver's state.
  • the conventional approach only deals with whether there is a driver's condition (phenomenon) that threatens safe driving, and has a disadvantage in that it cannot eliminate the cause of the condition threatening safe driving.
  • phenomenon a driver's condition
  • drowsy driving conventionally, only warning the driver of drowsy driving is limited, but there is a limitation in that the cause of the drowsy driving cannot be eliminated.
  • Such a conventional approach does not sufficiently achieve the objective of technically assisting the safe driving of a vehicle.
  • an object of the present invention is to form a neural network model by deep learning training on vehicle driving state information, including driver state information, vehicle environment state information, and vehicle driving state information, in an edge deep learning structure, and through this neural network model, safety driving threat factors It is to provide an edge deep learning-based safe driving system that can increase the effectiveness of safe driving by removing the cause of the threat of safe driving by affecting the driver during vehicle driving.
  • the present invention provides an edge deep learning-based vehicle safe driving system for assisting safe driving by identifying and notifying the driver of safe driving threat factors during vehicle driving using vehicle driving state information.
  • the edge deep learning-based vehicle safe driving system is mounted on an individual vehicle, acquires vehicle driving state information for the vehicle in real time, and includes an edge computing unit 130 with a built-in neural network model, a vehicle DMS device 100 that inputs the real-time acquired vehicle driving state information to the neural network model of the edge computing unit 130 to identify a safe driving threat factor while driving the vehicle and informs the vehicle driver;
  • vehicle driving state information and neural network model data from the vehicle DMS device 100 mounted on an individual vehicle, deep learning training the neural network model of the vehicle DMS device 100 based on the vehicle driving state information to generate an updated neural network model Cloud computer 200 for providing an updated neural network model to the edge computing unit 130 of the vehicle DMS device 100 later;
  • the local server 300 is installed and operated for each region and collects and relays data transmission/reception between the vehicle DMS device 100 and the cloud computer 200; and is configured to include.
  • the edge computing unit 130 configures a first training dataset from the vehicle driving state information obtained in real time, and performs a first training part requiring a relatively small amount of computation among deep learning training for the neural network model to model the neural network model.
  • the cloud computer 200 configures the second training dataset from the vehicle driving state information provided from the vehicle DMS device 100 and requires a relatively large amount of computation during deep learning training for the neural network model. It is configured to generate an updated neural network model by performing two training parts.
  • the edge computing unit 130 configures a first training dataset from the vehicle driving state information acquired in real time during a preset first time period and requires a relatively small amount of computation during deep learning training for the neural network model.
  • the vehicle is configured to perform 1 training part
  • the cloud computer 200 is provided from the vehicle DMS device 100 for a preset second time period (provided that the second time period is set longer than the first time period) It may be configured to configure a second training dataset from driving state information and perform a second training part requiring a relatively large amount of computation during deep learning training for a neural network model.
  • the edge computing unit 130 is configured to tune the neural network model by performing training on the connection weights between nodes in a state where the layer structure of the neural network model is fixed, and the cloud computer 200 includes the layer structure of the neural network model and It can be configured to generate an updated neural network model by performing training on all of the inter-node connection weights.
  • the edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to one or more of the input layer and the output layer of the neural network model, and the cloud computer 200 is located in the hidden layer of the neural network model. It may be configured to generate an updated neural network model by performing training on the relevant layer.
  • the edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to one or more of the input layer and the output layer of the neural network model, and the cloud computer 200 provides for the entire neural network model. It may be configured to perform training to generate an updated neural network model.
  • the vehicle DMS device 100 includes at least two of a driver monitoring camera 111 , an in-vehicle environment sensor 112 , and a vehicle driving condition sensor 113 to provide driver status information and vehicle environment status for the vehicle.
  • a state information acquisition unit 110 for acquiring vehicle driving state information including two or more of information and vehicle driving state information in real time;
  • a data processing unit 120 that inputs the real-time obtained vehicle driving state information to the neural network model, outputs a safe driving threat factor while driving the corresponding vehicle, and informs the vehicle driver;
  • a neural network model is built-in, and vehicle driving state information is received from the data processing unit 120, and safe driving threat factors are derived by the neural network model, and output to the data processing unit 120.
  • an edge computing unit 130 for tuning the neural network model by performing a first training part of the deep learning training for the model; a data storage unit 140 for temporarily storing the vehicle driving state information obtained in real time by the state information obtaining unit 110 and the data processing result by the data processing unit 120 ;
  • the data temporarily stored in the data storage unit 140 and the data of the neural network model held by the edge computing unit 130 are transmitted to the cloud computer 200, and an updated neural network model is provided from the cloud computer 200 for edge computing.
  • the vehicle communication unit 150 for transferring to the unit 130; It may be configured to include a;
  • the computer program according to the present invention is stored in the medium to execute the operating method of the vehicle safe driving system based on edge deep learning using the vehicle driving state information as described above in the computer.
  • the present invention by using a neural network model that has undergone deep learning training in the driver's usual vehicle driving environment, it is possible to fundamentally remove the cause that affects the driver during vehicle driving and causes a threat to safe driving of the vehicle, thereby improving the effect of safe driving of the vehicle. There is an advantage that can be improved compared to the prior art.
  • the neural network model learning and application for the vehicle driving situation can be effectively achieved at a low cost through the deep learning structure subdivided into a vehicle application device, an edge computer, and a cloud computer.
  • FIG. 1 is a view showing the overall configuration of a vehicle safe driving system based on edge deep learning according to the present invention.
  • FIG. 2 is a diagram showing a first embodiment of an edge deep learning configuration in the present invention
  • FIG. 3 is a diagram showing a second embodiment of an edge deep learning configuration in the present invention.
  • FIG. 4 is a diagram showing a third embodiment of an edge deep learning configuration in the present invention.
  • FIG. 5 is a block diagram showing the internal configuration of the vehicle DMS device in the present invention.
  • FIG. 1 is a view showing the overall configuration of a vehicle safe driving system based on edge deep learning according to the present invention.
  • the present invention is an edge deep learning-based vehicle safe driving system that assists vehicle safe driving by inputting vehicle driving state information into a deep learning trained neural network model to eliminate the cause of safe driving threat factors during vehicle driving.
  • the neural network model continuously learns the driver's usual driving pattern and driving environment.
  • the edge deep learning-based vehicle safe driving system is a vehicle DMS device 100 mounted on an individual vehicle, and a cloud computer 200 that performs deep learning training on a neural network model of an individual vehicle. , and a local server 300 in charge of relaying data transmission/reception between them.
  • the vehicle DMS device 100 is mounted on an individual vehicle, acquires state information related to the vehicle in real time, and inputs the state information into a neural network model to derive whether there is a safety driving threat factor.
  • the vehicle DMS device 100 acquires vehicle driving state information (driver state information, vehicle environment state information, vehicle driving state information) in real time through a camera and a sensor, and the edge computing unit 130 with a built-in neural network model ) is provided.
  • the vehicle DMS device 100 inputs the real-time acquired vehicle driving state information to the neural network model to identify the safe driving threat factors while driving the vehicle and appropriately notify the vehicle driver (eg, recommend rest, induce ventilation, etc.).
  • the cloud computer 200 is located in the core of the cloud and is a device that performs a neural network update by processing a deep learning training operation according to a complex algorithm for the vehicle DMS device 100 .
  • the cloud computer 200 receives vehicle driving state information and neural network model data from the vehicle DMS device 100, and a complex deep learning training neural network model of the vehicle DMS device 100 based on the vehicle driving state information. After generating an updated neural network model by performing an algorithm operation, it is provided to the edge computing unit 130 of the vehicle DMS device 100 . Through this update, the neural network model of the vehicle DMS device 100 operates more accurately.
  • the local server 300 is a device that is installed and operated for each region and collects and relays data transmission/reception between the vehicle DMS device 100 and the cloud computer 200 of the cloud core in the region. Meanwhile, as shown in FIG. 5 , it is also possible to configure the present invention so that the vehicle DMS device 100 and the cloud computer 200 directly transmit and receive data without going through the relay operation of the local server 300 .
  • a neural network model is trained by deep learning on the usual vehicle driving environment, that is, the driver's usual driving pattern and the in-vehicle environmental state or vehicle driving state. Since each driver has different vehicle conditions and driving habits, it is desirable to gradually learn from the neural network model rather than uniformly programming in advance.
  • the vehicle driving state can be distinguished by itself into a normal state, an abnormal state, and a state that is advancing to an abnormal state.
  • Deep learning consists of a training process and an application process.
  • the training process is the process of learning the neural network model through the training dataset
  • the application process is the process of using the neural network model in real situations.
  • the training process has a large amount of computation, and the application process has a small amount of computation.
  • the training process and the application process are separated and configured in consideration of the vehicle driving environment.
  • the training process is in charge of the cloud computer 200 and the edge computing unit 130 , and other components of the vehicle DMS device 100 are in charge of the application process.
  • the vehicle DMS device 100 performs a process of acquiring various state information in the vehicle, identifying a threat factor for safe driving through a neural network model, and notifying the driver.
  • the cloud computer 200 and the edge computing unit 130 perform training of the neural network model using various state information in the vehicle. Through this separation, it is possible to reduce the computational burden of the vehicle electronic control unit (ECU), thereby ultimately lowering the price.
  • ECU vehicle electronic control unit
  • a neural network model is trained to identify the current driver's state based on driver state information. That is, the camera image recognizes feature points that can be recognized as a person, such as eyes, nose, and mouth, and sets a criterion for determining when the pupil, mouth, and face have a certain behavior pattern.
  • the neural network model is trained to identify the driving environment inside the vehicle based on the vehicle environment state information. That is, based on the pattern that the in-vehicle environment (e.g., carbon dioxide concentration, internal temperature) changes depending on the number of passengers, respiration rate, and whether or not the ventilation system operates, it is determined whether the vehicle maintains a normal state, deteriorates rapidly, or deteriorates gently. Set standards.
  • the in-vehicle environment e.g., carbon dioxide concentration, internal temperature
  • the neural network model is trained to identify what kind of situation the vehicle is driving and whether it is normal or not. In other words, depending on whether the vehicle speed repeats going to and fro, driving at low speed, driving at high speed, repeating low speed and high speed, or repeating rapid acceleration and deceleration, traffic jams, city driving, highway driving, drowsy driving, reckless driving, etc. Set the criteria for judging
  • the edge computing unit 130 of the vehicle DMS device 100 By providing such a neural network model in the edge computing unit 130 of the vehicle DMS device 100, it is configured to identify the safe driving threat factors during vehicle driving according to vehicle driving state information.
  • the edge computing unit 130 and the cloud computer 200 iteratively additional learning based on the vehicle driving state information (driver state information, vehicle environment state information, vehicle driving state information) collected by the vehicle DMS device 100 A configuration that improves (updates) the neural network model was adopted.
  • the edge computing unit 130 provided in the vehicle DMS device 100 is the same concept as the human neural network circuit through the neural network processor unit (NPU) in the SoC chipset installed in the ECU device of the vehicle. It can be implemented as a device implemented to quickly process the number of cases.
  • the cloud computer 200 receives various data uploaded from the vehicle DMS device 100 and analyzes and learns a relatively long-time driver behavior pattern and a long-term vehicle environment to improve the neural network model, and the neural network model learned through improvement is provided to the edge computing unit 130 of the vehicle DMS device 100 of an individual vehicle to improve analysis performance.
  • the deep learning training process for the neural network model is separated and the edge deep learning structure divided into the cloud computer 200 and the edge computing unit 130 is adopted.
  • the process is performed by the cloud computer 200 of the cloud core.
  • a model capable of separation training is applied to the neural network model.
  • the edge computing unit 130 has lower computational performance than the cloud computer 200, but a light training process is applied, so that deep learning training is performed immediately according to vehicle driving state information so that it can be quickly applied to the driver.
  • the cloud computer 200 is a high-performance device and is configured to apply a large-scale update required. According to such an edge deep learning structure, training of a neural network model can be achieved quickly, and the training results can be quickly utilized in a vehicle.
  • the edge computing unit 130 tunes the neural network model by configuring a first training dataset from the real-time acquired vehicle driving state information and performing the first training part requiring a relatively small amount of computation during deep learning training for the neural network model. configure to do
  • the cloud computer 200 configures the second training dataset from the vehicle driving state information provided from the vehicle DMS device 100 and performs the second training part that requires a relatively large amount of computation during deep learning training for the neural network model. Configure to create an updated neural network model by performing
  • the edge computing unit 130 is common in that the cloud computer 200 performs deep learning training based on the vehicle driving state information (driver state information, vehicle environment state information, vehicle driving state information), but the training The training dataset is preferably set up differently. In this case, a smaller-scale training dataset is configured for the edge computing unit 130 .
  • the edge computing unit 130 performs a relatively small change (tuning) on the neural network model, while the cloud computer 200 is configured to perform a relatively large change (update) on the neural network model.
  • the edge computing unit 130 configures a deep learning training policy that requires a relatively small amount of computation compared to the cloud computer 200 .
  • the edge computing unit 130 configures a first training dataset from the vehicle driving state information acquired in real time during a preset first time period (eg, 1 hour), and performs a relative operation during deep learning training for the neural network model. It may be configured to perform the first training part requiring a small amount of computation.
  • the cloud computer 200 receives the vehicle driving state provided from the vehicle DMS device 100 for a preset second time period (however, the second time period is set longer than the first time period; for example, 7 days) It may be configured to construct a second training dataset from information and perform a second training part that requires a relatively large amount of computation during deep learning training for a neural network model.
  • FIGS. 2 to 4 three embodiments in which a deep learning training policy that requires a relatively different amount of computation for the edge computing unit 130 and the cloud computer 200 is configured is shown in FIGS. 2 to 4 .
  • the edge computing unit 130 performs a relatively small change (tuning) on the neural network model
  • the cloud computer 200 performs a relatively large change (update) on the neural network model.
  • FIG. 2 is a diagram illustrating a first embodiment of an edge deep learning configuration by the edge computing unit 130 and the cloud computer 200 in the present invention.
  • the edge computing unit 130 is configured to tune the neural network model by performing training on connection weights between nodes in a state where the layer structure of the neural network model is fixed.
  • the cloud computer 200 is configured to generate an updated neural network model by performing training on both the layer structure (node configuration) of the neural network model and the connection weight between nodes.
  • FIG. 3 is a diagram illustrating a second embodiment of an edge deep learning configuration by the edge computing unit 130 and the cloud computer 200 in the present invention.
  • the edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to an input layer and an output layer of the neural network model. Training may be performed on both the layer related to the input layer (bottom left in FIG. 3) and the layer related to the output layer (bottom right in FIG. 3), or only one of them may be trained. For the first and last layers of the hidden layer directly connected to the input layer or the output layer, the edge computing unit 130 may be configured to change the layer structure (node configuration) or may be configured not to change. At this time, the cloud computer 200 is configured to generate an updated neural network model by performing deep learning training on a plurality of layers related to hidden layers of the neural network model.
  • FIG. 4 is a diagram showing a third embodiment of the edge deep learning configuration by the edge computing unit 130 and the cloud computer 200 in the present invention.
  • the edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to an input layer and an output layer of the neural network model. Training may be performed on both the layer related to the input layer (bottom left in FIG. 3) and the layer related to the output layer (bottom right in FIG. 3), or only one of them may be trained.
  • the cloud computer 200 is configured to generate an updated neural network model by performing training on the overall structure of the neural network.
  • the edge computing unit 130 is configured to perform training related to the input layer and the output layer of the neural network model. Through this, it is possible to efficiently operate the edge computing unit 130 individually mounted in a plurality of vehicles, and to reduce the cost of the edge computing unit 130 .
  • the vehicle DMS device 100 includes a state information acquisition unit 110 , a data processing unit 120 , an edge computing unit 130 , a data storage unit 140 , a vehicle communication unit 150 , and a driver alert unit ( 160) is included.
  • the state information acquisition unit 110 is a component that acquires vehicle driving state information for a corresponding vehicle in real time.
  • the state information acquisition unit 110 includes a driver monitoring camera 111 for acquiring driver state information, an in-vehicle environmental sensor 112 for acquiring vehicle environment state information, and a vehicle for acquiring vehicle driving state information.
  • Two or more of the driving state sensors 113 are provided.
  • the present invention not only simply recognizes the driver's face and features, but also inputs various sensor data (vehicle motion information, vehicle interior information) into a neural network model to analyze the cause of the driver's state.
  • the driver monitoring camera 111 preferably uses infrared lighting and an infrared camera in order to be less affected by lighting, and recognizes the driver's condition through the camera captured image. Since the driver state recognition technology has been implemented in the prior art and is not the core of the present invention, a detailed description thereof will be omitted.
  • the vehicle internal environment sensor 112 acquires information such as carbon dioxide concentration and internal temperature inside the vehicle through an air quality sensor and a temperature sensor, and the vehicle driving state sensor 113 includes the vehicle driving speed, driving time, window opening, etc. Acquire information about the driving state of the vehicle.
  • the data processing unit 120 is a component that inputs the vehicle driving state information obtained in real time by the state information acquisition unit 110 into the neural network model, outputs the safe driving threat factors while driving the corresponding vehicle, and informs the vehicle driver.
  • the edge computing unit 130 is equipped with a neural network model trained by deep learning with vehicle driving state information, and the data processing unit 120 receives the vehicle driving state information provided by the state information acquisition unit 110 to the edge computing unit 130 . By inputting it into the neural network model of In addition, the data processing unit 120 provides the vehicle driving state information and the data processing result to the data storage unit 140 to be stored.
  • the edge computing unit 130 is a component that embeds a neural network model in the form of, for example, a computer program and performs various data processing related to the neural network model.
  • the edge computing unit 130 may be implemented as hardware of a neural network processor unit (NPU) in a SoC (System-on-Chip) chipset of a vehicle ECU device, and the neural network model may be implemented in the form of a hardware circuit of this NPU, It may be implemented in the form of a computer program running on this NPU.
  • NPU neural network processor unit
  • SoC System-on-Chip
  • the edge computing unit 130 receives vehicle driving state information from the data processing unit 120 , derives a safe driving threat factor during driving of the corresponding vehicle by a neural network model, and outputs it to the data processing unit 120 .
  • the edge computing unit 130 tunes the neural network model by performing a first training part that requires a relatively small amount of computation among deep learning training for the neural network model based on vehicle driving state information.
  • the edge computing unit 130 may be implemented as if the vehicle driving state information is directly transmitted from the state information acquisition unit 110 .
  • the data processing unit 120 and the edge computing unit 130 are installed together in the vehicle ECU, and the vehicle driving state information input from the state information obtaining unit 110 to the vehicle ECU is transferred to the data processing unit. It is transmitted to 120 and the edge computing unit 130 . Accordingly, the edge computing unit 130 may be modeled as receiving the vehicle driving state information from the data processing unit 120 .
  • the edge computing unit 130 normally performs deep learning training on the neural network model using the vehicle driving state information input from the data processing unit 120, while applying the vehicle driving state information to the neural network model to ensure safety during vehicle driving.
  • Driving threat factors are derived and provided to the data processing unit 120 , and data requiring additional learning and analysis results are transmitted to the data storage unit 140 according to preset criteria.
  • the data storage unit 140 is a component for temporarily storing the vehicle driving state information acquired in real time by the state information acquisition unit 110 and the data processing result by the data processing unit 120 .
  • a system mounted on a vehicle does not guarantee network access due to driving conditions (eg, remote areas, tunnels, etc.), so the data storage unit 140 temporarily stores sensor input and data processing results, and when connected to a network, the vehicle It serves to transmit data to the outside through the communication unit 150 .
  • the vehicle communication unit 150 provides a communication channel so that the vehicle DMS device 100 can transmit and receive data with the outside. That is, the vehicle communication unit 150 transmits the data temporarily stored in the data storage unit 140 and the neural network model data held by the edge computing unit 130 to the cloud computer 200 and from the cloud computer 200 . It is a component that receives the updated neural network model and transmits it to the edge computing unit 130 .
  • the vehicle communication unit 150 may be configured to be directly connected to the cloud computer 200 through a broadband network, or may be configured to be connected to the local server 300 via a local area network.
  • the vehicle communication unit 150 is preferably a wireless network, but does not exclude a wired network.
  • the driver warning unit 160 is a component that outputs safety driving threat factors to the vehicle driver in a recognizable way such as visual, auditory, tactile, and olfactory under the control of the data processing unit 120 .
  • the present invention can be implemented in the form of computer-readable codes on a computer-readable non-volatile recording medium.
  • Various types of storage devices exist as such non-volatile recording media.
  • a form in which it can be implemented and executed can also be implemented.
  • the present invention may be implemented in the form of a computer program stored in a medium to execute a specific procedure in combination with hardware.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Mathematical Physics (AREA)
  • Theoretical Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Traffic Control Systems (AREA)

Abstract

The present invention relates to an edge-deep-learning-based vehicle safe driving system which forms a neural network model through deep learning training using driver state information, vehicle environment state information, and vehicle driving state information including vehicle-traveling-state information in an edge deep learning structure, and which identifies safe driving threat factors through the neural network model, so as to influence a driver during vehicle traveling, and thus can remove the causes of a situation in which safe driving of the vehicle is threatened and increase a vehicle safe driving effect. According to the present invention, causes triggering a vehicle safe driving threat situation can fundamentally be removed by influencing a driver. In addition, neural network model training and application for vehicle driving situations can be effectively achieved at low cost through a deep learning structure.

Description

차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템Edge deep learning-based vehicle safety driving system using vehicle driving status information
본 발명은 일반적으로 차량 안전운전을 기술적으로 보조하기 위한 차량 안전운전 시스템에 관한 것이다.The present invention generally relates to a vehicle safe driving system for technically assisting safe driving of a vehicle.
특히, 본 발명은 에지 딥러닝 구조에서 운전자 상태정보와 차량환경 상태정보와 차량주행 상태정보를 비롯한 차량운전 상태정보를 딥러닝 훈련시켜 신경망 모델을 형성하고 이 신경망 모델을 통해 안전운전 위협요인을 식별해내도록 구성함으로써 차량 주행 중에 운전자에게 영향을 미쳐 차량 안전운전 위협 상황을 유발하는 원인을 제거하고 차량 안전운전 효과를 높일 수 있는 에지 딥러닝 기반의 차량 안전운전 시스템에 관한 것이다.In particular, the present invention forms a neural network model by deep learning training on vehicle driving state information, including driver state information, vehicle environment state information, and vehicle driving state information, in an edge deep learning structure, and identifies safe driving threat factors through this neural network model It relates to an edge deep learning-based safe vehicle driving system that can increase the effectiveness of safe driving by removing the cause that affects the driver during vehicle driving and causes the threat of safe driving.
차량 안전운전을 기술적으로 보조하기 위해 DMS(Driver Monitoring System)가 사용되고 있다. 차량 실내의 운전자 전면에 감시용 카메라를 설치하고, 이 카메라의 촬영 영상에서 운전자 상태를 모니터링하여 안전운전을 위협하는 운전자의 행위, 예컨대 전방 미주시, 졸음, 멍한 상태, 흡연, 통화 등을 인식하면 알람 등을 통해 경고 메세지를 제공한다.A Driver Monitoring System (DMS) is being used to technically assist the safe driving of a vehicle. If a surveillance camera is installed in front of the driver inside the vehicle, and the driver's condition is monitored from the captured image of the camera, when the driver's actions that threaten safe driving, such as straying forward, drowsiness, drowsiness, smoking, and phone calls, are recognized. A warning message is provided through an alarm, etc.
운전자의 상태를 인식하기 위해서는 카메라 촬영 영상에서 운전자의 얼굴 영역과 그 얼굴의 특징점(눈, 코, 입)을 추출한 후에 이들 특징점에 기초하여 운전자의 시선 방향, 눈동자 상태, 졸음 여부, 흡연 여부 등을 식별하는 방식이 주로 사용된다. 그에 따라, 기존의 DMS에서는 카메라 촬영 영상에서 특징점을 추출하고 운전자 상태를 식별하는 것에 관심이 집중되었다. In order to recognize the driver's condition, the driver's face region and facial feature points (eyes, nose, and mouth) are extracted from the camera-shot image, and the driver's gaze direction, pupil state, drowsiness, smoking, etc. are determined based on these feature points. The identification method is mainly used. Accordingly, in the existing DMS, attention has been focused on extracting feature points from a camera-captured image and identifying the driver's state.
그런데, 종래의 접근 방식은 안전운전을 위협하는 운전자의 상태(현상)가 있는지 여부만 다루고 있을 뿐, 그 안전운전을 위협하는 상태를 유발한 원인을 제거하지 못하는 단점이 있다. 예를 들어 졸음운전이 식별된 경우에, 종래에는 운전자에게 졸음운전을 경고하는데 그칠 뿐, 그 졸음운전을 유발한 원인을 제거하지는 못하는 한계가 있다. 이러한 종래의 접근 방식으로는 차량 안전운전을 기술적으로 보조하는 목적을 충분히 달성하지 못한다.However, the conventional approach only deals with whether there is a driver's condition (phenomenon) that threatens safe driving, and has a disadvantage in that it cannot eliminate the cause of the condition threatening safe driving. For example, when drowsy driving is identified, conventionally, only warning the driver of drowsy driving is limited, but there is a limitation in that the cause of the drowsy driving cannot be eliminated. Such a conventional approach does not sufficiently achieve the objective of technically assisting the safe driving of a vehicle.
본 발명의 목적은 일반적으로 차량 안전운전을 기술적으로 보조하기 위한 차량 안전운전 시스템을 제공하는 것이다. SUMMARY OF THE INVENTION It is an object of the present invention to provide a vehicle safe driving system for technically assisting vehicle safe driving in general.
특히, 본 발명의 목적은 에지 딥러닝 구조에서 운전자 상태정보와 차량환경 상태정보와 차량주행 상태정보를 비롯한 차량운전 상태정보를 딥러닝 훈련시켜 신경망 모델을 형성하고 이 신경망 모델을 통해 안전운전 위협요인을 식별해내도록 구성함으로써 차량 주행 중에 운전자에게 영향을 미쳐 차량 안전운전 위협 상황을 유발하는 원인을 제거하고 차량 안전운전 효과를 높일 수 있는 에지 딥러닝 기반의 차량 안전운전 시스템을 제공하는 것이다. In particular, an object of the present invention is to form a neural network model by deep learning training on vehicle driving state information, including driver state information, vehicle environment state information, and vehicle driving state information, in an edge deep learning structure, and through this neural network model, safety driving threat factors It is to provide an edge deep learning-based safe driving system that can increase the effectiveness of safe driving by removing the cause of the threat of safe driving by affecting the driver during vehicle driving.
한편, 본 발명의 해결 과제는 이들 사항에 제한되지 않으며 본 명세서의 기재로부터 다른 해결 과제가 이해될 수 있다.On the other hand, the problem to be solved of the present invention is not limited to these matters, and other problems to be solved can be understood from the description of the present specification.
본 발명은 차량운전 상태정보를 이용하여 차량 주행 중의 안전운전 위협요인을 식별하여 운전자에게 알림으로써 차량 안전운전은 보조하기 위한 에지 딥러닝 기반의 차량 안전운전 시스템을 제공한다.The present invention provides an edge deep learning-based vehicle safe driving system for assisting safe driving by identifying and notifying the driver of safe driving threat factors during vehicle driving using vehicle driving state information.
본 발명에 따른 에지 딥러닝 기반의 차량 안전운전 시스템은, 개별 차량에 장착되고, 해당 차량에 대해 차량운전 상태정보를 실시간으로 획득하고, 신경망 모델을 내장한 엣지 컴퓨팅부(130)를 구비하고, 그 실시간 획득된 차량운전 상태정보를 엣지 컴퓨팅부(130)의 신경망 모델에 입력하여 해당 차량 주행 중의 안전운전 위협요인을 식별하고 차량 운전자에게 알리는 차량 DMS 장치(100); 개별 차량에 장착된 차량 DMS 장치(100)로부터 차량운전 상태정보와 신경망 모델 데이터를 제공받아 차량운전 상태정보에 기초하여 차량 DMS 장치(100)의 신경망 모델을 딥러닝 훈련시켜 업데이트 신경망 모델을 생성한 후에 차량 DMS 장치(100)의 엣지 컴퓨팅부(130)로 업데이트 신경망 모델을 제공하는 클라우드 컴퓨터(200); 지역 별로 설치 운영되어 차량 DMS 장치(100)와 클라우드 컴퓨터(200) 사이의 데이터 송수신을 수집 중계하는 로컬 서버(300);를 포함하여 구성된다.The edge deep learning-based vehicle safe driving system according to the present invention is mounted on an individual vehicle, acquires vehicle driving state information for the vehicle in real time, and includes an edge computing unit 130 with a built-in neural network model, a vehicle DMS device 100 that inputs the real-time acquired vehicle driving state information to the neural network model of the edge computing unit 130 to identify a safe driving threat factor while driving the vehicle and informs the vehicle driver; By receiving vehicle driving state information and neural network model data from the vehicle DMS device 100 mounted on an individual vehicle, deep learning training the neural network model of the vehicle DMS device 100 based on the vehicle driving state information to generate an updated neural network model Cloud computer 200 for providing an updated neural network model to the edge computing unit 130 of the vehicle DMS device 100 later; The local server 300 is installed and operated for each region and collects and relays data transmission/reception between the vehicle DMS device 100 and the cloud computer 200; and is configured to include.
이때, 엣지 컴퓨팅부(130)는 그 실시간 획득하는 차량운전 상태정보로부터 제 1 훈련 데이터셋을 구성하고 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 적은 연산량을 요구하는 제 1 훈련 부분을 수행하여 신경망 모델을 튜닝하도록 구성되고, 클라우드 컴퓨터(200)는 차량 DMS 장치(100)로부터 제공받는 차량운전 상태정보로부터 제 2 훈련 데이터셋을 구성하고 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 많은 연산량을 요구하는 제 2 훈련 부분을 수행하여 업데이트 신경망 모델을 생성하도록 구성된다.At this time, the edge computing unit 130 configures a first training dataset from the vehicle driving state information obtained in real time, and performs a first training part requiring a relatively small amount of computation among deep learning training for the neural network model to model the neural network model. , and the cloud computer 200 configures the second training dataset from the vehicle driving state information provided from the vehicle DMS device 100 and requires a relatively large amount of computation during deep learning training for the neural network model. It is configured to generate an updated neural network model by performing two training parts.
본 발명에서 엣지 컴퓨팅부(130)는 미리 설정된 제 1 시간구간 동안 그 실시간 획득하는 차량운전 상태정보로부터 제 1 훈련 데이터셋을 구성하고 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 적은 연산량을 요구하는 제 1 훈련 부분을 수행하도록 구성되고, 클라우드 컴퓨터(200)는 미리 설정된 제 2 시간구간 동안(단, 제 2 시간구간은 제 1 시간구간보다 더 길게 설정됨) 차량 DMS 장치(100)로부터 제공받는 차량운전 상태정보로부터 제 2 훈련 데이터셋을 구성하고 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 많은 연산량을 요구하는 제 2 훈련 부분을 수행하도록 구성될 수 있다.In the present invention, the edge computing unit 130 configures a first training dataset from the vehicle driving state information acquired in real time during a preset first time period and requires a relatively small amount of computation during deep learning training for the neural network model. The vehicle is configured to perform 1 training part, and the cloud computer 200 is provided from the vehicle DMS device 100 for a preset second time period (provided that the second time period is set longer than the first time period) It may be configured to configure a second training dataset from driving state information and perform a second training part requiring a relatively large amount of computation during deep learning training for a neural network model.
또한 본 발명에서 엣지 컴퓨팅부(130)는 신경망 모델의 레이어 구조를 고정한 상태에서 노드간 연결 가중치에 대한 훈련을 수행하여 신경망 모델을 튜닝하도록 구성되고, 클라우드 컴퓨터(200)는 신경망 모델의 레이어 구조 및 노드간 연결 가중치 모두에 대한 훈련을 수행하여 업데이트 신경망 모델을 생성하도록 구성될 수 있다.In addition, in the present invention, the edge computing unit 130 is configured to tune the neural network model by performing training on the connection weights between nodes in a state where the layer structure of the neural network model is fixed, and the cloud computer 200 includes the layer structure of the neural network model and It can be configured to generate an updated neural network model by performing training on all of the inter-node connection weights.
또한, 본 발명에서 엣지 컴퓨팅부(130)는 신경망 모델의 입력층 및 출력층 중 하나 이상에 관련된 레이어에 대한 훈련을 수행하여 신경망 모델을 튜닝하도록 구성되고, 클라우드 컴퓨터(200)는 신경망 모델의 은닉층에 관련된 레이어에 대한 훈련을 수행하여 업데이트 신경망 모델을 생성하도록 구성될 수 있다.In addition, in the present invention, the edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to one or more of the input layer and the output layer of the neural network model, and the cloud computer 200 is located in the hidden layer of the neural network model. It may be configured to generate an updated neural network model by performing training on the relevant layer.
또한, 본 발명에서 엣지 컴퓨팅부(130)는 신경망 모델의 입력층 및 출력층 중 하나 이상에 관련된 레이어에 대한 훈련을 수행하여 신경망 모델을 튜닝하도록 구성되고, 클라우드 컴퓨터(200)는 신경망 모델 전체에 대한 훈련을 수행하여 업데이트 신경망 모델을 생성하도록 구성될 수 있다.In addition, in the present invention, the edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to one or more of the input layer and the output layer of the neural network model, and the cloud computer 200 provides for the entire neural network model. It may be configured to perform training to generate an updated neural network model.
본 발명에서 차량 DMS 장치(100)는, 운전자 감시 카메라(111), 차량내부 환경센서(112), 차량주행 상태센서(113) 중 둘 이상을 구비하여 해당 차량에 대해 운전자 상태정보, 차량환경 상태정보, 차량주행 상태정보 중 둘 이상을 포함한 차량운전 상태정보를 실시간으로 획득하는 상태정보 획득부(110); 그 실시간 획득된 차량운전 상태정보를 신경망 모델에 입력하여 해당 차량 주행 중의 안전운전 위협요인을 출력받고 이를 차량 운전자에게 알리는 데이터 처리부(120); 신경망 모델을 내장하고, 데이터 처리부(120)로부터 차량운전 상태정보를 입력받아 신경망 모델에 의해 해당 차량 주행 중의 안전운전 위협요인을 도출하여 데이터 처리부(120)로 출력하고 차량운전 상태정보에 기초하여 신경망 모델에 대한 딥러닝 훈련 중 제 1 훈련 부분을 수행하여 신경망 모델을 튜닝하는 엣지 컴퓨팅부(130); 상태정보 획득부(110)에 의해 실시간 획득되는 차량운전 상태정보 및 데이터 처리부(120)에 의한 데이터 처리 결과를 임시 저장하는 데이터 저장부(140); 데이터 저장부(140)에 임시 저장되어 있는 데이터 및 엣지 컴퓨팅부(130)가 보유하는 신경망 모델의 데이터를 클라우드 컴퓨터(200)에 대해 전달하고 클라우드 컴퓨터(200)로부터 업데이트 신경망 모델을 제공받아 엣지 컴퓨팅부(130)로 전달하는 차량 통신부(150); 데이터 처리부(120)의 제어에 의해 안전운전 위협요인을 차량 운전자에게 출력하는 운전자 경보부(160);를 포함하여 구성될 수 있다.In the present invention, the vehicle DMS device 100 includes at least two of a driver monitoring camera 111 , an in-vehicle environment sensor 112 , and a vehicle driving condition sensor 113 to provide driver status information and vehicle environment status for the vehicle. a state information acquisition unit 110 for acquiring vehicle driving state information including two or more of information and vehicle driving state information in real time; a data processing unit 120 that inputs the real-time obtained vehicle driving state information to the neural network model, outputs a safe driving threat factor while driving the corresponding vehicle, and informs the vehicle driver; A neural network model is built-in, and vehicle driving state information is received from the data processing unit 120, and safe driving threat factors are derived by the neural network model, and output to the data processing unit 120. an edge computing unit 130 for tuning the neural network model by performing a first training part of the deep learning training for the model; a data storage unit 140 for temporarily storing the vehicle driving state information obtained in real time by the state information obtaining unit 110 and the data processing result by the data processing unit 120 ; The data temporarily stored in the data storage unit 140 and the data of the neural network model held by the edge computing unit 130 are transmitted to the cloud computer 200, and an updated neural network model is provided from the cloud computer 200 for edge computing. The vehicle communication unit 150 for transferring to the unit 130; It may be configured to include a;
한편, 본 발명에 따른 컴퓨터프로그램은 컴퓨터에 이상과 같은 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템의 운영 방법을 실행시키기 위하여 매체에 저장된 것이다.On the other hand, the computer program according to the present invention is stored in the medium to execute the operating method of the vehicle safe driving system based on edge deep learning using the vehicle driving state information as described above in the computer.
본 발명에 따르면 운전자의 평소 차량운전 환경에서 딥러닝 훈련을 거친 신경망 모델을 이용함으로써 차량 주행 중에 운전자에게 영향을 미쳐 차량 안전운전 위협 상황을 유발하는 원인을 근원적으로 제거할 수 있어 차량 안전운전 효과를 종래에 비해 높일 수 있는 장점이 있다. According to the present invention, by using a neural network model that has undergone deep learning training in the driver's usual vehicle driving environment, it is possible to fundamentally remove the cause that affects the driver during vehicle driving and causes a threat to safe driving of the vehicle, thereby improving the effect of safe driving of the vehicle. There is an advantage that can be improved compared to the prior art.
또한, 본 발명에 따르면 차량적용 장치, 에지 컴퓨터, 클라우드 컴퓨터로 세분화된 딥러닝 구조를 통해 차량 운전 상황에 대한 신경망 모델 학습 및 적용을 적은 비용으로 효과적으로 달성할 수 있는 장점이 있다.In addition, according to the present invention, there is an advantage that the neural network model learning and application for the vehicle driving situation can be effectively achieved at a low cost through the deep learning structure subdivided into a vehicle application device, an edge computer, and a cloud computer.
도 1은 본 발명에 따른 에지 딥러닝 기반의 차량 안전운전 시스템의 전체 구성을 나타내는 도면.1 is a view showing the overall configuration of a vehicle safe driving system based on edge deep learning according to the present invention.
도 2는 본 발명에서 에지 딥러닝 구성의 제 1 실시예를 나타내는 도면.2 is a diagram showing a first embodiment of an edge deep learning configuration in the present invention;
도 3은 본 발명에서 에지 딥러닝 구성의 제 2 실시예를 나타내는 도면.3 is a diagram showing a second embodiment of an edge deep learning configuration in the present invention;
도 4는 본 발명에서 에지 딥러닝 구성의 제 3 실시예를 나타내는 도면.4 is a diagram showing a third embodiment of an edge deep learning configuration in the present invention.
도 5는 본 발명에서 차량 DMS 장치의 내부 구성을 나타내는 블록도.5 is a block diagram showing the internal configuration of the vehicle DMS device in the present invention.
이하에서는 도면을 참조하여 본 발명을 상세하게 설명한다.Hereinafter, the present invention will be described in detail with reference to the drawings.
도 1은 본 발명에 따른 에지 딥러닝 기반의 차량 안전운전 시스템의 전체 구성을 나타내는 도면이다.1 is a view showing the overall configuration of a vehicle safe driving system based on edge deep learning according to the present invention.
본 발명은 딥러닝 훈련된 신경망 모델에 차량운전 상태정보을 입력하여 차량 주행 중의 안전운전 위협요인의 원인을 제거함으로써 차량 안전운전을 보조하는 에지 딥러닝 기반의 차량 안전운전 시스템이다. The present invention is an edge deep learning-based vehicle safe driving system that assists vehicle safe driving by inputting vehicle driving state information into a deep learning trained neural network model to eliminate the cause of safe driving threat factors during vehicle driving.
예를 들어 운전자가 졸음 운전을 하는 경우에, 단순히 졸음 운전을 경고하는 것뿐만 아니라, 졸음의 원인이 이산화탄소 농도 때문인지, 졸릴 시간에 운전을 하고있기 때문인지, 운전 시간이 너무 길었기 때문인지 분석하여 알려줌으로써 원인까지 제거할 수 있도록 보조한다. 이를 위해, 해당 운전자의 평상시의 운전패턴과 운전 환경을 신경망 모델에 지속적으로 학습시킨다.For example, when a driver is driving drowsy, it not only warns of drowsy driving, but also analyzes whether the cause of drowsiness is due to carbon dioxide concentration, driving at a time when drowsy, or driving for too long It informs you so that you can eliminate the cause as well. To this end, the neural network model continuously learns the driver's usual driving pattern and driving environment.
도 1을 참조하면, 본 발명에 따른 에지 딥러닝 기반의 차량 안전운전 시스템은 개별 차량에 탑재되는 차량 DMS 장치(100), 개별 차량의 신경망 모델에 대한 딥러닝 훈련을 수행하는 클라우드 컴퓨터(200), 그리고 이들 간의 데이터 송수신 중계를 담당하는 로컬 서버(300)를 포함한다.1, the edge deep learning-based vehicle safe driving system according to the present invention is a vehicle DMS device 100 mounted on an individual vehicle, and a cloud computer 200 that performs deep learning training on a neural network model of an individual vehicle. , and a local server 300 in charge of relaying data transmission/reception between them.
먼저, 차량 DMS 장치(100)는 개별 차량에 탑재되어 해당 차량에 관련된 상태 정보를 실시간으로 획득하고 그 상태정보를 신경망 모델에 입력하여 안전운정 위협요인이 있는지 여부를 도출해낸다. 이를 위해, 차량 DMS 장치(100)는 카메라와 센서를 통해 차량운전 상태정보(운전자 상태정보, 차량환경 상태정보, 차량주행 상태정보)를 실시간으로 획득하고, 신경망 모델이 내장된 엣지 컴퓨팅부(130)를 구비한다. 또한, 차량 DMS 장치(100)는 그 실시간 획득된 차량운전 상태정보를 신경망 모델에 입력하여 차량 주행 중의 안전운전 위협요인을 식별하고 차량 운전자에게 적절히 알린다(예: 휴식 권장, 환기 유도 등).First, the vehicle DMS device 100 is mounted on an individual vehicle, acquires state information related to the vehicle in real time, and inputs the state information into a neural network model to derive whether there is a safety driving threat factor. To this end, the vehicle DMS device 100 acquires vehicle driving state information (driver state information, vehicle environment state information, vehicle driving state information) in real time through a camera and a sensor, and the edge computing unit 130 with a built-in neural network model ) is provided. In addition, the vehicle DMS device 100 inputs the real-time acquired vehicle driving state information to the neural network model to identify the safe driving threat factors while driving the vehicle and appropriately notify the vehicle driver (eg, recommend rest, induce ventilation, etc.).
다음으로, 클라우드 컴퓨터(200)는 클라우드의 코어(core)에 위치하여 차량 DMS 장치(100)에 대해 복잡한 알고리즘에 따른 딥러닝 훈련 연산을 처리함으로써 신경망 업데이트를 수행하는 장치이다. 이를 위해, 클라우드 컴퓨터(200)는 차량 DMS 장치(100)로부터 차량운전 상태정보와 신경망 모델 데이터를 제공받으며, 차량운전 상태정보에 기초하여 차량 DMS 장치(100)의 신경망 모델을 딥러닝 훈련하는 복잡한 알고리즘 연산을 수행하여 업데이트 신경망 모델을 생성한 후에 차량 DMS 장치(100)의 엣지 컴퓨팅부(130)로 제공한다. 이러한 업데이트를 통해 차량 DMS 장치(100)의 신경망 모델은 좀더 정확하게 작동하게 된다.Next, the cloud computer 200 is located in the core of the cloud and is a device that performs a neural network update by processing a deep learning training operation according to a complex algorithm for the vehicle DMS device 100 . To this end, the cloud computer 200 receives vehicle driving state information and neural network model data from the vehicle DMS device 100, and a complex deep learning training neural network model of the vehicle DMS device 100 based on the vehicle driving state information. After generating an updated neural network model by performing an algorithm operation, it is provided to the edge computing unit 130 of the vehicle DMS device 100 . Through this update, the neural network model of the vehicle DMS device 100 operates more accurately.
또한, 로컬 서버(300)는 지역 별로 설치 운영되어 해당 지역의 차량 DMS 장치(100)와 클라우드 코어의 클라우드 컴퓨터(200) 사이에서 데이터 송수신을 수집 중계하는 장치이다. 한편, 도 5에 나타낸 바와 같이, 로컬 서버(300)의 중계 동작을 거치지 않고 차량 DMS 장치(100)와 클라우드 컴퓨터(200)가 직접 데이터를 송수신하도록 본 발명을 구성하는 것도 가능하다.In addition, the local server 300 is a device that is installed and operated for each region and collects and relays data transmission/reception between the vehicle DMS device 100 and the cloud computer 200 of the cloud core in the region. Meanwhile, as shown in FIG. 5 , it is also possible to configure the present invention so that the vehicle DMS device 100 and the cloud computer 200 directly transmit and receive data without going through the relay operation of the local server 300 .
본 발명에서는 평소 차량운전 환경, 즉 운전자의 평소 운전패턴과 차량내 환경상태 또는 차량 주행 상태를 신경망 모델에 딥러닝 훈련시킨다. 운전자마다 차량 상태나 운전습관이 제각각이므로 미리 획일적으로 프로그래밍하는 것보다는 신경망 모델에 점차적으로 학습해나가는 것이 바람직하다. 특히, 딥러닝의 경우에는 관리자의 개입 없이도 스스로 카테고리를 분류해가는 능력이 있기 때문에 본 발명에서와 같이 차량 주행 상태를 정상 상태와 비정상 상태, 그리고 비정상으로 나아가고 있는 상태 등을 스스로 구별해낼 수 있다.In the present invention, a neural network model is trained by deep learning on the usual vehicle driving environment, that is, the driver's usual driving pattern and the in-vehicle environmental state or vehicle driving state. Since each driver has different vehicle conditions and driving habits, it is desirable to gradually learn from the neural network model rather than uniformly programming in advance. In particular, in the case of deep learning, since it has the ability to classify categories on its own without the intervention of an administrator, as in the present invention, the vehicle driving state can be distinguished by itself into a normal state, an abnormal state, and a state that is advancing to an abnormal state.
딥러닝(deep learning)은 훈련 과정과 적용 과정으로 이루어진다. 훈련 과정은 학습 데이터셋을 통해 신경망 모델을 학습시켜가는 과정이고, 적용 과정은 신경망 모델을 실제 상황에 활용하는 과정이다. 일반적으로, 훈련 과정은 연산량이 많고 적용 과정은 연산량이 적다.Deep learning consists of a training process and an application process. The training process is the process of learning the neural network model through the training dataset, and the application process is the process of using the neural network model in real situations. In general, the training process has a large amount of computation, and the application process has a small amount of computation.
본 발명에서는 차량 운행 환경을 감안하여 훈련 과정과 적용 과정을 분리하여 구성하였다. 훈련 과정은 클라우드 컴퓨터(200)와 엣지 컴퓨팅부(130)가 담당하고, 차량 DMS 장치(100)의 다른 구성요소가 적용 과정을 담당한다. 차량 내의 각종 상태정보를 획득하고 신경망 모델을 통해 안전운전 위협요인을 식별하여 운전자에게 알려주는 과정은 차량 DMS 장치(100)가 수행한다. 반면, 차량 내의 각종 상태정보를 이용하여 신경망 모델을 훈련시키는 것은 클라우드 컴퓨터(200)와 엣지 컴퓨팅부(130)가 수행한다. 이러한 분리를 통해 차량 전자제어장치(ECU)의 연산 부담을 줄일 수 있어 최종적으로는 가격을 낮출 수 있다. In the present invention, the training process and the application process are separated and configured in consideration of the vehicle driving environment. The training process is in charge of the cloud computer 200 and the edge computing unit 130 , and other components of the vehicle DMS device 100 are in charge of the application process. The vehicle DMS device 100 performs a process of acquiring various state information in the vehicle, identifying a threat factor for safe driving through a neural network model, and notifying the driver. On the other hand, the cloud computer 200 and the edge computing unit 130 perform training of the neural network model using various state information in the vehicle. Through this separation, it is possible to reduce the computational burden of the vehicle electronic control unit (ECU), thereby ultimately lowering the price.
본 발명에서는 운전자 상태정보에 기초하여 현재 운전자의 상태가 어떠한지 식별하도록 신경망 모델을 훈련시킨다. 즉, 카메라 영상에서 눈, 코, 입 등 사람이라고 인식할 수 있는 특징점을 인식하고 눈동자, 입, 얼굴 형태가 어떤 행동 패턴을 가질 때를 졸고 있는 것인지 판단 기준을 설정한다.In the present invention, a neural network model is trained to identify the current driver's state based on driver state information. That is, the camera image recognizes feature points that can be recognized as a person, such as eyes, nose, and mouth, and sets a criterion for determining when the pupil, mouth, and face have a certain behavior pattern.
또한, 차량환경 상태정보에 기초하여 차량 내부의 운전 환경이 어떠한지 식별하도록 신경망 모델을 훈련시킨다. 즉, 차량내 환경(예: 이산화탄소 농도, 내부 온도)이 탑승 인원 수, 호흡량, 환기장치 작동여부에 따라 변화되는 패턴을 기준으로 정상적인 상태를 유지하는지, 빠르게 나빠지는지, 완만하게 나빠지는지 등에 대한 판단 기준을 설정한다.In addition, the neural network model is trained to identify the driving environment inside the vehicle based on the vehicle environment state information. That is, based on the pattern that the in-vehicle environment (e.g., carbon dioxide concentration, internal temperature) changes depending on the number of passengers, respiration rate, and whether or not the ventilation system operates, it is determined whether the vehicle maintains a normal state, deteriorates rapidly, or deteriorates gently. Set standards.
또한, 차량주행 상태정보에 기초하여 차량 운전이 어떤 상황인지와 정상인지 아닌지도 식별하도록 신경망 모델을 훈련시킨다. 즉, 차량 속도가 가다서다를 반복하는지, 저속운행 하는지, 고속운행 하는지, 저속과 고속을 반복하는지, 급가속 급감속을 반복하는지에 따라 정체구간, 시내주행, 고속도로 주행, 졸음 운전, 난폭운전 등에 대한 판단 기준을 설정한다.In addition, based on the vehicle driving state information, the neural network model is trained to identify what kind of situation the vehicle is driving and whether it is normal or not. In other words, depending on whether the vehicle speed repeats going to and fro, driving at low speed, driving at high speed, repeating low speed and high speed, or repeating rapid acceleration and deceleration, traffic jams, city driving, highway driving, drowsy driving, reckless driving, etc. Set the criteria for judging
이와 같은 신경망 모델을 차량 DMS 장치(100)의 엣지 컴퓨팅부(130)에 구비함으로써 차량운전 상태정보에 따라 차량 주행 중의 안전운전 위협요인을 식별할 수 있도록 구성하였다. 또한, 엣지 컴퓨팅부(130)와 클라우드 컴퓨터(200)는 차량 DMS 장치(100)가 수집하는 차량운전 상태정보(운전자 상태정보, 차량환경 상태정보, 차량주행 상태정보)를 기반으로 반복적으로 추가 학습을 통해 신경망 모델을 개선(업데이트)해나가는 구성을 채택하였다.By providing such a neural network model in the edge computing unit 130 of the vehicle DMS device 100, it is configured to identify the safe driving threat factors during vehicle driving according to vehicle driving state information. In addition, the edge computing unit 130 and the cloud computer 200 iteratively additional learning based on the vehicle driving state information (driver state information, vehicle environment state information, vehicle driving state information) collected by the vehicle DMS device 100 A configuration that improves (updates) the neural network model was adopted.
이때, 차량 DMS 장치(100)에 구비된 엣지 컴퓨팅부(130)는 차량의 ECU 장치에 설치된 SoC 칩셋에서 뉴럴네트워크 프로세서 유닛(NPU)를 통해 사람의 신경망 회로와 동일한 개념으로 센서정보의 다양한 입력의 경우의 수를 빠르게 처리할 수 있도록 구현된 장치로 구현될 수 있다. 또한, 클라우드 컴퓨터(200)는 차량 DMS 장치(100)로부터 각종 데이터를 업로드 제공받아 상대적으로 장시간의 운전자 행동 패턴 및 장시간의 차량환경을 분석 학습하여 신경망 모델을 개선하고, 이를 통해 개선 학습한 신경망 모델을 개별 차량의 차량 DMS 장치(100)의 엣지 컴퓨팅부(130)로 제공함으로써 분석 성능을 향상시킨다.At this time, the edge computing unit 130 provided in the vehicle DMS device 100 is the same concept as the human neural network circuit through the neural network processor unit (NPU) in the SoC chipset installed in the ECU device of the vehicle. It can be implemented as a device implemented to quickly process the number of cases. In addition, the cloud computer 200 receives various data uploaded from the vehicle DMS device 100 and analyzes and learns a relatively long-time driver behavior pattern and a long-term vehicle environment to improve the neural network model, and the neural network model learned through improvement is provided to the edge computing unit 130 of the vehicle DMS device 100 of an individual vehicle to improve analysis performance.
본 발명에서는 신경망 모델에 대한 특히, 딥러닝 훈련 과정을 분리시켜 클라우드 컴퓨터(200)와 엣지 컴퓨팅부(130)로 분담시킨 에지 딥러닝 구조를 채택하는데, 이때 신경망 훈련 알고리즘에서 가장 복잡하여 연산량이 많은 과정은 클라우드 코어의 클라우드 컴퓨터(200)가 수행한다. 이를 위해, 본 발명에서 신경망 모델은 분리 훈련이 가능한 모델이 적용된 것으로 가정한다.In the present invention, in particular, the deep learning training process for the neural network model is separated and the edge deep learning structure divided into the cloud computer 200 and the edge computing unit 130 is adopted. The process is performed by the cloud computer 200 of the cloud core. To this end, in the present invention, it is assumed that a model capable of separation training is applied to the neural network model.
엣지 컴퓨팅부(130)는 클라우드 컴퓨터(200)보다는 연산 성능은 떨어지지만 가벼운 훈련 과정이 적용되어 차량운전 상태정보에 따라 즉각적으로 딥러닝 훈련이 이루어져 운전자에게 빠르게 적용할 수 있도록 한다. 반면에 클라우드 컴퓨터(200)는 고성능 장치로서 대규모 업데이트가 필요한 부분을 적용할 수 있도록 구성한다. 이러한 에지 딥러닝 구조에 따르면 신경망 모델의 훈련을 빠르게 달성할 수 있고 그 학습 결과를 차량에 신속하게 활용할 수 있다.The edge computing unit 130 has lower computational performance than the cloud computer 200, but a light training process is applied, so that deep learning training is performed immediately according to vehicle driving state information so that it can be quickly applied to the driver. On the other hand, the cloud computer 200 is a high-performance device and is configured to apply a large-scale update required. According to such an edge deep learning structure, training of a neural network model can be achieved quickly, and the training results can be quickly utilized in a vehicle.
그에 따라, 엣지 컴퓨팅부(130)와 클라우드 컴퓨터(200)에 대해 상이한 딥러닝 훈련 정책을 적용한다. 엣지 컴퓨팅부(130)는 그 실시간 획득하는 차량운전 상태정보로부터 제 1 훈련 데이터셋을 구성하고 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 적은 연산량을 요구하는 제 1 훈련 부분을 수행하여 신경망 모델을 튜닝하도록 구성한다. 반면, 클라우드 컴퓨터(200)는 차량 DMS 장치(100)로부터 제공받는 차량운전 상태정보로부터 제 2 훈련 데이터셋을 구성하고 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 많은 연산량을 요구하는 제 2 훈련 부분을 수행하여 업데이트 신경망 모델을 생성하도록 구성한다.Accordingly, different deep learning training policies are applied to the edge computing unit 130 and the cloud computer 200 . The edge computing unit 130 tunes the neural network model by configuring a first training dataset from the real-time acquired vehicle driving state information and performing the first training part requiring a relatively small amount of computation during deep learning training for the neural network model. configure to do On the other hand, the cloud computer 200 configures the second training dataset from the vehicle driving state information provided from the vehicle DMS device 100 and performs the second training part that requires a relatively large amount of computation during deep learning training for the neural network model. Configure to create an updated neural network model by performing
즉, 엣지 컴퓨팅부(130)가 클라우드 컴퓨터(200)가 차량운전 상태정보(운전자 상태정보, 차량환경 상태정보, 차량주행 상태정보)에 기초하여 딥러닝 훈련을 수행하는 점에서는 공통되지만, 그 훈련 데이터셋(training dataset)은 바람직하게는 상이하게 설정된다. 이때, 엣지 컴퓨팅부(130)에 대해 더 적은 규모의 훈련 데이터셋을 구성한다. 또한, 엣지 컴퓨팅부(130)는 신경망 모델에 대한 상대적으로 작은 변화(튜닝)을 수행하는 반면, 클라우드 컴퓨터(200)는 신경망 모델에 대한 상대적으로 큰 변화(업데이트)를 수행하도록 구성된다. 이처럼 연산량의 측면에서 비교할 때, 엣지 컴퓨팅부(130)는 클라우드 컴퓨터(200)에 비해 상대적으로 적은 연산량을 요구하는 딥러닝 훈련 정책을 구성한다.That is, the edge computing unit 130 is common in that the cloud computer 200 performs deep learning training based on the vehicle driving state information (driver state information, vehicle environment state information, vehicle driving state information), but the training The training dataset is preferably set up differently. In this case, a smaller-scale training dataset is configured for the edge computing unit 130 . In addition, the edge computing unit 130 performs a relatively small change (tuning) on the neural network model, while the cloud computer 200 is configured to perform a relatively large change (update) on the neural network model. As such, when compared in terms of the amount of computation, the edge computing unit 130 configures a deep learning training policy that requires a relatively small amount of computation compared to the cloud computer 200 .
예를 들어, 엣지 컴퓨팅부(130)는 미리 설정된 제 1 시간구간(예: 1시간) 동안 그 실시간 획득하는 차량운전 상태정보로부터 제 1 훈련 데이터셋을 구성하고 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 적은 연산량을 요구하는 제 1 훈련 부분을 수행하도록 구성될 수 있다. 이때, 클라우드 컴퓨터(200)는 미리 설정된 제 2 시간구간 동안(단, 제 2 시간구간은 제 1 시간구간보다 더 길게 설정됨; 예: 7일) 차량 DMS 장치(100)로부터 제공받는 차량운전 상태정보로부터 제 2 훈련 데이터셋을 구성하고 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 많은 연산량을 요구하는 제 2 훈련 부분을 수행하도록 구성될 수 있다.For example, the edge computing unit 130 configures a first training dataset from the vehicle driving state information acquired in real time during a preset first time period (eg, 1 hour), and performs a relative operation during deep learning training for the neural network model. It may be configured to perform the first training part requiring a small amount of computation. In this case, the cloud computer 200 receives the vehicle driving state provided from the vehicle DMS device 100 for a preset second time period (however, the second time period is set longer than the first time period; for example, 7 days) It may be configured to construct a second training dataset from information and perform a second training part that requires a relatively large amount of computation during deep learning training for a neural network model.
본 발명에서 엣지 컴퓨팅부(130)와 클라우드 컴퓨터(200)에 대해 상대적으로 상이한 연산량을 요구하는 딥러닝 훈련 정책을 구성한 세 가지 실시예를 도 2 내지 도 4에 제시한다. 이를 통해 엣지 컴퓨팅부(130)는 신경망 모델에 대한 상대적으로 작은 변화(튜닝)을 수행하고, 클라우드 컴퓨터(200)는 신경망 모델에 대한 상대적으로 큰 변화(업데이트)를 수행한다.In the present invention, three embodiments in which a deep learning training policy that requires a relatively different amount of computation for the edge computing unit 130 and the cloud computer 200 is configured is shown in FIGS. 2 to 4 . Through this, the edge computing unit 130 performs a relatively small change (tuning) on the neural network model, and the cloud computer 200 performs a relatively large change (update) on the neural network model.
도 2는 본 발명에서 엣지 컴퓨팅부(130)와 클라우드 컴퓨터(200)에 의한 에지 딥러닝 구성의 제 1 실시예를 나타내는 도면이다.FIG. 2 is a diagram illustrating a first embodiment of an edge deep learning configuration by the edge computing unit 130 and the cloud computer 200 in the present invention.
도 2에서는 엣지 컴퓨팅부(130)는 신경망 모델의 레이어 구조를 고정한 상태에서 노드간 연결 가중치(connection weights)에 대한 훈련을 수행하여 신경망 모델을 튜닝하도록 구성된다. 이때, 클라우드 컴퓨터(200)는 신경망 모델의 레이어 구조(노드 구성) 및 노드간 연결 가중치 모두에 대한 훈련을 수행하여 업데이트 신경망 모델을 생성하도록 구성된다.In FIG. 2 , the edge computing unit 130 is configured to tune the neural network model by performing training on connection weights between nodes in a state where the layer structure of the neural network model is fixed. At this time, the cloud computer 200 is configured to generate an updated neural network model by performing training on both the layer structure (node configuration) of the neural network model and the connection weight between nodes.
도 3은 본 발명에서 엣지 컴퓨팅부(130)와 클라우드 컴퓨터(200)에 의한 에지 딥러닝 구성의 제 2 실시예를 나타내는 도면이다.FIG. 3 is a diagram illustrating a second embodiment of an edge deep learning configuration by the edge computing unit 130 and the cloud computer 200 in the present invention.
엣지 컴퓨팅부(130)는 신경망 모델의 입력층(input layer) 및 출력층(output layer)에 관련된 레이어에 대한 훈련을 수행하여 신경망 모델을 튜닝하도록 구성된다. 입력층에 관련된 레이어(도 3에서 하단 좌측) 및 출력층에 관련된 레이어(도 3에서 하단 우측) 모두에 대한 훈련을 수행할 수도 있고, 이들 중에서 어느 하나에 대한 훈련만 수행할 수도 있다. 입력층 또는 출력층에 직접 연결되어 있는 은닉층의 첫번째 레이어와 마지막 레이어에 대해서는 엣지 컴퓨팅부(130)가 레이어 구조(노드 구성)을 변경하도록 구성할 수도 있고 변경하지 않도록 구성할 수도 있다. 이때, 클라우드 컴퓨터(200)는 신경망 모델의 은닉층(hidden layers)에 관련된 다수의 레이어에 대한 딥러닝 훈련을 수행하여 업데이트 신경망 모델을 생성하도록 구성된다.The edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to an input layer and an output layer of the neural network model. Training may be performed on both the layer related to the input layer (bottom left in FIG. 3) and the layer related to the output layer (bottom right in FIG. 3), or only one of them may be trained. For the first and last layers of the hidden layer directly connected to the input layer or the output layer, the edge computing unit 130 may be configured to change the layer structure (node configuration) or may be configured not to change. At this time, the cloud computer 200 is configured to generate an updated neural network model by performing deep learning training on a plurality of layers related to hidden layers of the neural network model.
도 4는 본 발명에서 엣지 컴퓨팅부(130)와 클라우드 컴퓨터(200)에 의한 에지 딥러닝 구성의 제 3 실시예를 나타내는 도면이다.FIG. 4 is a diagram showing a third embodiment of the edge deep learning configuration by the edge computing unit 130 and the cloud computer 200 in the present invention.
엣지 컴퓨팅부(130)는 신경망 모델의 입력층(input layer) 및 출력층(output layer)에 관련된 레이어에 대한 훈련을 수행하여 신경망 모델을 튜닝하도록 구성된다. 입력층에 관련된 레이어(도 3에서 하단 좌측) 및 출력층에 관련된 레이어(도 3에서 하단 우측) 모두에 대한 훈련을 수행할 수도 있고, 이들 중에서 어느 하나에 대한 훈련만 수행할 수도 있다. 이때, 클라우드 컴퓨터(200)는 신경망 모델 전체(overall structure of the neural network)에 대한 훈련을 수행하여 업데이트 신경망 모델을 생성하도록 구성된다.The edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to an input layer and an output layer of the neural network model. Training may be performed on both the layer related to the input layer (bottom left in FIG. 3) and the layer related to the output layer (bottom right in FIG. 3), or only one of them may be trained. At this time, the cloud computer 200 is configured to generate an updated neural network model by performing training on the overall structure of the neural network.
도 3과 도 4의 딥러닝 훈련 정책에서는 엣지 컴퓨팅부(130)와 클라우드 컴퓨터(200)의 연산량에 큰 차이가 있다. 안전운전 알고리즘이 발전할수록 딥러닝 학습이 진행됨에 따라 신경망 모델의 레이어(layers)가 많아지는데, 레이어의 수가 증가할수록 은닉층에서는 컨넥션이 급격히 증가하며 훈련에 많은 컴퓨팅 파워가 필요하게 된다. 이러한 점을 감안하여 엣지 컴퓨팅부(130)는 신경망 모델의 입력층 및 출력층에 관련된 훈련을 수행하도록 구성하였다. 이를 통해, 다수의 차량 내에 개별적으로 장착되는 엣지 컴퓨팅부(130)를 효율적으로 운영할 수 있고 엣지 컴퓨팅부(130)에 대한 비용을 감소시킬 수 있다.In the deep learning training policy of FIGS. 3 and 4 , there is a big difference in the amount of computation of the edge computing unit 130 and the cloud computer 200 . As the safe driving algorithm develops, the number of layers of the neural network model increases as deep learning learning proceeds. In consideration of this point, the edge computing unit 130 is configured to perform training related to the input layer and the output layer of the neural network model. Through this, it is possible to efficiently operate the edge computing unit 130 individually mounted in a plurality of vehicles, and to reduce the cost of the edge computing unit 130 .
도 5는 본 발명에서 차량 DMS 장치(100)의 내부 구성을 나타내는 블록도이다. 도 5를 참조하면, 차량 DMS 장치(100)는 상태정보 획득부(110), 데이터 처리부(120), 엣지 컴퓨팅부(130), 데이터 저장부(140), 차량 통신부(150), 운전자 경보부(160)를 포함하여 구성된다.5 is a block diagram showing the internal configuration of the vehicle DMS device 100 in the present invention. Referring to FIG. 5 , the vehicle DMS device 100 includes a state information acquisition unit 110 , a data processing unit 120 , an edge computing unit 130 , a data storage unit 140 , a vehicle communication unit 150 , and a driver alert unit ( 160) is included.
먼저, 상태정보 획득부(110)는 해당 차량에 대한 차량운전 상태정보를 실시간으로 획득하는 구성요소이다. 이를 위해, 상태정보 획득부(110)는 운전자 상태정보를 획득하기 위한 운전자 감시 카메라(111), 차량환경 상태정보를 획득하기 위한 차량내부 환경센서(112), 차량주행 상태정보를 획득하기 위한 차량주행 상태센서(113) 중 둘 이상을 구비한다. 본 발명은 단순히 운전자 얼굴 및 특징을 인식할 뿐 아니라 다양한 센서 데이터(차량 움직임 정보, 차량 내부 정보)를 신경망 모델에 입력하여 운전자 상태의 원인을 분석한다. First, the state information acquisition unit 110 is a component that acquires vehicle driving state information for a corresponding vehicle in real time. To this end, the state information acquisition unit 110 includes a driver monitoring camera 111 for acquiring driver state information, an in-vehicle environmental sensor 112 for acquiring vehicle environment state information, and a vehicle for acquiring vehicle driving state information. Two or more of the driving state sensors 113 are provided. The present invention not only simply recognizes the driver's face and features, but also inputs various sensor data (vehicle motion information, vehicle interior information) into a neural network model to analyze the cause of the driver's state.
이때, 운전자 감시 카메라(111)는 조명의 영향을 적게 받기 위하여 적외선 조명과 적외선 카메라가 바람직하며, 카메라 촬영 영상을 통해 운전자의 상태를 인식한다. 운전자 상태 인식 기술은 종래에 구현되어 있고 본 발명의 핵심도 아니므로 이에 관한 구체적인 설명을 생략한다. 차량내부 환경센서(112)는 공기질 센서와 온도 센서 등을 통해 차량 내부의 이산화탄소 농도, 내부 온도 등의 정보를 획득하며, 차량주행 상태센서(113)는 차량의 주행 속도, 주행 시간, 창문 열림 등 차량의 주행 상태에 대한 정보를 획득한다.In this case, the driver monitoring camera 111 preferably uses infrared lighting and an infrared camera in order to be less affected by lighting, and recognizes the driver's condition through the camera captured image. Since the driver state recognition technology has been implemented in the prior art and is not the core of the present invention, a detailed description thereof will be omitted. The vehicle internal environment sensor 112 acquires information such as carbon dioxide concentration and internal temperature inside the vehicle through an air quality sensor and a temperature sensor, and the vehicle driving state sensor 113 includes the vehicle driving speed, driving time, window opening, etc. Acquire information about the driving state of the vehicle.
데이터 처리부(120)는 상태정보 획득부(110)가 실시간 획득한 차량운전 상태정보를 신경망 모델에 입력하여 해당 차량 주행 중의 안전운전 위협요인을 출력받고 이를 차량 운전자에게 알리는 구성요소이다. 엣지 컴퓨팅부(130)에는 차량운전 상태정보로 딥러닝 학습된 신경망 모델이 구비되어 있는데, 데이터 처리부(120)는 상태정보 획득부(110)가 제공하는 차량운전 상태정보를 엣지 컴퓨팅부(130)의 신경망 모델에 입력함으로써 안전운전 위협요인(즉, 운전자의 상태 및 원인)을 도출하고 운전자에게 상태 이상을 알려주며 바람직하게는 원인 분석 결과도 알려준다. 또한, 데이터 처리부(120)는 차량운전 상태정보 및 데이터 처리 결과를 데이터 저장부(140)로 제공하여 저장되도록 한다.The data processing unit 120 is a component that inputs the vehicle driving state information obtained in real time by the state information acquisition unit 110 into the neural network model, outputs the safe driving threat factors while driving the corresponding vehicle, and informs the vehicle driver. The edge computing unit 130 is equipped with a neural network model trained by deep learning with vehicle driving state information, and the data processing unit 120 receives the vehicle driving state information provided by the state information acquisition unit 110 to the edge computing unit 130 . By inputting it into the neural network model of In addition, the data processing unit 120 provides the vehicle driving state information and the data processing result to the data storage unit 140 to be stored.
엣지 컴퓨팅부(130)는 신경망 모델(neural network model)을 예컨대 컴퓨터 프로그램의 형태로 내장하고 신경망 모델에 관련된 각종 데이터 처리를 수행하는 구성요소이다. 엣지 컴퓨팅부(130)는 차량 ECU 장치의 SoC(System-on-Chip) 칩셋에서 뉴럴네트워크 프로세서 유닛(NPU)의 하드웨어로 구현될 수 있는데, 신경망 모델은 이 NPU의 하드웨어 회로 형태로 구현될 수도 있고 이 NPU 상에서 구동되는 컴퓨터 프로그램의 형태로 구현될 수도 있다. The edge computing unit 130 is a component that embeds a neural network model in the form of, for example, a computer program and performs various data processing related to the neural network model. The edge computing unit 130 may be implemented as hardware of a neural network processor unit (NPU) in a SoC (System-on-Chip) chipset of a vehicle ECU device, and the neural network model may be implemented in the form of a hardware circuit of this NPU, It may be implemented in the form of a computer program running on this NPU.
엣지 컴퓨팅부(130)는 데이터 처리부(120)로부터 차량운전 상태정보를 입력받아 신경망 모델에 의해 해당 차량 주행 중의 안전운전 위협요인을 도출하여 데이터 처리부(120)로 출력한다. 또한, 엣지 컴퓨팅부(130)는 차량운전 상태정보에 기초하여 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 적은 연산량을 요구하는 제 1 훈련 부분을 수행하여 신경망 모델을 튜닝한다.The edge computing unit 130 receives vehicle driving state information from the data processing unit 120 , derives a safe driving threat factor during driving of the corresponding vehicle by a neural network model, and outputs it to the data processing unit 120 . In addition, the edge computing unit 130 tunes the neural network model by performing a first training part that requires a relatively small amount of computation among deep learning training for the neural network model based on vehicle driving state information.
실제 하드웨어 구현에 있어서 엣지 컴퓨팅부(130)가 차량운전 상태정보를 상태정보 획득부(110)로부터 직접 전달받는 것처럼 구현될 수도 있다. 하지만, 하드웨어 구현을 살펴보면, 데이터 처리부(120)와 엣지 컴퓨팅부(130)는 차량 ECU 장치에 함께 설치되어 있고, 상태정보 획득부(110)에서 차량 ECU 장치로 입력된 차량운전 상태정보가 데이터 처리부(120)와 엣지 컴퓨팅부(130)로 전달되는 것이다. 따라서, 엣지 컴퓨팅부(130)가 차량운전 상태정보를 데이터 처리부(120)로부터 전달받는 것으로 모델링할 수 있다.In actual hardware implementation, the edge computing unit 130 may be implemented as if the vehicle driving state information is directly transmitted from the state information acquisition unit 110 . However, looking at the hardware implementation, the data processing unit 120 and the edge computing unit 130 are installed together in the vehicle ECU, and the vehicle driving state information input from the state information obtaining unit 110 to the vehicle ECU is transferred to the data processing unit. It is transmitted to 120 and the edge computing unit 130 . Accordingly, the edge computing unit 130 may be modeled as receiving the vehicle driving state information from the data processing unit 120 .
본 발명에서 엣지 컴퓨팅부(130)는 데이터 처리부(120)에서 입력되는 차량운전 상태정보를 이용하여 평소에 신경망 모델을 딥러닝 훈련하는 한편, 차량운전 상태정보를 신경망 모델에 적용하여 차량 주행 중의 안전운전 위협요인을 도출하고 데이터 처리부(120)로 제공하고, 미리 설정된 기준에 따라 추가 학습이 필요한 데이터와 분석 결과를 데이터 저장부(140)에 전송한다. In the present invention, the edge computing unit 130 normally performs deep learning training on the neural network model using the vehicle driving state information input from the data processing unit 120, while applying the vehicle driving state information to the neural network model to ensure safety during vehicle driving. Driving threat factors are derived and provided to the data processing unit 120 , and data requiring additional learning and analysis results are transmitted to the data storage unit 140 according to preset criteria.
데이터 저장부(140)는 상태정보 획득부(110)에 의해 실시간 획득되는 차량운전 상태정보 및 데이터 처리부(120)에 의한 데이터 처리 결과를 임시 저장하는 구성요소이다. 일반적으로 차량에 장착된 시스템은 주행 상황(예: 오지, 터널 등)에 의해 네트워크 접속이 보장되지 않기 때문에 데이터 저장부(140)는 센서 입력 및 데이터 처리 결과를 임시 저장하며 네트워크에 연결될 경우에 차량 통신부(150)를 통해서 외부로 데이터를 전송하는 역할을 한다. The data storage unit 140 is a component for temporarily storing the vehicle driving state information acquired in real time by the state information acquisition unit 110 and the data processing result by the data processing unit 120 . In general, a system mounted on a vehicle does not guarantee network access due to driving conditions (eg, remote areas, tunnels, etc.), so the data storage unit 140 temporarily stores sensor input and data processing results, and when connected to a network, the vehicle It serves to transmit data to the outside through the communication unit 150 .
차량 통신부(150)는 차량 DMS 장치(100)가 외부와 데이터 송수신할 수 있도록 통신 채널을 제공한다. 즉, 차량 통신부(150)는 데이터 저장부(140)에 임시 저장되어 있는 데이터 및 엣지 컴퓨팅부(130)가 보유하는 신경망 모델의 데이터를 클라우드 컴퓨터(200)에 대해 전달하고 클라우드 컴퓨터(200)로부터 업데이트 신경망 모델을 제공받아 엣지 컴퓨팅부(130)로 전달하는 구성요소이다. 이때, 차량 통신부(150)는 광대역 네트워크를 통해 클라우드 컴퓨터(200)와 직접 연결되는 방식으로 구성될 수도 있고, 근거리 네트워크로 로컬 서버(300)와 연결되는 방식으로 구성될 수도 있다. 또한, 차량 통신부(150)는 무선 네트워크가 바람직하지만 유선 네트워크를 배제하는 것은 아니다.The vehicle communication unit 150 provides a communication channel so that the vehicle DMS device 100 can transmit and receive data with the outside. That is, the vehicle communication unit 150 transmits the data temporarily stored in the data storage unit 140 and the neural network model data held by the edge computing unit 130 to the cloud computer 200 and from the cloud computer 200 . It is a component that receives the updated neural network model and transmits it to the edge computing unit 130 . In this case, the vehicle communication unit 150 may be configured to be directly connected to the cloud computer 200 through a broadband network, or may be configured to be connected to the local server 300 via a local area network. In addition, the vehicle communication unit 150 is preferably a wireless network, but does not exclude a wired network.
운전자 경보부(160)는 데이터 처리부(120)의 제어에 의해 안전운전 위협요인을 차량 운전자에게 시각, 청각, 촉각, 후각 등 인지 가능한 방식으로 출력하는 구성요소이다.The driver warning unit 160 is a component that outputs safety driving threat factors to the vehicle driver in a recognizable way such as visual, auditory, tactile, and olfactory under the control of the data processing unit 120 .
한편, 본 발명은 컴퓨터가 읽을 수 있는 비휘발성 기록매체에 컴퓨터가 읽을 수 있는 코드의 형태로 구현되는 것이 가능하다. 이러한 비휘발성 기록매체로는 다양한 형태의 스토리지 장치가 존재하는데 예컨대 하드디스크, SSD, CD-ROM, NAS, 자기테이프, 웹디스크, 클라우드 디스크 등이 있고 네트워크로 연결된 다수의 스토리지 장치에 코드가 분산 저장되고 실행되는 형태도 구현될 수 있다. 또한, 본 발명은 하드웨어와 결합되어 특정의 절차를 실행시키기 위하여 매체에 저장된 컴퓨터프로그램의 형태로 구현될 수도 있다.Meanwhile, the present invention can be implemented in the form of computer-readable codes on a computer-readable non-volatile recording medium. Various types of storage devices exist as such non-volatile recording media. For example, hard disks, SSDs, CD-ROMs, NAS, magnetic tapes, web disks, cloud disks, etc. A form in which it can be implemented and executed can also be implemented. In addition, the present invention may be implemented in the form of a computer program stored in a medium to execute a specific procedure in combination with hardware.

Claims (10)

  1. 차량운전 상태정보를 이용하여 차량 주행 중의 안전운전 위협요인을 식별하여 운전자에게 알림으로써 차량 안전운전은 보조하기 위한 에지 딥러닝 기반의 차량 안전운전 시스템으로서,It is an edge deep learning-based safe driving system for vehicle safety that uses vehicle driving status information to identify risk factors for safe driving while driving and notify the driver to assist in safe driving.
    개별 차량에 장착되고, 해당 차량에 대해 차량운전 상태정보를 실시간으로 획득하고, 신경망 모델을 내장한 엣지 컴퓨팅부(130)를 구비하고, 상기 실시간 획득된 차량운전 상태정보를 상기 엣지 컴퓨팅부(130)의 상기 신경망 모델에 입력하여 해당 차량 주행 중의 안전운전 위협요인을 식별하고 차량 운전자에게 알리는 차량 DMS 장치(100); It is mounted on an individual vehicle, obtains vehicle driving state information for the corresponding vehicle in real time, has an edge computing unit 130 having a built-in neural network model, and uses the real-time acquired vehicle driving state information to the edge computing unit 130 ) input into the neural network model of the vehicle DMS device 100 to identify a safe driving threat factor while driving the vehicle and notify the vehicle driver;
    개별 차량에 장착된 상기 차량 DMS 장치(100)로부터 차량운전 상태정보와 신경망 모델 데이터를 제공받아 상기 차량운전 상태정보에 기초하여 상기 차량 DMS 장치(100)의 신경망 모델을 딥러닝 훈련시켜 업데이트 신경망 모델을 생성한 후에 상기 차량 DMS 장치(100)의 엣지 컴퓨팅부(130)로 상기 업데이트 신경망 모델을 제공하는 클라우드 컴퓨터(200);Update neural network model by receiving vehicle driving state information and neural network model data from the vehicle DMS device 100 mounted on an individual vehicle, and deep learning training the neural network model of the vehicle DMS device 100 based on the vehicle driving state information a cloud computer 200 that provides the updated neural network model to the edge computing unit 130 of the vehicle DMS device 100 after generating;
    를 포함하여 구성되고,consists of,
    상기 엣지 컴퓨팅부(130)는 상기 실시간 획득하는 상기 차량운전 상태정보로부터 제 1 훈련 데이터셋을 구성하고 상기 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 적은 연산량을 요구하는 제 1 훈련 부분을 수행하여 상기 신경망 모델을 튜닝하도록 구성되고, The edge computing unit 130 configures a first training dataset from the vehicle driving state information acquired in real time, and performs a first training part requiring a relatively small amount of computation during deep learning training for the neural network model. configured to tune a neural network model,
    상기 클라우드 컴퓨터(200)는 상기 차량 DMS 장치(100)로부터 제공받는 상기 차량운전 상태정보로부터 제 2 훈련 데이터셋을 구성하고 상기 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 많은 연산량을 요구하는 제 2 훈련 부분을 수행하여 상기 업데이트 신경망 모델을 생성하도록 구성되는 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템.The cloud computer 200 configures a second training dataset from the vehicle driving state information provided from the vehicle DMS device 100, and a second training that requires a relatively large amount of computation during deep learning training for the neural network model. Edge deep learning-based vehicle safe driving system using vehicle driving state information, configured to generate the updated neural network model by performing a part.
  2. 청구항 1에 있어서, The method according to claim 1,
    상기 엣지 컴퓨팅부(130)는 미리 설정된 제 1 시간구간 동안 상기 실시간 획득하는 상기 차량운전 상태정보로부터 상기 제 1 훈련 데이터셋을 구성하고 상기 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 적은 연산량을 요구하는 제 1 훈련 부분을 수행하도록 구성되고, The edge computing unit 130 configures the first training dataset from the vehicle driving state information acquired in real time during a preset first time period and requires a relatively small amount of computation during deep learning training for the neural network model. configured to perform a first training portion,
    상기 클라우드 컴퓨터(200)는 미리 설정된 제 2 시간구간 동안(단, 제 2 시간구간은 제 1 시간구간보다 더 길게 설정됨) 상기 차량 DMS 장치(100)로부터 제공받는 상기 차량운전 상태정보로부터 상기 제 2 훈련 데이터셋을 구성하고 상기 신경망 모델에 대한 딥러닝 훈련 중 상대적으로 많은 연산량을 요구하는 제 2 훈련 부분을 수행하도록 구성되는 것을 특징으로 하는 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템.The cloud computer 200 receives the second time period from the vehicle driving state information provided from the vehicle DMS device 100 during a preset second time period (provided that the second time period is set longer than the first time period). Edge deep learning-based vehicle safe driving using vehicle driving state information, characterized in that it is configured to configure 2 training datasets and perform a second training part that requires a relatively large amount of computation during deep learning training for the neural network model. system.
  3. 청구항 1에 있어서, The method according to claim 1,
    상기 엣지 컴퓨팅부(130)는 상기 신경망 모델의 레이어 구조를 고정한 상태에서 노드간 연결 가중치에 대한 훈련을 수행하여 상기 신경망 모델을 튜닝하도록 구성되고, The edge computing unit 130 is configured to tune the neural network model by performing training on the connection weights between nodes in a state where the layer structure of the neural network model is fixed,
    상기 클라우드 컴퓨터(200)는 상기 신경망 모델의 레이어 구조 및 노드간 연결 가중치 모두에 대한 훈련을 수행하여 상기 업데이트 신경망 모델을 생성하도록 구성되는 것을 특징으로 하는 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템.The cloud computer 200 is an edge deep learning-based vehicle using vehicle driving state information, characterized in that it is configured to generate the updated neural network model by performing training on both the layer structure of the neural network model and the connection weights between nodes. safe driving system.
  4. 청구항 2에 있어서, 3. The method according to claim 2,
    상기 엣지 컴퓨팅부(130)는 상기 신경망 모델의 레이어 구조를 고정한 상태에서 노드간 연결 가중치에 대한 훈련을 수행하여 상기 신경망 모델을 튜닝하도록 구성되고, The edge computing unit 130 is configured to tune the neural network model by performing training on the connection weights between nodes in a state where the layer structure of the neural network model is fixed,
    상기 클라우드 컴퓨터(200)는 상기 신경망 모델의 레이어 구조 및 노드간 연결 가중치 모두에 대한 훈련을 수행하여 상기 업데이트 신경망 모델을 생성하도록 구성되는 것을 특징으로 하는 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템.The cloud computer 200 is an edge deep learning-based vehicle using vehicle driving state information, characterized in that it is configured to generate the updated neural network model by performing training on both the layer structure of the neural network model and the connection weights between nodes. safe driving system.
  5. 청구항 1에 있어서, The method according to claim 1,
    상기 엣지 컴퓨팅부(130)는 상기 신경망 모델의 입력층 및 출력층 중 하나 이상에 관련된 레이어에 대한 훈련을 수행하여 상기 신경망 모델을 튜닝하도록 구성되고, The edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to one or more of an input layer and an output layer of the neural network model,
    상기 클라우드 컴퓨터(200)는 상기 신경망 모델 전체에 대한 훈련을 수행하여 상기 업데이트 신경망 모델을 생성하도록 구성되는 것을 특징으로 하는 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템.The cloud computer 200 is configured to generate the updated neural network model by performing training on the entire neural network model. Edge deep learning-based vehicle safe driving system using vehicle driving state information.
  6. 청구항 2에 있어서, 3. The method according to claim 2,
    상기 엣지 컴퓨팅부(130)는 상기 신경망 모델의 입력층 및 출력층 중 하나 이상에 관련된 레이어에 대한 훈련을 수행하여 상기 신경망 모델을 튜닝하도록 구성되고, The edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to one or more of an input layer and an output layer of the neural network model,
    상기 클라우드 컴퓨터(200)는 상기 신경망 모델 전체에 대한 훈련을 수행하여 상기 업데이트 신경망 모델을 생성하도록 구성되는 것을 특징으로 하는 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템.The cloud computer 200 is configured to generate the updated neural network model by performing training on the entire neural network model. Edge deep learning-based vehicle safe driving system using vehicle driving state information.
  7. 청구항 1에 있어서, The method according to claim 1,
    상기 엣지 컴퓨팅부(130)는 상기 신경망 모델의 입력층 및 출력층 중 하나 이상에 관련된 레이어에 대한 훈련을 수행하여 상기 신경망 모델을 튜닝하도록 구성되고, The edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to one or more of an input layer and an output layer of the neural network model,
    상기 클라우드 컴퓨터(200)는 상기 신경망 모델의 은닉층에 관련된 레이어에 대한 훈련을 수행하여 상기 업데이트 신경망 모델을 생성하도록 구성되는 것을 특징으로 하는 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템.The cloud computer 200 is configured to generate the updated neural network model by performing training on a layer related to the hidden layer of the neural network model. Edge deep learning-based vehicle safe driving system using vehicle driving state information.
  8. 청구항 2에 있어서, 3. The method according to claim 2,
    상기 엣지 컴퓨팅부(130)는 상기 신경망 모델의 입력층 및 출력층 중 하나 이상에 관련된 레이어에 대한 훈련을 수행하여 상기 신경망 모델을 튜닝하도록 구성되고, The edge computing unit 130 is configured to tune the neural network model by performing training on a layer related to one or more of an input layer and an output layer of the neural network model,
    상기 클라우드 컴퓨터(200)는 상기 신경망 모델의 은닉층에 관련된 레이어에 대한 훈련을 수행하여 상기 업데이트 신경망 모델을 생성하도록 구성되는 것을 특징으로 하는 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템.The cloud computer 200 is configured to generate the updated neural network model by performing training on a layer related to the hidden layer of the neural network model. Edge deep learning-based vehicle safe driving system using vehicle driving state information.
  9. 청구항 1에 있어서, The method according to claim 1,
    상기 차량 DMS 장치(100)는, The vehicle DMS device 100,
    운전자 감시 카메라(111), 차량내부 환경센서(112), 차량주행 상태센서(113) 중 둘 이상을 구비하여 해당 차량에 대해 운전자 상태정보, 차량환경 상태정보, 차량주행 상태정보 중 둘 이상을 포함한 차량운전 상태정보를 실시간으로 획득하는 상태정보 획득부(110); A driver monitoring camera 111, a vehicle internal environmental sensor 112, and two or more of the vehicle driving state sensor 113, including two or more of driver state information, vehicle environment state information, and vehicle driving state information for the vehicle a state information acquisition unit 110 for acquiring vehicle driving state information in real time;
    상기 실시간 획득된 차량운전 상태정보를 상기 신경망 모델에 입력하여 해당 차량 주행 중의 안전운전 위협요인을 출력받고 이를 차량 운전자에게 알리는 데이터 처리부(120); a data processing unit 120 that inputs the real-time acquired vehicle driving state information to the neural network model, outputs a safe driving threat factor while driving the corresponding vehicle, and informs the vehicle driver;
    상기 신경망 모델을 내장하고, 상기 데이터 처리부(120)로부터 차량운전 상태정보를 입력받아 상기 신경망 모델에 의해 해당 차량 주행 중의 안전운전 위협요인을 도출하여 상기 데이터 처리부(120)로 출력하고 상기 차량운전 상태정보에 기초하여 상기 신경망 모델에 대한 딥러닝 훈련 중 상기 제 1 훈련 부분을 수행하여 상기 신경망 모델을 튜닝하는 엣지 컴퓨팅부(130); The neural network model is built-in, and vehicle driving state information is input from the data processing unit 120, and by the neural network model, a safe driving threat factor during vehicle driving is derived and output to the data processing unit 120, and the vehicle driving state. an edge computing unit 130 for tuning the neural network model by performing the first training part of the deep learning training for the neural network model based on the information;
    상기 상태정보 획득부(110)에 의해 실시간 획득되는 차량운전 상태정보 및 상기 데이터 처리부(120)에 의한 데이터 처리 결과를 임시 저장하는 데이터 저장부(140); a data storage unit 140 for temporarily storing the vehicle driving state information obtained in real time by the state information obtaining unit 110 and the data processing result by the data processing unit 120;
    상기 데이터 저장부(140)에 임시 저장되어 있는 데이터 및 상기 엣지 컴퓨팅부(130)가 보유하는 상기 신경망 모델의 데이터를 상기 클라우드 컴퓨터(200)에 대해 전달하고 상기 클라우드 컴퓨터(200)로부터 상기 업데이트 신경망 모델을 제공받아 상기 엣지 컴퓨팅부(130)로 전달하는 차량 통신부(150);The data temporarily stored in the data storage unit 140 and the data of the neural network model held by the edge computing unit 130 are transmitted to the cloud computer 200 and the updated neural network is transmitted from the cloud computer 200 . a vehicle communication unit 150 that receives the model and transmits it to the edge computing unit 130 ;
    상기 데이터 처리부(120)의 제어에 의해 상기 안전운전 위협요인을 차량 운전자에게 출력하는 운전자 경보부(160);a driver warning unit 160 for outputting the safe driving threat factor to a vehicle driver under the control of the data processing unit 120;
    를 포함하여 구성되는 것을 특징으로 하는 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템.Edge deep learning-based vehicle safe driving system using vehicle driving state information, characterized in that it comprises a.
  10. 청구항 1에 있어서, The method according to claim 1,
    지역 별로 설치 운영되어 상기 차량 DMS 장치(100)와 상기 클라우드 컴퓨터(200) 사이의 데이터 송수신을 수집 중계하는 로컬 서버(300);a local server 300 installed and operated for each region to collect and relay data transmission/reception between the vehicle DMS device 100 and the cloud computer 200;
    를 더 포함하여 구성되는 것을 특징으로 하는 차량운전 상태정보를 이용한 에지 딥러닝 기반의 차량 안전운전 시스템.Edge deep learning-based vehicle safe driving system using vehicle driving state information, characterized in that it further comprises a.
PCT/KR2021/003886 2021-03-29 2021-03-30 Edge-deep-learning-based vehicle safe driving system using vehicle driving state information WO2022211140A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020210040751A KR102314864B1 (en) 2021-03-29 2021-03-29 safe driving system of a vehicle by use of edge deep learning of driving status information
KR10-2021-0040751 2021-03-29

Publications (1)

Publication Number Publication Date
WO2022211140A1 true WO2022211140A1 (en) 2022-10-06

Family

ID=78228041

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2021/003886 WO2022211140A1 (en) 2021-03-29 2021-03-30 Edge-deep-learning-based vehicle safe driving system using vehicle driving state information

Country Status (2)

Country Link
KR (1) KR102314864B1 (en)
WO (1) WO2022211140A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20230139666A (en) 2022-03-28 2023-10-05 부산대학교 산학협력단 Devices and Methods for Recognizing Drowsy Driving based on Classification of Drowsy Driving Types and Ensemble Deep Learning

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018027776A (en) * 2016-08-16 2018-02-22 トヨタ自動車株式会社 Individualized adaptation of driver action prediction models
KR20190095199A (en) * 2019-07-26 2019-08-14 엘지전자 주식회사 Method, system, and vehicle for preventing drowsy driving
KR20190110068A (en) * 2018-03-19 2019-09-27 에스알아이 인터내셔널 Dynamic adaptation of deep neural networks
KR102061320B1 (en) * 2018-10-25 2019-12-31 디토닉 주식회사 Multichannel Vehicle Communication System and Method based on Machine Learning
KR20200127381A (en) * 2019-05-02 2020-11-11 한국항공대학교산학협력단 Apparatus and method for isolating the network structure of neural networks when deep learning in embedded systems

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018027776A (en) * 2016-08-16 2018-02-22 トヨタ自動車株式会社 Individualized adaptation of driver action prediction models
KR20190110068A (en) * 2018-03-19 2019-09-27 에스알아이 인터내셔널 Dynamic adaptation of deep neural networks
KR102061320B1 (en) * 2018-10-25 2019-12-31 디토닉 주식회사 Multichannel Vehicle Communication System and Method based on Machine Learning
KR20200127381A (en) * 2019-05-02 2020-11-11 한국항공대학교산학협력단 Apparatus and method for isolating the network structure of neural networks when deep learning in embedded systems
KR20190095199A (en) * 2019-07-26 2019-08-14 엘지전자 주식회사 Method, system, and vehicle for preventing drowsy driving

Also Published As

Publication number Publication date
KR102314864B1 (en) 2021-10-19

Similar Documents

Publication Publication Date Title
WO2019103197A1 (en) System for predicting traffic accident on basis of artificial intelligence and method therefor
WO2017164478A1 (en) Method and apparatus for recognizing micro-expressions through deep learning analysis of micro-facial dynamics
WO2008127465A1 (en) Real-time driving danger level prediction
WO2022211140A1 (en) Edge-deep-learning-based vehicle safe driving system using vehicle driving state information
US10373062B2 (en) Mapper component for a neuro-linguistic behavior recognition system
JP2007122362A (en) State estimation method using neural network and state estimation apparatus using neural network
WO2013048159A1 (en) Method, apparatus and computer readable recording medium for detecting a location of a face feature point using an adaboost learning algorithm
CN109360362A (en) A kind of railway video monitoring recognition methods, system and computer-readable medium
WO2022039318A1 (en) Artificial intelligence training method and system using de-identified image data
CN111126153A (en) Safety monitoring method, system, server and storage medium based on deep learning
WO2020032506A1 (en) Vision detection system and vision detection method using same
KR20210068243A (en) System for judging siuation of elevator based on Artificial intelligence
WO2013077558A1 (en) Robot-based autism diagnosis device using electroencephalogram and method therefor
CN110620760A (en) FlexRay bus fusion intrusion detection method and detection device for SVM (support vector machine) and Bayesian network
CN117644880B (en) Fusion safety protection system and control method for intelligent network-connected automobile
WO2019039659A1 (en) Emotion-based user management method and apparatuses performing same
CN107329445A (en) The method of robot behavior criterion intelligent supervision
CN106991400A (en) A kind of fire hazard smoke detecting method and device
CN112698660B (en) Driving behavior visual perception device and method based on 9-axis sensor
WO2020096111A1 (en) Deep learning-based abnormal behavior recognition device and method
Vyas et al. Federated learning based driver recommendation for next generation transportation system
CN117542027A (en) Unit disabling state monitoring method based on non-contact sensor
CN206301463U (en) Traffic safety supervisory systems
Bylykbashi et al. Effect of vehicle technical condition on real-time driving risk management in Internet of Vehicles: Design and performance evaluation of an integrated fuzzy-based system
JP2019214249A (en) Detection device, computer program, detection method, and learning model

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21935256

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21935256

Country of ref document: EP

Kind code of ref document: A1