CN109409173A - Driver's state monitoring method, system, medium and equipment based on deep learning - Google Patents
Driver's state monitoring method, system, medium and equipment based on deep learning Download PDFInfo
- Publication number
- CN109409173A CN109409173A CN201710716216.1A CN201710716216A CN109409173A CN 109409173 A CN109409173 A CN 109409173A CN 201710716216 A CN201710716216 A CN 201710716216A CN 109409173 A CN109409173 A CN 109409173A
- Authority
- CN
- China
- Prior art keywords
- information
- module
- single frames
- queue
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
Driver's state monitoring method, system, medium and equipment based on deep learning, comprising: initialisation image information collecting device and storage equipment, presupposed information handle logic, complete initialization operation and issue prompt information；Prompt information is received, the video data of characteristic information detection device and image information collecting equipment acquisition driver is triggered, extracts single frames pictorial information from video data and obtain video pictures sample；Extract the face orientation characteristic information and focus angle information in current single frames pictorial information, state-detection model is constructed according to face orientation characteristic information and focus angle information, the study of state-detection model depth is carried out according to video pictures sample, facial concern detection information is obtained according to default processing logical process face orientation characteristic information and focus angle information；Extract sign information；By in single frames pictorial information, face concern detection information deposit detection information and video data deposit queue, log information is determined according to queue generation driving condition and is stored.
Description
Technical field
The present invention relates to subject three, examination driver monitors system, more particularly to driver's state based on deep learning
Monitoring method, system, medium and equipment.
Background technique
With the propulsion of time, Chinese driver's quantity persistently increases, in addition driving school is artificial by coach in traditional technology
Cooperate Simple electronic detection alarm set and system monitoring student to carry out driving test, causes driver's culture efficiency of driving school
Lowly, the driving efficiency learning quality of driver is also unable to get guarantee, therefore, along with driver's Educational Training Service efficiency
Undesirable with effect, the problem of driving efficiency training resource growing tension, further highlights.Due to seeing in daily motor vehicles
During knowing examination detection, a critical function requirement in driver's examination detection process when driver status detects, and it is big
The state change state relation of part driver examination fault and examination personnel are close, and driving school coach is sitting in examination in traditional technology
Personnel side side by side, accurately can not accurately detect examination personnel state concern direction.
Currently, driver's detection method is mainly include the following types: sensor-based detection, such method is based primarily upon can
Wearable sensor, real-time measurement drive the acceleration information or angular velocity information of each section of human body, then according to measurement
Infomation detection driver behavior state.The shortcomings that such method is to need to carry wearable sensor, equipment cost
Higher, use is extremely inconvenient.Another kind of technology is based primarily upon the detection method of video image analysis, by directly extracting image spy
Sign and detection data.Such method the shortcomings that be that background modeling is inaccurate, directly using extract characteristic detection error
It is larger, cause erroneous detection and missing inspection more, feature robustness is lower.
It needs to carry wearable sensor in the prior art, equipment cost is higher, and inconvenient for use, there are detection values
Error is larger, causes erroneous detection and missing inspection more, and there are hardware cost height, and feature robustness is lower, and information utilization is low and examines
Survey the low technical problem of result accuracy.
Summary of the invention
High in view of the hardware cost of the above prior art, feature robustness is lower, and information utilization is low and testing result
The low technical problem of accuracy, the purpose of the present invention is to provide based on deep learning driver's state monitoring method, system,
Medium and equipment.Driver's state monitoring method based on deep learning solves hardware cost height in the prior art, algorithm
Robustness is weaker, the technical problem that information utilization is low and face action monitoring result accuracy is low,
In order to achieve the above objects and other related objects, the present invention provides a kind of driver's state prison based on deep learning
Survey method, comprising: initialisation image information collecting device and storage equipment, presupposed information handle logic, complete initialization operation
And issue prompt information；Prompt information is received, characteristic information detection device is triggered and image information collecting equipment acquires driver
Video data, from video data extract single frames pictorial information obtain video pictures sample；Extract current single frames pictorial information
In face orientation characteristic information and focus angle information, constructed according to face orientation characteristic information and focus angle information
State-detection model carries out the study of state-detection model depth according to video pictures sample, according to default processing logical process face
Portion obtains facial concern detection information towards characteristic information and focus angle information；By single frames pictorial information, face concern detection
Information is stored in detection information and video data deposit queue, is generated driving condition judgement log information according to queue and is stored.
In one embodiment of the present invention, prompt information is received, system acquisition video data is triggered according to prompt information,
Single frames pictorial information is extracted from video data and saves as picture sample, comprising: is received prompt information, is opened according to prompt information
Open camera；Obtain the video data of driver in real time with camera；Reading video data；It is extracted according to video data and time
Current single frames pictorial information；The single frames pictorial information extracted in queue obtains video pictures sample.
In one embodiment of the present invention, the face orientation characteristic information in current single frames pictorial information and concern are extracted
Point angle information constructs state-detection model according to face orientation characteristic information and focus angle information, according to video pictures
Sample carries out the study of state-detection model depth, according to default processing logical process face orientation characteristic information and focus angle
Information obtains facial concern detection information, comprising: extracts the characteristic in single frames pictorial information；Normalization characteristic data obtain spy
Levy vector；State-detection model is constructed according to feature vector；Deep learning is carried out according to video pictures sample, updates video pictures
Sample；Contrast characteristic's vector and comparison described eigenvector with observed in above-mentioned video pictures sample left B column, left-hand mirror,
Inside rear-view mirror, head-down instrumentation disk right B column, front, bow and see that eight motion characteristics such as shelves obtain analog information at right rear view mirror；It is right
Analog information sorts to obtain facial concern detection information.
In one embodiment of the present invention, by single frames pictorial information, face concern detection information deposit detection information and
Video data is stored in queue, is generated driving condition judgement log information according to queue and is stored, comprising: extraction video data,
Single frames pictorial information and face concern detection information；Video data is stored in Image Acquisition buffer queue；By single frames pictorial information
It is stored in the queue of single frames image cache；Face concern detection information is stored in algorithm output queue；According to image cache queue and calculation
The driving condition that method output queue generates driver determines log information and is stored in log library.
In one embodiment of the present invention, the present invention provides a kind of driver's status monitoring system based on deep learning
System characterized by comprising system preparation module, video pictures sample module, state detection module and queue memory module；
System preparation module handles logic for initialisation image information collecting device and storage equipment, presupposed information, completes initialization
It operates and issues prompt information；Video pictures sample module triggers characteristic information detection device and figure for receiving prompt information
As the video data of information collecting device acquisition driver, single frames pictorial information is extracted from video data and obtains video pictures sample
This, video pictures sample module is connect with system preparation module；State detection module, for extracting in current single frames pictorial information
Face orientation characteristic information and focus angle information, shape is constructed according to face orientation characteristic information and focus angle information
State detection model carries out the study of state-detection model depth according to video pictures sample, according to default processing logical process face
Facial concern detection information, state detection module and video pictures sample module are obtained towards characteristic information and focus angle information
Connection；Queue memory module, for depositing single frames pictorial information, face concern detection information deposit detection information and video data
In enqueue, driving condition judgement log information is generated according to queue and is stored, queue memory module and state detection module connect
It connects, queue memory module is connect with characteristic information module.
In one embodiment of the present invention, video pictures sample module, comprising: camera opening module, video acquisition
Module, video read module, single frames abstraction module and sample generation module；Camera opening module, for receiving prompt information,
Camera is opened according to prompt information；Video acquiring module, for obtaining the video data of driver, video in real time with camera
Module is obtained to connect with camera opening module；Single frames abstraction module, for extracting current list according to video data and time
Frame pictorial information, single frames abstraction module are connect with video read module；Sample generation module, for extracting the single frames figure in queue
Piece information obtains video pictures sample, and sample generation module is connect with single frames abstraction module.
In one embodiment of the present invention, state detection module includes: characteristic extracting module, feature vector module, mould
Type constructs module, model adjustment module, feature comparison module and status data computing module；Characteristic extracting module, for extracting
Characteristic in single frames pictorial information；Feature vector module obtains feature vector, feature vector for normalization characteristic data
Module is connect with characteristic extracting module；Model construction module, for constructing state-detection model, model construction according to feature vector
Module is connect with feature vector module；Model adjustment module, for carrying out deep learning, more new video according to video pictures sample
Picture sample, model adjustment module are connect with model construction module；Feature comparison module, for contrast characteristic's vector and comparison institute
State feature vector with observed in above-mentioned video pictures sample the right B column of left B column, left-hand mirror, inside rear-view mirror, head-down instrumentation disk,
Right rear view mirror, front bow and see that eight motion characteristics such as shelves obtain analog information, and feature comparison module and model adjustment module connect
It connects；Analog information sorting module, for the facial concern detection information that sorts to analog information to obtain, message ordering module and feature pair
It is connected than module.
In one embodiment of the present invention, queue memory module, comprising: data extraction module, Image Acquisition queue mould
Block, single frames cache module, arithmetic result module and log module；Data extraction module, for extracting video data, single frames picture
Information and face concern detection information；Image Acquisition Queue module, for video data to be stored in Image Acquisition buffer queue, figure
As acquisition Queue module is connect with data extraction module；Single frames cache module, for single frames pictorial information to be stored in single frames picture
Buffer queue, single frames cache module are connect with data extraction module；Arithmetic result module, for depositing face concern detection information
Enter algorithm output queue, arithmetic result module is connect with data extraction module；Log module, for according to image cache queue and
Algorithm output queue generates the driving condition judgement log information of driver and is stored in log library, log module and Image Acquisition team
The connection of column module, log module are connect with single frames cache module, and log module is connect with arithmetic result module.
In one embodiment of the present invention, the present invention provides a kind of computer readable storage medium, is stored thereon with meter
Calculation machine program, the program realize the driver status monitoring side provided by the invention based on deep learning when being executed by processor
Method.
In of the invention with embodiment, the present invention provides a kind of driver's status monitoring based on deep learning and sets
It is standby, comprising: processor and memory；Memory is for storing computer program, and processor is for executing the memory storage
Computer program so that driver's condition monitoring device based on deep learning executes and provided by the invention is based on deep learning
Driver's state monitoring method.
As above, driver's state monitoring method, system, medium and the equipment provided by the invention based on deep learning, tool
Have following the utility model has the advantages that the present invention is in order to realize the whole electronic monitoring of the examination of motor vehicle driving subject three and judge, driving is examined
It tries visual pursuit technology model machine and the video datas such as driver gestures is extracted by vehicle-mounted camera, utilize deep learning neural network
Equal tools carry out the computer vision algorithms make processing including Face datection, light stream detection etc., complete the inspection of driver's focus
Survey, whether body stretches out the behavioural analyses such as vehicle is outer, promote objectivity and accuracy that subject three is taken an examination, reduce human cost.
To sum up, the present invention solves that hardware cost in the prior art is high, and feature robustness is weaker, information utilization it is low with
And the technical problem that testing result accuracy is low, each frame image is checked, one second 15 frame, each frame can all have one
As a result after (one of eight movements) are transmitted to the entire examinee of three higher level equipments (the corresponding timestamp of current image and state),
Entire examinee's picture and status data are broken into compressed package and are sent to higher level equipment, sample training is used for face tracking, identifies face
Motion characteristic, for judgement act, the head pose picture obtained using in monitor video as sample database, do not need to feature into
Row design, feature strong robustness, actually detected accuracy rate are higher.
Detailed description of the invention
Fig. 1 shows a kind of flow chart of driver's state monitoring method embodiment based on deep learning of the invention.
Fig. 2 is shown as the specific flow chart of step S2 in one embodiment in Fig. 1.
Fig. 3 is shown as the specific flow chart of step S3 in one embodiment in Fig. 1.
Fig. 4 is shown as the specific flow chart of step S5 in one embodiment in Fig. 1.
Fig. 5 is shown as a kind of driver's condition monitoring system module embodiments signal based on deep learning of the invention
Figure.
Fig. 6 is shown as the specific module diagram of video pictures sample module 12 in one embodiment in Fig. 5.
Fig. 7 is shown as the specific module diagram of state detection module 13 in one embodiment in Fig. 5.
Fig. 8 is shown as in Fig. 5 the specific module diagram in one embodiment of data extraction module 15.
Component label instructions
1 driver's condition monitoring system based on deep learning
11 system preparation modules
12 video pictures sample modules
13 state detection modules
14 queue memory modules
121 camera opening modules
122 video acquiring modules
123 video read modules
124 single frames abstraction modules
125 sample generation modules
131 characteristic extracting modules
132 feature vector modules
133 model construction modules
134 model adjustment modules
135 feature comparison modules
136 status data computing modules
151 data extraction modules
152 Image Acquisition Queue modules
153 single frames cache modules
154 arithmetic result modules
155 log modules
Step numbers explanation
S1~S5 method and step
S21~S26 method and step
S31~S36 method and step
S51~S55 method and step
Specific embodiment
Embodiments of the present invention are illustrated by particular specific embodiment below, those skilled in the art can be by this explanation
Content disclosed by book is understood other advantages and efficacy of the present invention easily.
Fig. 1 is please referred to Fig. 8, it should however be clear that this specification structure depicted in this specification institute accompanying drawings, only to cooperate specification institute
The content of announcement is not intended to limit the invention enforceable qualifications so that those skilled in the art understands and reads,
Therefore not having technical essential meaning, the modification of any structure, the change of proportionate relationship or the adjustment of size are not influencing this reality
It can be contained under novel the effect of can be generated and the purpose that can reach, should all still fall in disclosed technology contents
In the range of lid.Meanwhile in this specification it is cited such as " on ", " under ", " left side ", " right side ", " centre " and " one " term,
It is merely convenient to being illustrated for narration, rather than to limit the scope of the invention, relativeness is altered or modified,
It is changed under technology contents without essence, when being also considered as the enforceable scope of the present invention.
Referring to Fig. 1, showing a kind of stream of driver's state monitoring method embodiment based on deep learning of the invention
Cheng Tu, such as Fig. 1, the method, comprising: driver's state monitoring method based on deep learning is related to driving based on deep learning
Sail people's state monitoring method characterized by comprising
Step S1, initialisation image information collecting device and storage equipment, presupposed information handle logic, complete initialization behaviour
Make and issue prompt information, user is monitored on the client terminals such as the control plate of system, computer by installation pilot's line of vision
Installation detection and setting are carried out automatically by pressing system start button open system, system in system main interface, and to camera shooting
The hardware devices such as head and storage disk are initialized；
Step S2, prompt information is received, characteristic information detection device is triggered and image information collecting equipment acquires driver
Video data, extract single frames pictorial information from video data and obtain video pictures sample, receive processing triggering information, according to
Processing triggering information triggers system acquisition video data, and single frames pictorial information is extracted from video data and saves as image analysis
Sample, by being installed on the eye video data for driving indoor camera and acquiring driver, by the single frames figure in video data
Picture information is stored as image analysis sample, and video data is stored in SD card；
Step S3, the face orientation characteristic information and focus angle information in current single frames pictorial information are extracted, according to
Face orientation characteristic information and focus angle information construct state-detection model, carry out state-detection according to video pictures sample
Model depth study obtains facial concern according to default processing logical process face orientation characteristic information and focus angle information and examines
Measurement information；
Step S4, single frames pictorial information, face concern detection information deposit detection information and video data are stored in queue
In, driving condition judgement log information is generated according to queue and is stored, deep neural network will be passed through and handle the sight inspection obtained
Measurement information is converted into data flow and stores into response queue, and generates pilot's line of vision according to line-of-sight detection information and detect log
And it is stored in server end.
Referring to Fig. 2, being shown as the specific flow chart of step S2 in one embodiment in Fig. 1, specifically include:
Step S21, prompt information is received, camera is opened according to prompt information, user passes through master control interface power on operation
The power supply of open system hardware device, the cursor that pilot's line of vision detection system is clicked in the main interface of mobile terminal are opened,
System hardware equipment mainly includes several cameras being installed on driver's cabin driver's seat and driver relative position, Driving Test system
Equipment does not provide operation interface in deployment.System software is arranged under Ubuntu system autostart catalogue soft during installation
Part self-starting configuration.Hardware powers on, and starting script is executed when Ubuntu system starts, automatic to start Driving Test system program；
Step S22, it obtains the video data of driver in real time with camera, opens camera and video counts are carried out to driver
According to acquisition, original USB camera video data is obtained from camera；
Step S23, reading video data, camera obtain driver in driving procedure by photosensitive imaging element in real time
Video image, and the video data that camera shooting obtains is sent to image procossing by way of data/address bus or wireless transmission and patrols
Volume；
Step S24, current single frames pictorial information, pilot's line of vision detection system root are extracted according to video data and time
Video information is handled according to preset image processing logic, obtains single frames original size picture, compressed format picture, it is preferred that right
The single frames picture of generation is used for image algorithm library and carried out accordingly by the video data that camera obtains according to timestamp sub-frame processing
Analysis, and picture compression storage is used for report generation；
Step S25, the single frames pictorial information extracted in queue obtains video pictures sample, extracts from image storage queue
Single frames pictorial information is simultaneously collected as image analysis sample, and the image analysis sample is for training deep neural network model.
Referring to Fig. 3, being shown as the specific flow chart of step S3 in one embodiment in Fig. 1, specifically include:
Step S31, the characteristic in single frames pictorial information is extracted, sight image in head to be measured and pose presentation are passed through
Pretreatment, extracts head local feature vectors and global header feature vector and fusion obtains global characteristics vector；
Step S32, normalization characteristic data obtain feature vector, extract part to treated head image data set
Then set of eigenvectors merges local feature vectors collection to obtain head pose feature vector；
Step S33, state-detection model is constructed according to feature vector；
Step S34, deep learning is carried out according to video pictures sample, video pictures sample is updated, to image analysis sample
In every head pose picture pre-processed, obtain with picture to be measured pre-process information；
Step S35, contrast characteristic's vector and comparison described eigenvector with observe left B in above-mentioned video pictures sample
Column, left-hand mirror, inside rear-view mirror, head-down instrumentation disk right B column, front, bow and see that eight motion characteristics such as shelves obtain at right rear view mirror
Analog information, according to the picture to be measured of image analysis sample pre-process acquisition of information sample in include sample global characteristics to
It measures, above-mentioned Palestine and China's action data is contained in global characteristics vector, by the global characteristics information of sample and feature vector pair to be measured
Than obtaining cosine similarity information, the global characteristics vector of image analysis sample carries out pre- mainly for following three scene of subject
If:
Scene one: before starting, do not observe in, outside rear-view mirror later observes rear traffic conditions.Before starting, observation is left
Right rear view mirror: head deflects 30 degree to the left, and driver does not observe left-hand mirror；Head deflects to the left is greater than 30 degree；In viewing
Rearview mirror, head, which deflects to the right, is greater than 30 degree, and head tilt angle is greater than 30 degree；Left back, head deflects to the left is greater than 60
Degree, above situation determine in violation of rules and regulations.
Scene two: it is more than 2 seconds that sight, which leaves driving direction,.In vehicle travel process, pilot's line of vision leaves front,
When the amesiality duration is more than two seconds, determine in violation of rules and regulations.
Scene three: it bows in driving process and sees shelves.It in driving process, bows and sees that shelves are more than 2 seconds, bow when seeing grade, head
The duration that portion deflects to the right greater than 30 degree is more than 2 seconds, and angle of bowing is greater than 30 degree, and the duration is more than 2 seconds, is determined
In violation of rules and regulations.
Scene four: during vehicle driving turning, road traffic condition is not observed by left-hand mirror；Open left steering
After lamp, if examinee does not observe left-hand mirror, head does not deflect 30 degree to 60 degree to the left, determines in violation of rules and regulations.
Scene five: during vehicle driving turning, road traffic condition is not observed by right rear view mirror；Open right turn
Deng after, if examinee does not observe right rear view mirror, head does not deflect to the right 45 degree to 60 degree, determines in violation of rules and regulations.
Scene six: it before change lane, not by inside and outside backsight sem observation, and is put after later being observed to change lane direction
Road traffic condition；It receives after " change lane " phonetic order or driver opens after turn signal in certain time length, if not having
Outside rear-view mirror and the rear accordingly surveyed in observing, when head deflection is greater than 60 degree, determine in violation of rules and regulations.
Scene seven: before parking, not by inside and outside backsight sem observation rear and right side traffic conditions, and confirmation is later observed
After the turn on right turn lamp of safety, during speed is down to 0, if driver does not observe inside rear-view mirror, right rear view mirror, it is right after
Side determines in violation of rules and regulations.
Scene eight: needing to get off, and does not observe left back traffic conditions later before opening car door；When speed is 0, if
Driver determines in violation of rules and regulations when before opening car door without observation left back；
S36, sort to obtain to analog information facial concern detection information.
Referring to Fig. 4, being shown as the specific flow chart of step S5 in one embodiment in Fig. 1, specifically include:
S51, video data, single frames pictorial information and face concern detection information are extracted, from camera and image data
Adjustment method output end extracts video data, single frames pictorial information and video surveillance information；
S52, video data is stored in Image Acquisition buffer queue, Image Acquisition buffer queue is protected for image data storage
Deposit the processed image data of image algorithm, for Driving Test after summarizing and reporting and back up before the deadline, deposited from image
Single frames pictorial information is extracted in storage queue and is collected as image analysis sample, and the image analysis sample is for training depth neural
Network model；
S53, single frames pictorial information is stored in the queue of single frames image cache, single frames image cache queue is predominantly rolled up and depth
The input rank of network model algorithm, video data are the queue element (QE) of Image Acquisition buffer queue, are suitable for according to video data
Input data of the sequence of enqueue as image data processing algorithm；
S54, face concern detection information is stored in algorithm output queue, algorithm output queue is image algorithm resume module
Result queue, for reporting higher level equipment, suitable for being transmitted to server end according to the sequence uplink of video surveillance information enqueue
It is checked for monitoring personnel；
S55, log information is determined simultaneously according to the driving condition that image cache queue and algorithm output queue generate driver
It is stored in log library, the work log generated in the operation of system log memory retention system, management driving condition determines log information
And it is sent to server end.
Implement referring to Fig. 5, being shown as a kind of driver's condition monitoring system module based on deep learning of the invention
It illustrates and is intended to, the system comprises: system preparation module 11, video pictures sample module 12, state detection module 13 and queue
Memory module 14；System preparation module 11 is set for initialisation image information collecting device, sign information detection device and storage
Standby, presupposed information handles logic, completes initialization operation and issues prompt information, and user passes through installation pilot's line of vision monitoring system
In system main interface by pressing system start button open system, system on the client terminals such as the control plate of system, computer
Automatically installation detection and setting are carried out, and the hardware devices such as camera and storage disk are initialized；Video pictures sample
Module 12 triggers the body of characteristic information detection device and image information collecting equipment acquisition driver for receiving prompt information
Reference breath and video data extract single frames pictorial information from video data and obtain video pictures sample, receive processing triggering letter
Breath triggers system acquisition video data according to processing triggering information, single frames pictorial information is extracted from video data and is saved as
Image analysis sample will be in video data by being installed on the eye video data for driving indoor camera and acquiring driver
Single-frame images information stored as image analysis sample, and video data is stored in SD card, video pictures sample module
12 connect with system preparation module 11；State detection module 13, for extracting the spy of the face orientation in current single frames pictorial information
Reference breath and focus angle information construct state-detection model according to face orientation characteristic information and focus angle information,
The study of state-detection model depth is carried out according to video pictures sample, according to default processing logical process face orientation characteristic information
Facial concern detection information is obtained with focus angle information, state detection module 13 is connect with video pictures sample module 12；Team
Column memory module 14, for single frames pictorial information, face concern detection information deposit detection information and video data to be stored in team
In column, driving condition judgement log information is generated according to queue and is stored, deep neural network will be passed through and handle the sight obtained
Detection information is converted into data flow and stores into response queue, and generates pilot's line of vision according to line-of-sight detection information and detect day
Will is simultaneously stored in server end, and queue memory module 14 is connect with state detection module 13.
Referring to Fig. 6, it is shown as the specific module diagram of video pictures sample module 12 in one embodiment in Fig. 5,
It specifically includes: camera opening module 121, video acquiring module 122, video read module 123,124 and of single frames abstraction module
Sample generation module 125；Camera opening module 121 is opened camera according to prompt information, is used for receiving prompt information
Family passes through the power supply of master control interface power on operation open system hardware device, and driver's view is clicked in the main interface of mobile terminal
The cursor of line detection system is opened, and Driving Test system equipment does not provide operation interface in deployment.System software exists during installation
Software self-starting is arranged under Ubuntu system autostart catalogue to configure.Hardware powers on, and executes and opens when Ubuntu system starts
Dynamic script, it is automatic to start Driving Test system program；Video acquiring module 122, for obtaining the video of driver in real time with camera
Data, open camera and carry out video data acquiring to driver, obtain original USB camera video data from camera；Depending on
Frequency read module 123, is used for reading video data, and camera obtains driver in driving procedure by photosensitive imaging element in real time
In video image, and the video data that camera shooting obtains is sent to image procossing by way of data/address bus or wireless transmission and patrols
Volume, video read module 123 is connect with video acquiring module 122；Single frames abstraction module 124, for timely according to video data
Between extract current single frames pictorial information, pilot's line of vision detection system handles video according to preset image processing logic and believes
Breath, obtain single frames original size picture, compressed format picture, it is preferred that camera obtain video data according to timestamp
The single frames picture of generation is used for image algorithm library and carries out corresponding analysis, and picture compression is stored for reporting by sub-frame processing
It generates, single frames abstraction module 124 is connect with video read module 123；Sample generation module 125, for extracting the list in queue
Frame pictorial information obtains video pictures sample, and single frames pictorial information is extracted from image storage queue and is collected as image analysis sample
This, for training deep neural network model, sample generation module 125 and single frames abstraction module 124 connect the image analysis sample
It connects.
Referring to Fig. 7, being shown as the specific module diagram of state detection module 13 in one embodiment in Fig. 5, specifically
It include: characteristic extracting module 131, feature vector module 132, model construction module 133, model adjustment module 134, aspect ratio pair
Module 135 and status data computing module 136；Characteristic extracting module 131, for extracting the characteristic in single frames pictorial information
According to by sight image in head to be measured and pose presentation by pretreatment, extraction head local feature vectors and global header feature
Vector and merge obtain global characteristics vector；Feature vector module 132 obtains feature vector for normalization characteristic data, right
Treated head image data set extracts local feature vectors collection, then merges local feature vectors collection to obtain head appearance
State feature vector, feature vector module 132 are connect with characteristic extracting module 131；Model construction module 133, for according to feature
Vector constructs state-detection model, and model construction module 133 connect 132 with feature vector module；Model adjustment module 134 is used
Deep learning is carried out according to video pictures sample in root, video pictures sample is updated, to every head in image analysis sample
Posture picture is pre-processed, and is obtained and is pre-processed information with picture to be measured, and model adjustment module 134 and model construction module 133 connect
Connect feature comparison module 135, for contrast characteristic's vector with comparison described eigenvector with seen in above-mentioned video pictures sample
The right B column of left B column, left-hand mirror, inside rear-view mirror, head-down instrumentation disk is examined, right rear view mirror, front, bows and sees shelves etc. eight movements
Feature obtains analog information, and it is global special that the sample for including in acquisition of information sample is pre-processed according to the picture to be measured of image analysis sample
Levy vector, contain above-mentioned Palestine and China's action data in global characteristics vector, by the global characteristics information of sample and feature to be measured to
Amount comparison obtains cosine similarity information, and feature comparison module 135 is connect with model adjustment module 134；Analog information sequence mould
Block 136, for the facial concern detection information that sorts to analog information to obtain, yearning between lovers message ordering module 136 and feature comparison module
135 connections.
Referring to Fig. 8, being shown as in Fig. 5 the specific module diagram in one embodiment of data extraction module 15, specifically
It include: data extraction module 151, Image Acquisition Queue module 152, single frames cache module 153, arithmetic result module 154 and day
Will module 155；Data extraction module 151, for extracting video data, single frames pictorial information and face concern detection information, from
Camera and image data processing algorithm output end extract video data, single frames pictorial information and video surveillance information；Image is adopted
Collect Queue module 152, for video data to be stored in Image Acquisition buffer queue, Image Acquisition buffer queue is used for image data
Storage saves the processed image data of image algorithm, for Driving Test after summarizing and reporting and back up before the deadline, from
Single frames pictorial information is extracted in image storage queue and is collected as image analysis sample, and the image analysis sample is deep for training
Neural network model is spent, Image Acquisition Queue module 152 is connect with data extraction module 151；Single frames cache module 153, is used for
Single frames pictorial information is stored in the queue of single frames image cache, and single frames image cache queue is mainly volume and depth network model algorithm
Input rank, video data are the queue element (QE) of Image Acquisition buffer queue, suitable for the sequence work according to video data enqueue
For the input data of image data processing algorithm, single frames cache module 153 is connect with data extraction module 151；Arithmetic result mould
Block 154, for face concern detection information to be stored in algorithm output queue, algorithm output queue is image algorithm resume module knot
Fruit queue is supplied for reporting higher level equipment suitable for being transmitted to server end according to the sequence uplink of video surveillance information enqueue
Monitoring personnel checks that arithmetic result module 154 is connect with data extraction module 151；Log module 155, for slow according to picture
It deposits queue and algorithm output queue generates the driving condition judgement log information of driver and is stored in log library, system log storage
The work log generated in the operation of preservation system, management driving condition determine log information and are sent to server end, log mould
Block 155 is connect with Image Acquisition Queue module 152, and log module 155 is connect with single frames cache module 154, log module 155 with
Arithmetic result module 154 connects.
In one embodiment of the present invention, the present invention provides a kind of computer readable storage medium, is stored thereon with meter
Calculation machine program, the program realize the driver status monitoring side provided by the invention based on deep learning when being executed by processor
Method, those of ordinary skill in the art will appreciate that: realize that all or part of the steps of above-mentioned each method embodiment can pass through meter
Calculation machine program relevant hardware is completed.Computer program above-mentioned can be stored in a computer readable storage medium.It should
When being executed, execution includes the steps that above-mentioned each method embodiment to program；And storage medium above-mentioned includes: ROM, RAM, magnetic disk
Or the various media that can store program code such as CD.
In of the invention with embodiment, the present invention provides a kind of driver's status monitoring based on deep learning and sets
It is standby, comprising: processor and memory；Memory is for storing computer program, and processor is by executing based on memory storage
Calculation machine program, so that driver's condition monitoring device based on deep learning executes driving based on deep learning provided by the invention
People's state monitoring method is sailed, memory may include random access memory (RandomAccessMemory, abbreviation RAM),
It may further include nonvolatile memory (non-volatilememory), a for example, at least magnetic disk storage.Above-mentioned place
Reason device can be general processor, including central processing unit (CentralProcessingUnit, abbreviation CPU), network processing unit
(NetworkProcessor, abbreviation NP) etc.；It can also be digital signal processor (DigitalSignalProcessing, letter
Claim DSP), specific integrated circuit (ApplicationSpecificIntegratedCircuit, abbreviation ASIC), scene can compile
Journey gate array (Field- ProgrammableGateArray, abbreviation FPGA) or other programmable logic device, discrete gate
Or transistor logic, discrete hardware components.
In conclusion driver's state monitoring method provided by the invention based on deep learning, system, medium and setting
It is standby.The invention has the following advantages: the present invention in order to realize motor vehicle driving subject three take an examination whole electronic monitoring with
It judges, driving test visual pursuit technology model machine extracts the video datas such as driver gestures by vehicle-mounted camera, utilizes depth
The tools such as learning neural network carry out the computer vision algorithms make processing including Face datection, light stream detection etc., complete to drive
Whether the detection of the person's of sailing focus, body stretch out the behavioural analyses such as vehicle is outer, promote objectivity and accuracy that subject three is taken an examination, reduce
Human cost.In the examination of motor vehicle driving subject three, Driving Test system acquires examinee's driving video using camera, completes examinee
The detection of face orientation, to confirm the possible object observing of examinee；The detection that left front window region is stretched out whether there is or not object outside vehicle is completed,
To confirm examinee, whether there is or not physical feelings to stretch out outside vehicle；Camera imaging quality evaluation is completed, whether there is or not other articles to block with confirmation
Camera.Complete driver's focus, body whether there is or not stretch out outside window, after whether camera the detection such as cover, Driving Test system meeting
According to the communication protocol for (being finally completed Driving Test Subject and close the equipment that rule are adjudicated) agreement with higher level equipment, correlated condition is reported
To higher level equipment, to help higher level equipment to complete Driving Test Subject judgement.To sum up, the present invention solve hardware in the prior art at
This height, feature robustness is weaker, the technical problem that information utilization is low and testing result accuracy is low, to obtain in monitor video
The head pose picture taken does not need to be designed feature, feature strong robustness as sample database, actually detected accuracy rate compared with
Height has very high commercial value and practicability.
Claims (10)
1. a kind of driver's state monitoring method based on deep learning characterized by comprising
Initialisation image information collecting device and storage equipment, presupposed information handle logic, complete initialization operation and issue to mention
Show information；
The prompt information is received, the view of the characteristic information detection device and image information collecting equipment acquisition driver is triggered
Frequency evidence extracts single frames pictorial information from the video data and obtains video pictures sample；
The face orientation characteristic information and focus angle information in presently described single frames pictorial information are extracted, according to the face
State-detection model is constructed towards characteristic information and focus angle information, the state is carried out according to the video pictures sample
Detection model deep learning, according to face orientation characteristic information and focus angle information described in the default processing logical process
Obtain the face concern detection information；
The single frames pictorial information, the face concern detection information deposit detection information and video data are stored in queue,
Driving condition judgement log information is generated according to the queue and is stored.
2. being believed the method according to claim 1, wherein described receive the prompt information according to the prompt
Breath triggering system acquisition video data extracts single frames pictorial information from the video data and saves as picture sample, comprising:
The prompt information is received, the camera is opened according to prompt information；
Obtain the video data of driver in real time with camera；
Read the video data；
The current single frames pictorial information is extracted according to the video data and time；
The single frames pictorial information extracted in the queue obtains the video pictures sample.
3. the method according to claim 1, wherein the face extracted in presently described single frames pictorial information
Towards characteristic information and focus angle information, state is constructed according to the face orientation characteristic information and focus angle information
Detection model carries out the state-detection model depth according to the video pictures sample and learns, and is patrolled according to the default processing
It collects the processing face orientation characteristic information and focus angle information obtains the face concern detection information, comprising:
Extract the characteristic in the single frames pictorial information；
It normalizes the characteristic and obtains feature vector；
The state-detection model is constructed according to described eigenvector；
Deep learning is carried out according to the video pictures sample, updates the video pictures sample；
Comparison described eigenvector with observed in above-mentioned video pictures sample left B column, left-hand mirror, inside rear-view mirror, overlook instrument
Dial plate right B column, front, bows and sees that eight motion characteristics such as shelves obtain analog information at right rear view mirror；
It sorts to obtain the facial concern detection information to the analog information.
4. according to right want 1 described in method, which is characterized in that it is described by the single frames pictorial information, it is described face concern inspection
Measurement information is stored in detection information and video data deposit queue, is generated driving condition according to the queue and is determined log information simultaneously
Storage, comprising:
Extract the video data, the single frames pictorial information and the face concern detection information；
The video data is stored in Image Acquisition buffer queue；
The single frames pictorial information is stored in the queue of single frames image cache；
The face concern detection information is stored in algorithm output queue；
Log letter is determined according to the driving condition that the image cache queue and the algorithm output queue generate the driver
It ceases and is stored in log library.
5. a kind of driver's condition monitoring system based on deep learning characterized by comprising system preparation module, video
Picture sample module, state detection module, sign information module and queue memory module；
The system preparation module, for initialisation image information collecting device and storage equipment, presupposed information handles logic, complete
At initialization operation and issue prompt information；
The video pictures sample module triggers the characteristic information detection device and image for receiving the prompt information
Information collecting device acquires the video data of driver, and single frames pictorial information is extracted from the video data and obtains video pictures
Sample；
The state detection module, for extracting face orientation characteristic information and focus in presently described single frames pictorial information
Angle information constructs state-detection model according to the face orientation characteristic information and focus angle information, according to the view
Frequency picture sample carries out the state-detection model depth study, according to the spy of face orientation described in the default processing logical process
Reference breath and focus angle information obtain the face concern detection information；
The queue memory module, for the single frames pictorial information, the face concern detection information to be stored in detection information
In video data deposit queue, driving condition judgement log information is generated according to the queue and is stored.
6. system according to claim 5, which is characterized in that the video pictures sample module, comprising: camera is opened
Module, video acquiring module, video read module, single frames abstraction module and sample generation module；
The camera opening module opens the camera according to prompt information for receiving the prompt information；
The video acquiring module, for obtaining the video data of driver in real time with camera；
The video read module, for reading the video data；
The single frames abstraction module, for extracting the current single frames pictorial information according to the video data and time；
The sample generation module, the single frames pictorial information for extracting in the queue obtain the video pictures sample.
7. system according to claim 5, which is characterized in that the state detection module includes: characteristic extracting module, spy
Levy vector module, model construction module, model adjustment module, feature comparison module and status data computing module；
The characteristic extracting module, for extracting the characteristic in the single frames pictorial information；
Described eigenvector module obtains feature vector for normalizing the characteristic；
The model construction module, for constructing the state-detection model according to described eigenvector；
The model adjustment module updates the video pictures sample for carrying out deep learning according to the video pictures sample
This；
The feature comparison module, for compare described eigenvector and comparison described eigenvector with above-mentioned video pictures sample
The right B column of left B column, left-hand mirror, inside rear-view mirror, head-down instrumentation disk is observed in this, right rear view mirror, front, bows and sees shelves etc. eight
A motion characteristic obtains analog information；
The analog information sorting module, for the face concern detection information that sorts to the analog information to obtain.
8. system according to claim 5, which is characterized in that the queue memory module, comprising: data extraction module,
Image Acquisition Queue module, single frames cache module, arithmetic result module and log module；
The data extraction module, for extracting the video data, the single frames pictorial information and the face concern detection
Information；
Described image acquires Queue module, for the video data to be stored in Image Acquisition buffer queue；
The single frames cache module, for the single frames pictorial information to be stored in the queue of single frames image cache；
The arithmetic result module, for the face concern detection information to be stored in algorithm output queue；
The log module, for generating driving for the driver according to the image cache queue and the algorithm output queue
State is sailed to determine log information and be stored in log library.
9. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is held by processor
Driver's state monitoring method described in any one of claims 1 to 4 based on deep learning is realized when row.
10. a kind of driver's condition monitoring device based on deep learning characterized by comprising processor and memory；
The memory is used to execute the computer journey of the memory storage for storing computer program, the processor
Sequence, so that driver's condition monitoring device execution based on deep learning is based on as described in any one of claims 1 to 4
Driver's state monitoring method of deep learning.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710716216.1A CN109409173B (en) | 2017-08-18 | 2017-08-18 | Driver state monitoring method, system, medium and equipment based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710716216.1A CN109409173B (en) | 2017-08-18 | 2017-08-18 | Driver state monitoring method, system, medium and equipment based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109409173A true CN109409173A (en) | 2019-03-01 |
| CN109409173B CN109409173B (en) | 2021-06-04 |
Family
ID=65463290
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710716216.1A Active CN109409173B (en) | 2017-08-18 | 2017-08-18 | Driver state monitoring method, system, medium and equipment based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109409173B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110618635A (en) * | 2019-10-08 | 2019-12-27 | 中兴飞流信息科技有限公司 | Train cab operation specification monitoring system based on AI technology |
| CN111724119A (en) * | 2019-09-26 | 2020-09-29 | 中国石油大学(华东) | Efficient automatic data annotation auditing method |
| CN113901895A (en) * | 2021-09-18 | 2022-01-07 | 武汉未来幻影科技有限公司 | Door opening action recognition method and device for vehicle and processing equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102830793A (en) * | 2011-06-16 | 2012-12-19 | 北京三星通信技术研究有限公司 | Sight tracking method and sight tracking device |
| CN104637246A (en) * | 2015-02-02 | 2015-05-20 | 合肥工业大学 | Driver multi-behavior early warning system and danger evaluation method |
| CN105354986A (en) * | 2015-11-12 | 2016-02-24 | 熊强 | Driving state monitoring system and method for automobile driver |
| CN106446811A (en) * | 2016-09-12 | 2017-02-22 | 北京智芯原动科技有限公司 | Deep-learning-based driver's fatigue detection method and apparatus |
| CN106599994A (en) * | 2016-11-23 | 2017-04-26 | 电子科技大学 | Sight line estimation method based on depth regression network |
| CN107123340A (en) * | 2017-06-30 | 2017-09-01 | 无锡合壮智慧交通有限公司 | A kind of method of automatic detection driver observation state |
-
2017
- 2017-08-18 CN CN201710716216.1A patent/CN109409173B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102830793A (en) * | 2011-06-16 | 2012-12-19 | 北京三星通信技术研究有限公司 | Sight tracking method and sight tracking device |
| CN104637246A (en) * | 2015-02-02 | 2015-05-20 | 合肥工业大学 | Driver multi-behavior early warning system and danger evaluation method |
| CN105354986A (en) * | 2015-11-12 | 2016-02-24 | 熊强 | Driving state monitoring system and method for automobile driver |
| CN106446811A (en) * | 2016-09-12 | 2017-02-22 | 北京智芯原动科技有限公司 | Deep-learning-based driver's fatigue detection method and apparatus |
| CN106599994A (en) * | 2016-11-23 | 2017-04-26 | 电子科技大学 | Sight line estimation method based on depth regression network |
| CN107123340A (en) * | 2017-06-30 | 2017-09-01 | 无锡合壮智慧交通有限公司 | A kind of method of automatic detection driver observation state |
Non-Patent Citations (1)
| Title |
|---|
| 赵峰: "实现道路驾驶技能考试自动化的技术基础", 《交通信息化》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111724119A (en) * | 2019-09-26 | 2020-09-29 | 中国石油大学(华东) | Efficient automatic data annotation auditing method |
| CN110618635A (en) * | 2019-10-08 | 2019-12-27 | 中兴飞流信息科技有限公司 | Train cab operation specification monitoring system based on AI technology |
| CN113901895A (en) * | 2021-09-18 | 2022-01-07 | 武汉未来幻影科技有限公司 | Door opening action recognition method and device for vehicle and processing equipment |
| CN113901895B (en) * | 2021-09-18 | 2022-09-27 | 武汉未来幻影科技有限公司 | Door opening action recognition method and device for vehicle and processing equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109409173B (en) | 2021-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105354988B (en) | A kind of driver tired driving detecting system and detection method based on machine vision | |
| CN108549854B (en) | A kind of human face in-vivo detection method | |
| CN109284738B (en) | Irregular face correction method and system | |
| US10963741B2 (en) | Control device, system and method for determining the perceptual load of a visual and dynamic driving scene | |
| CN105528577B (en) | Recognition methods based on intelligent glasses | |
| CN107071344A (en) | A kind of large-scale distributed monitor video data processing method and device | |
| CN109409172A (en) | Pilot's line of vision detection method, system, medium and equipment | |
| WO2018171875A1 (en) | Control device, system and method for determining the perceptual load of a visual and dynamic driving scene | |
| CN108596041A (en) | A kind of human face in-vivo detection method based on video | |
| CN104134364B (en) | Real-time traffic sign identification method and system with self-learning capacity | |
| CN109409173A (en) | Driver's state monitoring method, system, medium and equipment based on deep learning | |
| CN113920491A (en) | Fatigue detection system, method, medium and detection device based on facial skeleton model | |
| CN108664887A (en) | Prior-warning device and method are fallen down in a kind of virtual reality experience | |
| CN109543651A (en) | A kind of driver's dangerous driving behavior detection method | |
| CN113705349A (en) | Attention power analysis method and system based on sight estimation neural network | |
| CN109670517A (en) | Object detection method, device, electronic equipment and target detection model | |
| CN108983966B (en) | Criminal reconstruction assessment system and method based on virtual reality and eye movement technology | |
| CN106295474A (en) | The fatigue detection method of deck officer, system and server | |
| CN111832373A (en) | Automobile driving posture detection method based on multi-view vision | |
| CN114332911A (en) | Head posture detection method and device and computer equipment | |
| CN115841651A (en) | Constructor intelligent monitoring system based on computer vision and deep learning | |
| CN107527060B (en) | Refrigerating device storage management system and refrigerating device | |
| CN106981169A (en) | One kind race of bowing cycles safety monitoring and method for warming | |
| CN110443221A (en) | A kind of licence plate recognition method and system | |
| CN109426757A (en) | Driver's head pose monitoring method, system, medium and equipment based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |