CN111127699A - Method, system, equipment and medium for automatically recording automobile defect data - Google Patents
Method, system, equipment and medium for automatically recording automobile defect data Download PDFInfo
- Publication number
- CN111127699A CN111127699A CN201911164828.XA CN201911164828A CN111127699A CN 111127699 A CN111127699 A CN 111127699A CN 201911164828 A CN201911164828 A CN 201911164828A CN 111127699 A CN111127699 A CN 111127699A
- Authority
- CN
- China
- Prior art keywords
- defect
- automobile
- data
- image
- identification information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000007547 defect Effects 0.000 title claims abstract description 243
- 238000000034 method Methods 0.000 title claims abstract description 58
- 230000002950 deficient Effects 0.000 claims abstract description 24
- 210000000746 body region Anatomy 0.000 claims description 29
- 239000013598 vector Substances 0.000 claims description 20
- 238000003860 storage Methods 0.000 claims description 17
- 230000015654 memory Effects 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 13
- 238000013479 data entry Methods 0.000 claims description 9
- 238000012790 confirmation Methods 0.000 claims description 5
- 238000012550 audit Methods 0.000 abstract description 19
- 230000008569 process Effects 0.000 abstract description 14
- 238000005516 engineering process Methods 0.000 abstract description 10
- 238000007689 inspection Methods 0.000 description 32
- 230000006870 function Effects 0.000 description 8
- 230000003068 static effect Effects 0.000 description 8
- 238000012549 training Methods 0.000 description 8
- 239000011521 glass Substances 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000005236 sound signal Effects 0.000 description 6
- 238000012360 testing method Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 3
- 239000012141 concentrate Substances 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012552 review Methods 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000005111 flow chemistry technique Methods 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000003973 paint Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000010845 search algorithm Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 238000013179 statistical model Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000011179 visual inspection Methods 0.000 description 1
- 230000002618 waking effect Effects 0.000 description 1
- 238000003466 welding Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C5/00—Registering or indicating the working of vehicles
- G07C5/08—Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
- G07C5/0808—Diagnosing performance data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C5/00—Registering or indicating the working of vehicles
- G07C5/08—Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
- G07C5/0841—Registering performance data
- G07C5/085—Registering performance data using electronic data carriers
- G07C5/0866—Registering performance data using electronic data carriers the electronic data carrier being a digital video recorder in combination with video camera
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/06—Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice
- G10L15/063—Training
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an automatic recording method, system, equipment and medium of automobile defect data, wherein the method comprises the following steps: receiving audio data of a user; performing voice recognition on the audio data, and determining defect information corresponding to the audio data; acquiring a defect area image of a defect position of an automobile; identifying the image of the defect area and determining the identification information of the defect area; storing the defect information and identification information of the defective area. The invention provides an automobile audit auxiliary scheme, which provides an intelligent auxiliary function for an audit process by combining technologies such as voice recognition, image recognition and the like, automatically inputs defect information and identification information of a defect area, and improves audit efficiency and accuracy.
Description
Technical Field
The invention relates to the technical field of automobile audit, in particular to an automatic automobile defect data entry method, system, equipment and medium.
Background
Automobile Audit, namely, a specially trained reviewer independently stands on the product using standpoint of a user, and performs random sampling evaluation on vehicles which are determined to be qualified according to automobile quality evaluation standards and with professional and comprehensive eyes. The result of automotive audio is an objective manifestation of vehicle manufacturing quality levels and capabilities. The quality inspection department analyzes the reasons of the defects according to the quality defect problems found by spot inspection, implements the responsibility department, and further adopts rectification measures to eliminate the defects, thereby achieving the purpose of gradually improving the quality of the products. In the automobile audio work flow, because the number of inspection items is large, the inspection requirement is strict, and the time consumption is long usually.
The current automotive audio workflow is shown in fig. 1 and can be roughly divided into two types, namely static inspection and dynamic inspection. Static inspection is carried out in indoor light canopy, and main work content is: inspectors inspect defects such as pits, convex hulls, vehicle paint, gaps and the like through visual inspection, touch and the like or by means of a measuring tool; the dynamic inspection is mainly carried out outdoors, and in the process of testing the running of the vehicle on a test road section, an inspector listens to the sound emitted by the vehicle and parts and judges whether the defects of part noise, abnormal sound and the like exist. The dynamic inspection mainly aims at the finished automobile Audit, and the static inspection has requirements in the processes of finished automobile Audit, body-in-white Audit, stamping, welding Audit and the like.
According to the working mode at the present stage, after an inspector finds a defect, the vehicle needs to be stopped firstly for dynamic inspection, a special pen needs to be used for marking the position of the defect on a vehicle body or a part for static inspection, and then a paper pen is used for temporarily recording defect information, including the defect position, a defect code, a defect grade and the like. After the recording is complete, the inspector continues to inspect the next location or component. After all the positions are inspected, the inspector returns to a computer in an office area, and each piece of defect information is input into the system one by using a keyboard according to a defect list temporarily recorded during inspection.
Because the defects to be inspected in the automobile audio process are various, the problems caused by the above operation modes are as follows: (1) the defect information entry process takes longer time and affects the review efficiency, and (2) because of the secondary entry problem, there may be a less noticeable risk of entry errors and affect the review result.
Disclosure of Invention
Aiming at the problems in the prior art, the invention aims to provide a method, a system, equipment and a medium for automatically inputting automobile defect data based on deep learning, which provide an intelligent auxiliary function for an auditing process by combining technologies such as voice recognition and image recognition, automatically input defect information and identification information of a defect area, and improve auditing efficiency and accuracy.
The embodiment of the invention provides an automatic entry method of automobile defect data, which comprises the following steps:
receiving audio data of a user;
performing voice recognition on the audio data, and determining defect information corresponding to the audio data;
acquiring a defect area image of a defect position of an automobile;
carrying out defect area identification on the defect area image, and determining identification information of the defect area;
storing the defect information and identification information of the defective area.
Optionally, the performing voice recognition on the audio data includes the following steps:
inputting the audio data into a trained acoustic feature model, and extracting acoustic features in the audio data;
inputting the acoustic features in the audio data into a trained voice recognition model, and recognizing text recognition results corresponding to the acoustic features;
and extracting a defect attribute value corresponding to a preset defect attribute from the text recognition result to be used as defect information corresponding to the audio data.
Optionally, the performing defect area identification on the defect area image includes the following steps:
and comparing the defect area image with a plurality of prestored vehicle body area images, and selecting the identification information of the vehicle body area image with the maximum similarity with the defect area image as the identification information of the defect area.
Optionally, the comparing the defect area image with a plurality of pre-stored vehicle body area images includes:
inputting the defect area image into a trained feature extraction network, and extracting a feature vector of the defect area image;
and comparing the feature vectors of the defective region images with the feature vectors of a plurality of vehicle body region images in a vehicle body region feature library, and selecting identification information corresponding to the feature vector with the maximum similarity as the identification information of the defective region.
Optionally, the identification information of the vehicle body region image includes a number of the vehicle body region, a name of the vehicle body region, a part number corresponding to the vehicle body region, or a part name corresponding to the vehicle body region.
Optionally, the acquiring a defect area image of a defect position of the automobile includes the following steps:
displaying a light spot indicating a shooting position in front of a user;
and when a voice shooting instruction of a user is received, acquiring a defect area image at the position corresponding to the light spot.
Optionally, the displaying the spot indicating the shooting position in front of the user comprises displaying the spot indicating the shooting position in a display panel at a lens of the AR device worn by the user;
when a shooting voice command of a user is received through a microphone arranged on the AR equipment, acquiring a defect area image at a position corresponding to the light spot through a camera arranged on the AR equipment.
Optionally, the storing the defect information and the identification information of the defective area includes:
displaying the defect information and the identification information of the defect area in a display panel of the AR device;
and when receiving a voice confirmation instruction of a user, storing the defect information and the identification information of the defect area.
The embodiment of the invention also provides an automatic entry system of the automobile defect data, which is applied to the automatic entry method of the automobile defect data, and the system comprises the following components:
the audio acquisition module is used for receiving audio data of a user;
the audio recognition module is used for carrying out voice recognition on the audio data and determining the defect information corresponding to the audio data;
the image acquisition module is used for acquiring a defect area image of a defect position of the automobile;
the image identification module is used for identifying the defective area of the defective area image and determining the identification information of the defective area;
and the data entry module is used for storing the defect information and the identification information of the defect area.
Optionally, the audio acquisition module and the image acquisition module are arranged in an AR device, and a camera and a microphone are arranged on the AR device;
the audio acquisition module receives audio data of a user through the microphone;
the image acquisition module acquires a defect area image of a defect position of the automobile through the camera.
Optionally, a display screen is further disposed at the lens of the AR device;
the image acquisition module is also used for displaying light spots indicating shooting positions in front of the user and controlling the camera to acquire the images of the defect areas of the defect positions of the automobile when the audio recognition module recognizes the shooting voice command of the user.
The embodiment of the invention also provides an automatic recording device for the automobile defect data, which comprises:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the automatic entry method of automobile defect data via execution of the executable instructions.
The embodiment of the invention also provides a computer-readable storage medium for storing a program, wherein when the program is executed, the steps of the automatic entry method of the automobile defect data are realized.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
The method, the system, the equipment and the medium for automatically inputting the automobile defect data have the following advantages:
the invention solves the problems in the prior art, provides an automobile audit auxiliary scheme, provides an intelligent auxiliary function for the audit process by combining technologies such as voice recognition and image recognition, automatically inputs the defect information and the identification information of the defect area, simplifies the work flow, and simultaneously enables inspectors to concentrate on the inspection more without changing the original operation habits of the inspectors, thereby improving the audit efficiency; compared with a pure manual mode in the prior art, the method avoids errors possibly caused by secondary input, and avoids the problem of easy error in the prior art, so that the auditing accuracy is improved.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings.
FIG. 1 is a flow diagram of a prior art automobile review;
FIG. 2 is a flow chart of a method for automatically entering defect data of a vehicle according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of an automatic entry system for vehicle defect data according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a system for implementing automatic entry of vehicle defect data by an AR device according to an embodiment of the present invention;
FIG. 5 is a flow chart of a vehicle audit after adding a method for automatic entry of vehicle defect data in accordance with one embodiment of the present invention;
FIG. 6 is a flow diagram of speech recognition according to an embodiment of the present invention;
FIG. 7 is a flow chart of image recognition according to an embodiment of the present invention;
FIG. 8 is a schematic view of an automatic entry device for defect data of a vehicle according to an embodiment of the present invention;
fig. 9 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
In order to solve the technical problems in the prior art, an embodiment of the present invention provides an automatic entry method for automobile defect data, where the defect data mainly includes defect information and a defect area image. As shown in fig. 2, in this embodiment, the automatic entry method for the automobile defect data includes the following steps:
s100: receiving audio data of a user;
s200: performing voice recognition on the audio data, and determining defect information corresponding to the audio data, wherein the defect information can comprise information such as a defect code, a severity degree, a defect type, a responsibility department and the like;
s300: acquiring a defect area image of a defect position of an automobile;
s400: carrying out defect area identification on the defect area image, and determining identification information of the defect area;
s500: storing the defect information and identification information of the defective area. Here, the defect information and the identification information of the defect area may be stored locally, uploaded to a server, and stored.
The identification information of the defective region may include a name of the vehicle body region corresponding to the defective region, a number of the vehicle body region corresponding to the defective region, a name of the part corresponding to the defective region, or a number of the part corresponding to the defective region.
The invention provides an intelligent auxiliary function for the auditing process by combining technologies such as voice recognition, image recognition and the like, wherein the defect information is automatically recognized by combining the voice recognition technology in steps S100 and S200, the image recognition technology is combined in steps S300 and S400, the image of the defect area is automatically acquired and the identification information of the defect area is recognized, and the defect information and the identification information of the defect area are automatically input in step S500, so that the working flow is simplified, meanwhile, inspectors can concentrate on the inspection more, the original operation habits of the inspectors are not changed, and the auditing efficiency is improved. Because the data entry is directly carried out after the defect information is detected and the defect area image is identified, errors easily caused by secondary entry in the prior art are avoided, and the accuracy of audit is improved.
As shown in fig. 3, an embodiment of the present invention further provides an automatic entry system for automobile defect data, which is applied to the automatic entry method for automobile defect data, and the system includes:
the audio acquisition module M100 is used for receiving audio data of a user;
the audio recognition module M200 is configured to perform voice recognition on the audio data, and determine defect information corresponding to the audio data;
the image acquisition module M300 is used for acquiring a defect area image of a defect position of the automobile;
the image identification module M400 is configured to perform defect area identification on the defect area image, and determine identification information of the defect area;
and a data entry module M500, configured to store the defect information and the identification information of the defective area.
When carrying out the defect detection, audio acquisition module M100 with audio recognition module M200 combines the speech recognition technique, and automatic identification defect information, image acquisition module M300 and image recognition module M400 combine the image recognition technique, and the regional image of automatic acquisition defect to the regional identification information of automatic identification determination defect, and data entry module M500 can type in defect information and the regional identification information of defect automatically, has simplified work flow, can make the inspection personnel more be concentrated on the inspection simultaneously, and does not change the original operation custom of inspection personnel, thereby has improved audit efficiency.
Further, as shown in fig. 4, in this embodiment, an AR (Augmented Reality) glasses and a server (e.g., a cloud server) are used to implement the automatic entry method and system for car defect data. Specifically, the audio acquisition module M100 and the image acquisition module M300 in the automatic automobile defect data entry system are arranged in AR glasses, and the AR glasses are further provided with a microphone, a camera and a display screen. The audio acquisition module M100 receives audio data of a user through the microphone; the image acquisition module M300 acquires a defect area image of a defect position of the automobile through the camera. The audio recognition module M200, the image recognition module M400, and the data entry module M500 are disposed in a server, and the AR glasses and the server may communicate via a network to transmit data such as images and audio. Therefore, the inspectors only need to wear the AR glasses during working, and the normal inspection work of the inspectors cannot be influenced.
In this embodiment, a display screen is further disposed at the lens of the AR glasses. The image capturing module M300 is further configured to display a light spot indicating a shooting position in front of the user after the audio recognition module M200 recognizes the defect information, and control the camera to capture a defect area image of the defect position of the vehicle when the audio recognition module recognizes a shooting voice command of the user.
In other embodiments, the audio acquisition module M100 and the image acquisition module M300 in the automatic automobile defect data entry system may also be disposed in other types of wearable AR devices, and a high-definition camera, a semitransparent display panel, an earphone, a microphone, a bluetooth module, a WIFI module, or other data transmission modules are disposed inside the system. The interaction with the cloud server can be realized through the data transmission module. The data transmission module can also receive signals such as voice, images and videos from the cloud server, sound signals are played through the earphones, the pictures or the video information are displayed through the semitransparent display panel, and the pictures or the video information and the real scenes are overlaid to the sight line of the user. By means of the AR equipment, man-machine interaction can be realized more conveniently and efficiently, and the working efficiency is improved.
The cloud server can receive sound, image or video signals from the AR equipment, analyze and store the sound, can extract corresponding text information from the sound signals by applying a voice recognition technology, receives related voice commands, and can extract useful information such as object types and positions from the images and videos by applying a computer vision technology. Meanwhile, the cloud server can send the sound, image and video signals needing to be fed back to the user to the AR device, and therefore human-computer interaction is achieved.
As shown in fig. 5, it is a flow chart of car audit after the car defect data automatic entry method of this embodiment is adopted. The specific process is as follows:
(1) and (4) defect inspection: after the indoor static inspection or the outdoor dynamic inspection is started, an inspector performs defect inspection (static inspection) on a certain inspection area or part, or drives a test vehicle to run on a test road section (dynamic inspection);
(2) marking defects: after the defects are found, the special pen is used for static inspection and inspection personnel to mark the defects on the vehicle body, and the vehicle is stopped for an outdoor dynamic test emergency stop or after the road test is finished;
(3) and (3) waking up the system: the inspector awakens the automatic automobile defect entry system through the voice keyword, and the system starts to receive subsequent voice signals and performs real-time analysis;
(4) information entry: the inspector dictates defect information, such as defect type, severity, department of responsibility, and the like. Receiving audio data through an AR device microphone, transmitting the audio data to a cloud server, and obtaining related defect information after voice recognition;
(5) defect photographing, storing and positioning: the inspector can start the functions of defect photographing, storing and positioning through a specific voice keyword command. After the function is started, the AR equipment displays light spots in the display panel, and the inspector moves the head to enable the light spots to coincide with the defect positions in the real scene and sends out a photographing instruction through voice. The scene in the sight of the inspection personnel is collected by a camera on the AR equipment and is transmitted to the cloud server in the form of the image of the defect area. The cloud server divides the image of the defect area according to different vehicle body areas or parts, such as a vehicle door, a front cover and the like, and compares and identifies the division result of the area where the light spot is located with a plurality of vehicle body area images prestored in a vehicle body area image library, so that identification information of the position area where the defect is located is obtained;
(6) and (3) defect information confirmation: (4) and (5) feeding back the defect information input in the step (5) including a defect code, a defect position, a responsibility department, a defect picture and the like to an inspector for confirmation in the form of pictures or voice through the AR equipment. If the error content is found in the defect information, returning to (4) or (5) for correction;
specifically, in this embodiment, the step S500 of storing the defect information and the defect area image includes the following steps:
displaying the defect information and the identification information of the defect area in a display panel of the AR glasses; if a confirmation voice command of a user is received, storing the defect information and the identification information of the defect area; and if a modification instruction of the user is received, returning to the step S100 or returning to the step S300 for modification.
Further, in step S500, the defect data such as the defect area image and the audio data of the inspector may be stored.
(6) After the defect information is confirmed to be correct, the information is input into the system and added into a defect list;
(7) and (3) for static inspection, after the information is recorded, returning to the step (1) to continuously inspect the next defect of the area or inspect the next area until all defect inspection is recorded.
As shown in fig. 6, in this embodiment, in the step S200, performing speech recognition on the audio data includes the following steps:
inputting the audio data into a trained acoustic feature model, and extracting acoustic features in the audio data;
inputting the acoustic features in the audio data into a trained voice recognition model, and recognizing text recognition results corresponding to the acoustic features;
and extracting a defect attribute value corresponding to a preset defect attribute from the text recognition result to be used as defect information corresponding to the audio data.
Specifically, in this embodiment, the voice recognition of the audio data in step S200 includes the following steps:
for voice commands or defect information received by the microphone of the AR equipment, voice recognition technology is needed to be used for converting audio signals into characters for triggering related functions or storing the defect information. In the present invention, the flow of the speech recognition algorithm is shown in fig. 6, and the main steps include:
(1) training an acoustic feature model: and (4) using the voice library data set, carrying out feature extraction on the voice library data set, and then training an acoustic feature model. In the speech recognition application process, the same feature extraction algorithm is used for extracting feature parameters of the speech to be recognized, and the feature parameters are matched with the acoustic feature model, so that a speech recognition result can be obtained. In the present invention, the acoustic feature Model may be a Hidden Markov Model (HMM), an LSTM (Long Short-Term Memory), a CTC (connecting temporal classification based on a neural network), or the like.
(2) Training a voice recognition model: and (3) using the Chinese text database to perform grammar and semantic analysis on the Chinese text database, and training based on a statistical model to obtain a voice recognition model. The speech recognition modeling can effectively combine knowledge of Chinese grammar and semantics to describe the internal relation between words, thereby improving the recognition rate, reducing the search range and improving the search efficiency.
(3) Inputting an audio signal: and (3) manually designing a dictionary (namely a pronunciation model), and completing the preparation stage of the system after obtaining the trained acoustic feature model and the trained voice recognition model. The audio signal can be input into an automatic recording system of the automobile defect data for voice recognition.
The above is a process of training the acoustic feature model and the speech recognition model, which may be performed before step S100 to obtain a trained acoustic feature model and a trained speech recognition model that can be used.
(4) A preprocessing module: firstly, the input original voice signal is preprocessed by algorithms such as high-pass filtering and the like, and the unimportant information and the background noise in the signal are filtered and eliminated.
(5) Feature extraction: and extracting key characteristic parameters capable of reflecting the characteristics of the voice signals to form a characteristic vector sequence. In the invention, MFCC (Mel Frequency Cepstrum Coefficient) features are selected to form a feature vector of a voice signal. For example, a speech waveform is divided into frames of about 25ms in length, MFCC features are extracted from each frame, 39 dimensions in total, and feature vectors representing the frame are obtained.
(6) Speech decoding and searching: i.e. a speech recognition procedure. Aiming at the input audio signal, establishing a WFST (weighted finite state transducer) search space according to a trained HMM acoustic feature model, a trained voice recognition model and a trained dictionary, and searching an optimal path with the maximum matching probability in the search space by using a search algorithm to obtain an optimal text recognition result.
The method of speech recognition is presented in one embodiment only, and the various model algorithms used are only examples. In other alternative embodiments, method changes, selection of other types of model algorithms, etc. may be performed, and are within the scope of the present invention.
As shown in fig. 7, in this embodiment, in the step S400, performing defect area identification on the defect area image includes the following steps:
and comparing the defect area image with a plurality of prestored vehicle body area images, and selecting the identification information of the vehicle body area image with the maximum similarity with the defect area image as the identification information of the defect area. The identification information of the vehicle body region image may include a number of the vehicle body region, a name of the vehicle body region, a part number corresponding to the vehicle body region, a part name corresponding to the vehicle body region, and the like.
In this embodiment, the comparing the defect area image with a plurality of pre-stored vehicle body area images includes:
inputting the defect area image into a trained feature extraction network, and extracting a feature vector of the defect area image;
and calculating the similarity of the feature vectors of the defective region image and each vehicle body region image, wherein the similarity can adopt methods such as Euclidean distance and cosine similarity.
Specifically, in this embodiment, as shown in fig. 7, the image recognition of the defective area image in step S400 includes the following steps:
and (3) using the AR equipment camera to photograph the defect area, and after the picture information is transmitted to the cloud server, the cloud server applies a related computer vision algorithm to identify the defect position (the part or the area where the part belongs). The identification process is shown in fig. 7, and mainly includes the following steps:
(1) training a feature extraction network: using a plurality of body region images in the body region image library, a CNN (Convolutional Neural Networks) network is trained for feature extraction of a body region where a defect may exist. The network input is a vehicle body area image, and the network input is a feature vector corresponding to the part. The training target is that the feature vectors of different vehicle body regions have the largest inter-class distance and separability.
(2) Establishing a vehicle body region feature library: and (3) performing feature extraction on all vehicle body region images which are possibly defective by using the trained feature extraction network to obtain corresponding feature vectors to form a vehicle body region feature library.
The above (1) and (2) are processes of model training and feature library preparation before image recognition, and need to be performed at least before step S300, and further before step S100.
(3) Extracting edges of an input picture: and for the defect position picture shot by the AR equipment, extracting the edge points of the defect position picture by using a correlation algorithm, such as a canny operator and the like.
(4) Image segmentation: and (3) segmenting the image after the edge extraction by using an image segmentation algorithm based on an image connected domain to obtain a contour map of the part to which the defect belongs or the region to which the defect belongs.
(5) Feature extraction: using the trained feature extraction network to extract features of the segmented image to obtain a feature vector of the segmented image;
(6) and (3) feature matching and identification: and comparing and matching the feature vector of the image of the defect area with the features in the feature library of the vehicle body area, searching the feature vector of the vehicle body area with the maximum similarity, and outputting the identification information corresponding to the feature vector to serve as the identification information of the defect area, thereby completing the identification of the defect position.
The embodiment of the invention also provides automatic recording equipment of the automobile defect data, which comprises a processor; a memory having stored therein executable instructions of the processor; wherein the processor is configured to perform the steps of the automatic entry method of automobile defect data via execution of the executable instructions.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" platform.
An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 8. The electronic device 600 shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 8, the electronic device 600 is embodied in the form of a general purpose computing device. The combination of the electronic device 600 may include, but is not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting different platform combinations (including memory unit 620 and processing unit 610), a display unit 640, etc.
Wherein the storage unit stores program code executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present invention described in the above-mentioned electronic prescription flow processing method section of the present specification. For example, the processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, to name a few.
The embodiment of the invention also provides a computer-readable storage medium for storing a program, wherein when the program is executed, the steps of the automatic entry method of the automobile defect data are realized. In some possible embodiments, aspects of the present invention may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above-mentioned electronic prescription flow processing method section of this specification, when the program product is run on the terminal device.
Referring to fig. 9, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
In summary, compared with the prior art, the method, the system, the device and the medium for automatically inputting the automobile defect data provided by the invention have the following advantages:
the invention solves the problems in the prior art, provides an automobile audit auxiliary scheme, provides an intelligent auxiliary function for the audit process by combining technologies such as voice recognition and image recognition, automatically inputs the defect information and the identification information of the defect area, simplifies the work flow, and simultaneously enables inspectors to concentrate on the inspection more without changing the original operation habits of the inspectors, thereby improving the audit efficiency; compared with a pure manual mode in the prior art, the method avoids errors possibly caused by secondary input, and avoids the problem of easy error in the prior art, so that the auditing accuracy is improved.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (13)
1. An automatic entry method for automobile defect data is characterized by comprising the following steps:
receiving audio data of a user;
performing voice recognition on the audio data, and determining defect information corresponding to the audio data;
acquiring a defect area image of a defect position of an automobile;
carrying out defect area identification on the defect area image, and determining identification information of the defect area;
storing the defect information and identification information of the defective area.
2. The automatic entry method for the automobile defect data as claimed in claim 1, wherein the voice recognition of the audio data comprises the following steps:
inputting the audio data into a trained acoustic feature model, and extracting acoustic features in the audio data;
inputting the acoustic features in the audio data into a trained voice recognition model, and recognizing text recognition results corresponding to the acoustic features;
and extracting a defect attribute value corresponding to a preset defect attribute from the text recognition result to be used as defect information corresponding to the audio data.
3. The automatic entry method for the automobile defect data according to claim 1, wherein the defect area identification of the defect area image comprises the following steps:
and comparing the defect area image with a plurality of prestored vehicle body area images, and selecting the identification information of the vehicle body area image with the maximum similarity with the defect area image as the identification information of the defect area.
4. The automatic entry method for the automobile defect data according to claim 3, wherein the step of comparing the defect area image with a plurality of pre-stored automobile body area images comprises the following steps:
inputting the defect area image into a trained feature extraction network, and extracting a feature vector of the defect area image;
and comparing the feature vectors of the defective region images with the feature vectors of a plurality of vehicle body region images in a vehicle body region feature library, and selecting identification information corresponding to the feature vector with the maximum similarity as the identification information of the defective region.
5. The automatic entry method for the vehicle defect data according to claim 3, wherein the identification information of the vehicle body region image comprises a number of a vehicle body region, a name of a vehicle body region, a part number corresponding to a vehicle body region or a part name corresponding to a vehicle body region.
6. The automatic entry method for the defect data of the automobile according to claim 1, wherein the step of acquiring the defect area image of the defect position of the automobile comprises the following steps:
displaying a light spot indicating a shooting position in front of a user;
and when a voice shooting instruction of a user is received, acquiring a defect area image at the position corresponding to the light spot.
7. The automatic entry method of car defect data according to claim 6, wherein said displaying a spot of light indicative of a shooting location in front of a user comprises displaying a spot of light indicative of a shooting location in a display panel at a lens of an AR device worn by the user;
when a shooting voice command of a user is received through a microphone arranged on the AR equipment, acquiring a defect area image at a position corresponding to the light spot through a camera arranged on the AR equipment.
8. The automatic entry method for vehicle defect data according to claim 7, wherein said storing said defect information and said identification information of said defect area comprises the steps of:
displaying the defect information and the identification information of the defect area in a display panel of the AR device;
and when receiving a voice confirmation instruction of a user, storing the defect information and the identification information of the defect area.
9. An automatic entry system for automobile defect data, which is applied to the automatic entry method for automobile defect data of any one of claims 1 to 8, the system comprising:
the audio acquisition module is used for receiving audio data of a user;
the audio recognition module is used for carrying out voice recognition on the audio data and determining the defect information corresponding to the audio data;
the image acquisition module is used for acquiring a defect area image of a defect position of the automobile;
the image identification module is used for identifying the defective area of the defective area image and determining the identification information of the defective area;
and the data entry module is used for storing the defect information and the identification information of the defect area.
10. The automatic entry system for the automobile defect data of claim 9, wherein the audio acquisition module and the image acquisition module are arranged in an AR device, and the AR device is provided with a camera and a microphone;
the audio acquisition module receives audio data of a user through the microphone;
the image acquisition module acquires a defect area image of a defect position of the automobile through the camera.
11. The automatic entry system for automobile defect data of claim 10, wherein a display screen is further disposed at the lens of the AR device;
the image acquisition module is also used for displaying light spots indicating shooting positions in front of the user and controlling the camera to acquire the images of the defect areas of the defect positions of the automobile when the audio recognition module recognizes the shooting voice command of the user.
12. An automatic entry device for automobile defect data, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the automatic entry method of automobile defect data of any one of claims 1 to 8 via execution of the executable instructions.
13. A computer-readable storage medium storing a program, wherein the program, when executed, implements the steps of the automatic entry method of vehicle defect data of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911164828.XA CN111127699A (en) | 2019-11-25 | 2019-11-25 | Method, system, equipment and medium for automatically recording automobile defect data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911164828.XA CN111127699A (en) | 2019-11-25 | 2019-11-25 | Method, system, equipment and medium for automatically recording automobile defect data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111127699A true CN111127699A (en) | 2020-05-08 |
Family
ID=70496587
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911164828.XA Pending CN111127699A (en) | 2019-11-25 | 2019-11-25 | Method, system, equipment and medium for automatically recording automobile defect data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111127699A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111709352A (en) * | 2020-06-12 | 2020-09-25 | 浪潮集团有限公司 | Vehicle scratch detection method based on neural network |
| CN112365468A (en) * | 2020-11-11 | 2021-02-12 | 南通大学 | AA-gate-Unet-based offshore wind power tower coating defect detection method |
| CN112669822A (en) * | 2020-12-16 | 2021-04-16 | 爱驰汽车有限公司 | Audio processing method and device, electronic equipment and storage medium |
| CN112819055A (en) * | 2021-01-25 | 2021-05-18 | 北京巅峰科技有限公司 | Vehicle defect identification method, device, equipment and storage medium |
| CN113221710A (en) * | 2021-04-30 | 2021-08-06 | 深圳市水务工程检测有限公司 | Neural network-based drainage pipeline defect identification method, device, equipment and medium |
| WO2021189647A1 (en) * | 2020-03-27 | 2021-09-30 | 深圳光启超材料技术有限公司 | Multimedia information determination method, head-mounted device, storage medium, and electronic device |
| CN117576102A (en) * | 2024-01-17 | 2024-02-20 | 长春汽车工业高等专科学校 | Automobile defect detection method and system based on reflection spectrum |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107292901A (en) * | 2017-07-24 | 2017-10-24 | 北京小米移动软件有限公司 | Edge detection method and device |
| CN108564940A (en) * | 2018-03-20 | 2018-09-21 | 平安科技（深圳）有限公司 | Audio recognition method, server and computer readable storage medium |
| CN208239883U (en) * | 2018-04-27 | 2018-12-14 | 重庆车云金服科技股份有限公司 | Vehicle failure intelligent checking system |
| CN109163914A (en) * | 2018-11-12 | 2019-01-08 | 金瓜子科技发展（北京）有限公司 | A kind of vehicle remote detection method and device |
| CN109788089A (en) * | 2018-10-16 | 2019-05-21 | 华为技术有限公司 | The method and terminal of microspur imaging |
| CN110148106A (en) * | 2019-01-18 | 2019-08-20 | 华晨宝马汽车有限公司 | A kind of system and method using deep learning model inspection body surface defect |
| CN110335238A (en) * | 2019-05-08 | 2019-10-15 | 吉林大学 | A kind of Automotive Paint Film Properties defect recognition system constituting method of deep learning |
-
2019
- 2019-11-25 CN CN201911164828.XA patent/CN111127699A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107292901A (en) * | 2017-07-24 | 2017-10-24 | 北京小米移动软件有限公司 | Edge detection method and device |
| CN108564940A (en) * | 2018-03-20 | 2018-09-21 | 平安科技（深圳）有限公司 | Audio recognition method, server and computer readable storage medium |
| CN208239883U (en) * | 2018-04-27 | 2018-12-14 | 重庆车云金服科技股份有限公司 | Vehicle failure intelligent checking system |
| CN109788089A (en) * | 2018-10-16 | 2019-05-21 | 华为技术有限公司 | The method and terminal of microspur imaging |
| CN109163914A (en) * | 2018-11-12 | 2019-01-08 | 金瓜子科技发展（北京）有限公司 | A kind of vehicle remote detection method and device |
| CN110148106A (en) * | 2019-01-18 | 2019-08-20 | 华晨宝马汽车有限公司 | A kind of system and method using deep learning model inspection body surface defect |
| CN110335238A (en) * | 2019-05-08 | 2019-10-15 | 吉林大学 | A kind of Automotive Paint Film Properties defect recognition system constituting method of deep learning |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021189647A1 (en) * | 2020-03-27 | 2021-09-30 | 深圳光启超材料技术有限公司 | Multimedia information determination method, head-mounted device, storage medium, and electronic device |
| CN111709352A (en) * | 2020-06-12 | 2020-09-25 | 浪潮集团有限公司 | Vehicle scratch detection method based on neural network |
| CN111709352B (en) * | 2020-06-12 | 2022-10-04 | 浪潮集团有限公司 | Vehicle scratch detection method based on neural network |
| CN112365468A (en) * | 2020-11-11 | 2021-02-12 | 南通大学 | AA-gate-Unet-based offshore wind power tower coating defect detection method |
| CN112669822A (en) * | 2020-12-16 | 2021-04-16 | 爱驰汽车有限公司 | Audio processing method and device, electronic equipment and storage medium |
| CN112669822B (en) * | 2020-12-16 | 2022-11-25 | 爱驰汽车有限公司 | Audio processing method and device, electronic equipment and storage medium |
| CN112819055A (en) * | 2021-01-25 | 2021-05-18 | 北京巅峰科技有限公司 | Vehicle defect identification method, device, equipment and storage medium |
| CN113221710A (en) * | 2021-04-30 | 2021-08-06 | 深圳市水务工程检测有限公司 | Neural network-based drainage pipeline defect identification method, device, equipment and medium |
| CN117576102A (en) * | 2024-01-17 | 2024-02-20 | 长春汽车工业高等专科学校 | Automobile defect detection method and system based on reflection spectrum |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111127699A (en) | Method, system, equipment and medium for automatically recording automobile defect data | |
| US11393191B2 (en) | Method and system for obtaining vehicle target views from a video stream | |
| CN108962282B (en) | Voice detection analysis method and device, computer equipment and storage medium | |
| US10522136B2 (en) | Method and device for training acoustic model, computer device and storage medium | |
| CN107945792B (en) | Voice processing method and device | |
| CN108573701A (en) | Inquiry based on lip detecting is endpoint formatting | |
| EP3839942A1 (en) | Quality inspection method, apparatus, device and computer storage medium for insurance recording | |
| US10643605B2 (en) | Automatic multi-performance evaluation system for hybrid speech recognition | |
| US11676257B2 (en) | Method and device for detecting defect of meal box, server, and storage medium | |
| CN110909613A (en) | Video character recognition method and device, storage medium and electronic equipment | |
| CN102023703B (en) | Combined lip reading and voice recognition multimodal interface system | |
| Tawari et al. | Speech emotion analysis in noisy real-world environment | |
| CN108305618B (en) | Voice acquisition and search method, intelligent pen, search terminal and storage medium | |
| US20210224752A1 (en) | Work support system and work support method | |
| CN109835260B (en) | Vehicle information display method, device, terminal and storage medium | |
| Sahoo et al. | Emotion recognition from audio-visual data using rule based decision level fusion | |
| US9311930B2 (en) | Audio based system and method for in-vehicle context classification | |
| CN107578770B (en) | Voice recognition method and device for network telephone, computer equipment and storage medium | |
| CN111199541A (en) | Image quality evaluation method, image quality evaluation device, electronic device, and storage medium | |
| WO2020056995A1 (en) | Method and device for determining speech fluency degree, computer apparatus, and readable storage medium | |
| EP4020312B1 (en) | Traffic light recognition method, apparatus, storage medium and program product | |
| CN112309398B (en) | Method and device for monitoring working time, electronic equipment and storage medium | |
| CN113822187A (en) | Sign language translation, customer service, communication method, device and readable medium | |
| Slaney et al. | Gaze-enhanced speech recognition | |
| CN110705523B (en) | Entrepreneur performance evaluation method and system based on neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200508 |
|
| WD01 | Invention patent application deemed withdrawn after publication |