CN110852133A - Automatic walking equipment, control method and control device thereof, and computer equipment - Google Patents
Automatic walking equipment, control method and control device thereof, and computer equipment Download PDFInfo
- Publication number
- CN110852133A CN110852133A CN201810840595.XA CN201810840595A CN110852133A CN 110852133 A CN110852133 A CN 110852133A CN 201810840595 A CN201810840595 A CN 201810840595A CN 110852133 A CN110852133 A CN 110852133A
- Authority
- CN
- China
- Prior art keywords
- images
- user
- automatic walking
- image information
- sound source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 239000013598 vector Substances 0.000 claims abstract description 103
- 230000009471 action Effects 0.000 claims abstract description 62
- 238000012549 training Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 8
- 230000004913 activation Effects 0.000 claims description 7
- 238000010801 machine learning Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 6
- 230000004807 localization Effects 0.000 claims description 5
- 238000013434 data augmentation Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 238000003062 neural network model Methods 0.000 claims description 3
- 230000002452 interceptive effect Effects 0.000 claims 1
- 230000008569 process Effects 0.000 abstract description 8
- 230000008859 change Effects 0.000 abstract description 7
- 238000004364 calculation method Methods 0.000 abstract description 3
- 239000000284 extract Substances 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 4
- 230000001815 facial effect Effects 0.000 description 4
- 238000010276 construction Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000009966 trimming Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 241000203475 Neopanax arboreus Species 0.000 description 1
- 241001417527 Pempheridae Species 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 238000011065 in-situ storage Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000005406 washing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D34/00—Mowers; Mowing apparatus of harvesters
- A01D34/006—Control or measuring arrangements
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0212—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
- G05D1/0221—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory involving a learning process
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
- G05D1/0253—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0276—Control of position or course in two dimensions specially adapted to land vehicles using signals provided by a source external to the vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Aviation & Aerospace Engineering (AREA)
- Multimedia (AREA)
- Automation & Control Theory (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Environmental Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Electromagnetism (AREA)
- Life Sciences & Earth Sciences (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Manipulator (AREA)
Abstract
The invention relates to automatic walking equipment, a control method, a control device and computer equipment thereof, wherein the automatic walking equipment acquires a plurality of images corresponding to the current action of a user and respectively extracts corresponding first feature vectors to obtain N images corresponding to the current action and N corresponding first feature vectors, combines the first feature vector corresponding to the N image with N-1 first feature vectors corresponding to the previous N-1 images to generate a second feature vector, and controls the state of the automatic walking equipment according to the second feature vector, so that the technical problem of low convenience caused by a key mode or a voice instruction mode in the prior art is solved, and the calculation amount in the process of gesture recognition or other body posture change recognition is reduced.
Description
Technical Field
The invention relates to the field of garden processes, in particular to automatic walking equipment, a control method, a control device and computer equipment thereof.
Background
With the development of science and technology, the automatic walking equipment usually has the ability of autonomous walking, and does not need to be controlled and operated manually and directly when the automatic walking equipment is used for executing work. Taking an intelligent mower as an example, the existing intelligent mower can usually finish the lawn trimming work independently, greatly reduces manual operation, and is a tool suitable for lawn trimming and maintenance in places such as family courtyards and public greenbelts.
In the conventional technology, a user controls the automatic walking device in a key pressing mode such as a start key and a stop key or a voice instruction mode. Because the key-press mode requires that a user closely contacts the automatic walking equipment, and the voice instruction has the problems of language, dialect communication obstacle and over-far sound source and high noise, the human-computer interaction between the existing automatic walking equipment and the user has the technical problem of low convenience.
Disclosure of Invention
Therefore, it is necessary to provide an automatic walking device, a control method thereof, a control device thereof, and a computer device, in order to solve the technical problem in the prior art that the convenience of human-computer interaction of the automatic walking device is not high.
A method of controlling an automated walking device, the method comprising: acquiring a plurality of images corresponding to the current action of a user, and extracting a plurality of first characteristic vectors corresponding to the plurality of images respectively, wherein the number of the plurality of images is N, and the plurality of images comprise the current Nth image and the previous N-1 images; combining the first feature vectors corresponding to the Nth image with the N-1 first feature vectors corresponding to the previous N-1 images respectively to generate second feature vectors; and controlling the state of the automatic walking equipment according to the second feature vector.
In one embodiment, before the acquiring the plurality of images corresponding to the current action of the user, the method further includes: and entering an activated state according to preset voice in the voice information of the current scene.
In one embodiment, after the automatic walking device is in the activated state, the method further includes: detecting whether the image information of the current scene comprises face image information or not; if yes, executing face recognition and authenticating the control authority of the user; if not, carrying out sound source positioning according to the voice signal and determining the position information of the sound source; and controlling the automatic walking equipment to move towards the sound source according to the position information of the sound source so as to detect the face image information of the sound source and execute face recognition and authenticate the control authority of the user.
In one embodiment, the executing of the face recognition and the authenticating of the control authority of the user includes: authenticating the control authority of the user according to the face image information;
the collecting of the plurality of images corresponding to the current action of the user comprises the following steps: and acquiring a plurality of images corresponding to the current action of the user if the authentication is passed.
In one embodiment, the facial image information comprises a plurality of pieces of facial image information; the authenticating the control authority of the user according to the face image information comprises the following steps: according to the multiple pieces of face image information, control authorities of multiple users corresponding to the multiple pieces of face image information are authenticated; determining control priorities corresponding to the multiple users; determining a user corresponding to the highest control priority according to the control priorities corresponding to the multiple users;
and if the authentication is passed, acquiring an image corresponding to the current action of the user at preset time intervals, wherein the image acquisition step comprises the following steps: and acquiring an image corresponding to the current action of the user corresponding to the highest control priority at preset time intervals.
In one embodiment, the controlling the state of the automatic walking device according to the second feature vector includes: acquiring a corresponding instruction according to the second feature vector; and controlling the state of the automatic walking equipment according to the instruction.
In one embodiment, before the acquiring the plurality of images corresponding to the current action of the user, the method further includes: collecting an image sequence sample corresponding to a preset action; and training a corresponding machine learning model in a data augmentation mode according to the image sequence sample, wherein the machine learning model is a deep convolution neural network model.
A control device of an automatic walking apparatus, the device comprising: the image acquisition module is used for acquiring a plurality of images corresponding to the current action of the user; the extraction module is used for extracting a plurality of first characteristic vectors corresponding to the plurality of images respectively; the number of the multiple images is N, and the multiple images comprise the current Nth image and the previous N-1 images; a merging module, configured to merge the first feature vector corresponding to the nth image with N-1 first feature vectors corresponding to the previous N-1 images, respectively, to generate a second feature vector; and the control module is used for controlling the state of the automatic walking equipment according to the second characteristic vector.
In one embodiment, the apparatus further comprises: and the activation module is used for enabling the automatic walking equipment to enter an activation state according to preset voice in the voice information of the current scene.
In one embodiment, the apparatus further comprises an image processing module, a sound source localization module, and a face recognition module: the image processing module is used for detecting whether the image information of the current scene comprises face image information or not; the face recognition module is used for executing face recognition to authenticate the control authority of the user when the image information of the current scene comprises face image information; the sound source positioning module is used for positioning a sound source according to the voice signal and determining the position information of the sound source when the image information of the current scene does not include face image information; the control module is further used for controlling the automatic walking equipment to move towards the sound source according to the position information of the sound source so as to detect the face image information of the sound source and execute face recognition and authenticate the control authority of the user.
An automatic walking device comprises a driving component, an image acquisition component, a voice acquisition component and a controller; the driving component is used for driving the automatic walking equipment to move; the voice acquisition component is arranged on the automatic walking equipment and is used for acquiring voice signals in the current scene; the image acquisition component is arranged on the automatic walking equipment and is used for acquiring a plurality of images corresponding to the current action of the user; the controller is used for extracting a plurality of first feature vectors corresponding to the plurality of images respectively, the number of the plurality of images is N, and the plurality of images comprise the current Nth image and the previous N-1 images; combining the first feature vectors corresponding to the Nth image with the N-1 first feature vectors corresponding to the previous N-1 images respectively to generate second feature vectors; and controlling the state of the automatic walking equipment according to the second feature vector.
In one embodiment, the controller is further configured to detect whether the image information of the current scene includes face image information; if yes, executing face recognition to authenticate the control authority of the user; if not, carrying out sound source positioning according to the voice signal and determining the position information of the sound source; and controlling the automatic walking equipment to move towards the sound source according to the position information of the sound source so as to enable the image acquisition component to acquire the face image information of the sound source, and then detecting the face image information and executing face recognition to authenticate the control authority of the user.
A computer device comprising a memory storing a computer program and a processor implementing the method steps in the above embodiments when executing the computer program.
According to the automatic walking device, the control method, the control device and the computer device, the automatic walking device identifies the current action sent by the user to control the state of the automatic walking device by acquiring a plurality of images corresponding to the current action of the user, the current action sent by the user comprises gesture change or posture change of other body parts, and the technical problem of low convenience caused by a key pressing mode or a voice instruction mode in the prior art is solved. Furthermore, a plurality of images corresponding to the current action of the user are collected, corresponding first feature vectors are extracted respectively to obtain N images corresponding to the current action and N corresponding first feature vectors, the first feature vectors corresponding to the N images and N-1 first feature vectors corresponding to the previous N-1 images are combined to generate second feature vectors, the state of the automatic walking equipment is controlled according to the second feature vectors, and the calculation amount in the gesture recognition process or other body posture change recognition processes is reduced.
Drawings
FIG. 1a is a schematic flow chart illustrating a method for controlling an automated walking apparatus according to an embodiment;
FIG. 1b is a diagram illustrating the extraction of a first feature vector according to one embodiment;
FIG. 2a is a schematic flow chart illustrating a method for controlling an automated walking device after an activation state according to one embodiment;
FIG. 2b is a schematic diagram of an embodiment of an automated walking device performing sound source localization;
FIG. 3 is a flowchart illustrating the step S220 according to one embodiment;
FIG. 4 is a flowchart illustrating the step S130 according to an embodiment;
FIGS. 5 a-5 b are schematic flow charts illustrating the model training steps in one embodiment;
fig. 6a to 6d are schematic flow charts illustrating a control method of the automatic walking device according to an embodiment;
FIG. 7 is a block diagram showing the construction of a control device of the automatic walking apparatus in one embodiment;
FIG. 8 is a block diagram showing the construction of a control device of the automatic walking apparatus in one embodiment;
fig. 9 is a schematic structural view of the automatic walking apparatus in one embodiment.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.
In one embodiment, referring to fig. 1a, the present application provides a method for controlling an automatic walking device, the method comprising the steps of:
s110, collecting a plurality of images corresponding to the current action of the user, and extracting a plurality of first feature vectors corresponding to the plurality of images respectively, wherein the number of the plurality of images is N, and the plurality of images comprise the current Nth image and the previous N-1 images.
S120, combining the first feature vectors corresponding to the Nth image and the N-1 first feature vectors corresponding to the previous N-1 images respectively to generate second feature vectors.
And S130, controlling the state of the automatic walking equipment according to the second feature vector.
The automatic walking equipment can be intelligent equipment with an automatic walking function, such as an intelligent mower, an intelligent snow sweeper, an intelligent ground washing vehicle and the like. The current motion may be a gesture motion or a motion gesture made by the body.
The automatic walking equipment collects an image corresponding to the current action of a user through an image collecting device at intervals of preset time, and extracts a first feature vector corresponding to the image immediately so as to obtain a plurality of images corresponding to the current action and a plurality of first feature vectors corresponding to the images. Referring to FIG. 1b, RGB represents three color channels per image. Assuming that the number of the plurality of images corresponding to the current action is N, the plurality of images include the current Nth image and the previous N-1 images. A plurality of first feature vectors corresponding to the plurality of images are recorded as x(i). Corresponding first feature vector x to the Nth image(N)N-1 first feature vectors x corresponding to the previous N-1 images respectively(1)、x(2)...x(N-1)The combining is performed to generate a second feature vector. And sending the second feature vector to a classifier, and outputting a corresponding instruction by the classifier according to the second feature vector so as to control the state of the automatic walking equipment. In machine learning, the classifier functions to determine the class to which a new observation sample belongs based on training data that has been labeled with a class. In this embodiment, the generated second feature vector is sent to the classifier, so that the classifier determines the instruction corresponding to the second feature vector and outputs the corresponding instruction.
In the embodiment, the automatic walking device identifies the state of the automatic walking device by acquiring a plurality of images corresponding to the current action of the user and identifying the current action sent by the user, and the current action sent by the user comprises gesture change or posture change of other body parts, so that the technical problem of low convenience caused by a key mode or a voice instruction mode in the prior art is solved. Furthermore, a plurality of images corresponding to the current action of the user are collected, corresponding first feature vectors are extracted respectively to obtain N images corresponding to the current action and N corresponding first feature vectors, the first feature vectors corresponding to the N images and N-1 first feature vectors corresponding to the previous N-1 images are combined to generate second feature vectors, the state of the automatic walking equipment is controlled according to the second feature vectors, the recognition of the current action is completed through the calculated amount of a single image, and the calculated amount in the gesture recognition or other body posture change recognition processes is reduced.
In one embodiment, before acquiring a plurality of images corresponding to the current action of the user, the method further includes: and entering an activated state according to preset voice in the voice information of the current scene.
Wherein the preset voice refers to voice information set in advance for activating the automatic walking apparatus. When the automatic walking equipment is in a dormant state, if the voice information of the current scene comprises preset voice, the automatic walking equipment enters an activated state. For example, the preset voice may be the name of the automatic walking device. And the automatic walking equipment performs waveform recognition according to the collected sound, and enters an activated state when judging that the voice information in the current scene comprises the name of the automatic walking equipment.
In one embodiment, referring to fig. 2a, after the automatic walking device is in the activated state, the method further comprises the following steps:
s210, detecting whether the image information of the current scene comprises face image information.
And S220, if so, executing face recognition and authenticating the control authority of the user.
And S230, if not, carrying out sound source positioning according to the voice signal and determining the position information of the sound source.
And S240, controlling the automatic walking equipment to move towards the sound source according to the position information of the sound source so as to detect the face image information at the sound source and execute face recognition and authenticate the control authority of the user.
Specifically, after the automatic walking device enters an activated state, the automatic walking device starts to detect the image information of the current scene and judges whether the image information of the current scene includes face image information. And if the image information of the current scene comprises face image information, the automatic walking equipment starts to execute face recognition and authenticate the control authority of the user.
Please refer to fig. 2b, the automatic walking device is installed with an image collecting device and a voice collecting device, the image collecting device may be a camera 210, the voice collecting device may be a microphone array, the microphone array includes a microphone 221, a microphone 222, and a microphone 223. the user sends a preset voice to call the automatic walking device, the automatic walking device collects a sound waveform through the microphone array, and performs matching to calculate the time difference of the current sound source transmitted to each microphone in the microphone array to be △ t12、△t13And △ t23Transit time difference △ t12、△t13And △ t23Calculating the distances d of the microphones 221, 222, 223 from the sound source1、d2And d3。
Where c is the speed of sound under the current operating conditions.
The position information of the sound source, i.e., the coordinates of the sound source, can be determined according to the above formula. The automatic walking equipment completes steering action to the sound source position according to the sound source coordinate, and controls the automatic walking equipment to move to the sound source, so that the camera is aligned with the user, face image information at the sound source is detected, and face recognition and user control authority authentication are performed.
Further, executing face recognition and authenticating the control authority of the user includes: and authenticating the control authority of the user according to the detected face image information. The method comprises the following steps of collecting a plurality of images corresponding to the current action of a user, including: and if the control authority of the user passes the authentication, the automatic walking equipment collects a plurality of images corresponding to the current action of the user.
Specifically, after the face image information appears in the visual angle of the image acquisition device, the automatic walking equipment authenticates the control authority of the user corresponding to the face image according to the detected face image information, and if the authentication is passed, the image acquisition device starts to acquire a plurality of images corresponding to the current action started by the user corresponding to the face image. It can be understood that, if the user corresponding to the face image information does not pass the authority authentication, the current action sent by the user is not identified, then the automatic walking device may start to detect whether the voice information in the current scene includes the preset voice, and if the voice information in the current scene includes the preset voice, the automatic walking device starts to perform the sound source localization. Or, if the user corresponding to the face image information does not pass the authority authentication, the automatic walking device can also enter a dormant state to reduce the use of electric quantity.
In the embodiment, the automatic walking equipment finds the sound source position in a specific posture through voice recognition and positioning, and a user can conveniently control the automatic walking equipment without moving in situ.
In one embodiment, the facial image information includes a plurality of pieces of facial image information. Referring to fig. 3, authenticating the control authority of the user according to the face image information includes the following steps:
and S310, authenticating the control authority of the plurality of users respectively corresponding to the plurality of pieces of face image information according to the plurality of pieces of face image information.
And S320, determining control priorities corresponding to the multiple users.
S330, determining the user corresponding to the highest control priority according to the control priorities corresponding to the multiple users.
And acquiring a plurality of images corresponding to the current action of the user if the authentication is passed, wherein the method comprises the following steps:
and S340, authenticating through collecting a plurality of images corresponding to the current action of the user corresponding to the highest control priority.
Wherein, the automatic walking equipment is provided with an image acquisition device which can be a camera. When a plurality of users appear in the visual angle of the camera, the face image information comprises a plurality of pieces of face image information. And the automatic walking equipment authenticates the control authority of the multiple users respectively corresponding to the multiple pieces of face image information according to the multiple pieces of face image information, and determines the control priority corresponding to the multiple users. Therefore, according to the control priorities corresponding to the multiple users, the user corresponding to the highest control priority is determined, the action sent by the user corresponding to the highest control priority is identified, and the multiple images corresponding to the current action of the user corresponding to the highest control priority are collected.
For example, when the user a and the user b appear in the camera viewing angle, face recognition is performed according to the face image information of the user a and the user b, and the control priorities corresponding to the user a and the user b are determined respectively. And if the control priority of the user A is higher than that of the user B, simultaneously acquiring the face image information of the user A and a plurality of images corresponding to the current action through the camera.
In the embodiment, the control authority of the user is judged through face recognition. And when the automatic walking device identifies a plurality of users, the automatic walking device can determine the control priorities corresponding to the plurality of users, so that the gesture command can be read according to the control priorities of the users. The intelligence of the automatic walking equipment is improved, and the application range of the automatic walking equipment is expanded.
In one embodiment, referring to fig. 4, controlling the state of the automatic walking device according to the second eigenvector includes:
and S410, acquiring a corresponding instruction according to the second feature vector.
And S420, controlling the state of the automatic walking equipment according to the acquired instruction.
And combining the first feature vector corresponding to the Nth image with the N-1 first feature vectors corresponding to the previous N-1 images respectively to generate a second feature vector. The automatic walking equipment can send the generated second feature vector to the classifier to complete action recognition, and then the corresponding instruction can be obtained. And the automatic walking equipment controls the state of the automatic walking equipment according to the obtained instruction. Since the action commands are mutually exclusive, and Softmax is used for the mutual exclusion classification problem with good effect. Therefore, in this embodiment, the classifier can use Softmax, which is expressed as:
wherein: x is the number of(i)Is the ith feature vector; k is the total number of instruction sets, θ is the model parameter (result of debug portion), p (y)(i)＝k|x(i)(ii) a θ) is the probability that the current instruction is action k.
In this embodiment, a specific command may be specified, for example, the palm facing the camera may be a stop command, and the command being executed by the automatic walking device may be interrupted. The five-finger fingertips are closed to form a cancel instruction, and the automatic walking equipment can return to the position and the state before the instruction.
In one embodiment, referring to fig. 5a, before acquiring a plurality of images corresponding to the current motion of the user, the method further includes the following model training steps:
and S510, collecting image sequence samples corresponding to the preset actions.
S520, training a corresponding machine learning model in a data augmentation mode according to the image sequence samples, wherein the machine learning model is a deep convolution neural network model.
Please refer to fig. 5b, an image sequence sample I corresponding to a preset action is acquired through an image acquisition device, the image sequence sample I0 includes N images, the image sequence sample I is sent to a convolution Pool Conv & Pool, a feature vector corresponding to the image sequence sample I is obtained through convolution operation, a classifier and the feature vector corresponding to the image sequence sample I are connected in a point-to-point manner, and a corresponding actual instruction is output through calculation. And a certain error exists between the actual instruction and the prediction instruction, and the characteristic vector corresponding to the image sequence sample I is adjusted according to the error. When the above process is repeated for each image sequence sample I until the error does not exceed the specified range for the entire image sequence sample set.
Specifically, model parameters of the motion recognition instruction are debugged according to the image sequence samples, the used model is a deep convolutional neural network, and the deep convolutional neural network model has certain advantages in extracting feature vectors from the pictures. For example, the method has better robustness on image trailing conditions, image distortion and action posture shooting angles acquired during the traveling of the automatic walking equipment. In the model debugging stage, the distortion of the image and the training of the model are completed in sequence through the data amplification module and the training model module. Specifically, the model may include two parts, i.e., a feature extractor and a classifier, and the training mode is as shown in fig. 5b, and the trained model is finally stored.
In one embodiment, referring to fig. 6a, the present application provides a method for controlling an automatic walking device, the method comprising the steps of:
s610, entering an activated state according to preset voice in the voice information of the current scene.
Referring to fig. 6b, when the automatic walking device is in the sleep state, if the voice information of the current scene includes the preset voice, for example, the preset voice is the name of the automatic walking device, the automatic walking device detects that the voice information of the current scene includes the name. The automatic walking device enters an activated state.
S620, detecting whether the image information of the current scene comprises face image information.
After the automatic walking equipment enters an activated state, the automatic walking equipment starts to detect the image information of the current scene and judges whether the image information of the current scene comprises face image information or not.
And S630, if not, carrying out sound source positioning and determining the position information of the sound source according to the voice signal.
And if the image information of the current scene does not comprise the face image information, positioning the sound source according to the voice signal and determining the position information of the sound source.
Referring to fig. 6b, the user sends a preset voice call to the automatic walking device, the automatic walking device collects sound waveforms through a microphone array, the microphone array comprises 3 microphones, and the sound waveforms are matchedCalculating the time difference of current sound source transmitted to each microphone in the microphone array to be △ t12、△t13And △ t23Transit time difference △ t12、△t13And △ t23Calculating the distance d of each microphone from the sound source1、d2And d3。
Where c is the speed of sound under the current operating conditions. The position information of the sound source, i.e., the coordinates of the sound source, can be obtained according to the above formula.
And S640, controlling the automatic walking equipment to move towards the sound source according to the position information of the sound source so as to detect the face image information at the sound source.
Referring to fig. 6c, the automatic walking device completes a steering action to the sound source position according to the sound source coordinates, and controls the automatic walking device to move to the sound source, so that the camera 210 is aligned with the user, and the face image information at the sound source is detected.
And S650, authenticating the control authority of the plurality of users respectively corresponding to the plurality of pieces of face image information according to the plurality of pieces of face image information.
When multiple users appear in the viewing angle of the camera 210, the face image information includes multiple pieces of face image information. The automatic walking equipment authenticates the control authority of a plurality of users respectively corresponding to a plurality of pieces of face image information according to the plurality of pieces of face image information,
and S660, determining control priorities corresponding to the multiple users.
And after the control authorities of the multiple users respectively corresponding to the multiple pieces of face image information are authenticated, determining the control priorities corresponding to the multiple users.
And S670, determining the user corresponding to the highest control priority according to the control priorities corresponding to the multiple users.
And S680, collecting a plurality of images corresponding to the current action of the user corresponding to the highest control priority.
Referring to fig. 6c and 6d, according to the control priorities corresponding to the multiple users 610, the user corresponding to the highest control priority is determined. The automatic walking device starts to collect a plurality of images 620 corresponding to the current actions of the user corresponding to the highest control priority through the camera 210, and identifies the actions sent by the user corresponding to the highest control priority.
S681, extracting a plurality of first feature vectors corresponding to the plurality of images respectively, wherein the number of the plurality of images is N, and the plurality of images comprise the current Nth image and the previous N-1 images.
The number of the multiple images is marked as N, and the multiple images comprise the current Nth image and the previous N-1 images. A plurality of first feature vectors corresponding to the plurality of images are recorded as x(i)。
S682, combining the first characteristic vector corresponding to the Nth image with the N-1 first characteristic vectors corresponding to the previous N-1 images respectively to generate a second characteristic vector.
Corresponding first feature vector x to the Nth image(N)N-1 first feature vectors x corresponding to the previous N-1 images respectively(1)、x(2)...x(N-1)The combining is performed to generate a second feature vector.
And S683, acquiring a corresponding instruction according to the generated second feature vector.
The automatic walking equipment can send the generated second feature vector to the classifier to complete action recognition, and then the corresponding instruction can be obtained.
And S684, controlling the state of the automatic walking equipment according to the obtained instruction.
And the automatic walking equipment controls the state of the automatic walking equipment according to the obtained instruction.
It should be understood that although the various steps in the flow charts of fig. 1-6 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 1-6 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, the present application provides a control device for an automatic walking apparatus, please refer to fig. 7, the control device 700 includes:
the image acquisition module 710 is configured to acquire a plurality of images corresponding to the current action of the user.
An extracting module 720, configured to extract a plurality of first feature vectors corresponding to the plurality of images, respectively; the number of the multiple images is N, and the multiple images comprise the current Nth image and the previous N-1 images.
The merging module 730 is configured to merge the first feature vector corresponding to the nth image with N-1 first feature vectors corresponding to the previous N-1 images, respectively, to generate a second feature vector.
And the control module 740 is configured to control the state of the automatic walking device according to the second eigenvector.
In one embodiment, the control device comprises: and the activation module is used for enabling the automatic walking equipment to enter an activation state according to preset voice in the voice information of the current scene.
In one embodiment, referring to fig. 8, the control device further comprises an image processing module 810, a sound source localization module 820 and a face recognition module 830.
And the image processing module 810 is configured to detect whether the image information of the current scene includes face image information.
A face recognition module 820, configured to perform face recognition to authenticate the control authority of the user when the image information of the current scene includes face image information.
And a sound source positioning module 830, configured to, when the image information of the current scene does not include the face image information, perform sound source positioning according to the voice signal and determine position information of the sound source.
The control module 730 is further configured to control the automatic walking device to move to the sound source according to the position information of the sound source, so as to detect face image information at the sound source, and perform face recognition and authenticate the control authority of the user.
For specific limitations of the control device, reference may be made to the limitations of the control method above, and details are not repeated here. The respective modules in the above control device may be wholly or partially implemented by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent of a processor in a display screen, and can also be stored in a memory in computer equipment in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, referring to fig. 9, the present application provides an automatic walking device 900, which includes a driving part 910, an image capturing part 920, a voice capturing part 930, and a controller 940.
And a driving part 910 for driving the automatic walking device to move.
And a voice collecting unit 930, disposed on the automatic walking device, for collecting the voice signal in the current scene.
And the image acquisition part 920 is arranged on the automatic walking device and is used for acquiring a plurality of images corresponding to the current action of the user.
A controller 940, configured to extract a plurality of first feature vectors corresponding to a plurality of images, respectively, where the number of the plurality of images is N, and the plurality of images include a current nth image and a previous N-1 images; combining the first feature vector corresponding to the Nth image with the N-1 first feature vectors corresponding to the previous N-1 images respectively to generate a second feature vector; and controlling the state of the automatic walking equipment according to the second feature vector.
In one embodiment, the controller is further configured to detect whether the image information of the current scene includes face image information; if yes, executing face recognition to authenticate the control authority of the user; if not, positioning the sound source according to the voice signal and determining the position information of the sound source; and controlling the automatic walking equipment to move towards the sound source according to the position information of the sound source so as to enable the image acquisition component to acquire the face image information at the sound source, and then detecting the face image information and executing face recognition to authenticate the control authority of the user.
For specific limitations of the automatic walking device, reference may be made to the above limitations of the control method, which are not described herein again. The respective modules in the above control device may be wholly or partially implemented by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent of a processor in a display screen, and can also be stored in a memory in computer equipment in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, the present application provides a computer device comprising a memory storing a computer program and a processor implementing the method steps in the above embodiments when the processor executes the computer program. It should be noted that, as will be understood by those skilled in the art, all or part of the processes in the methods of the above embodiments may be implemented by a computer program, which may be stored in a non-volatile computer readable storage medium, and the computer program may include the processes of the above embodiments of the methods when executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
In addition, the terms "first", "second", and the like used in the embodiments of the present application may be used herein to describe various elements, but the elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, the first display region may be referred to as a second feature vector, and similarly, the second feature vector may be referred to as a first feature vector, without departing from the scope of the present application. The first feature vector and the second feature vector are both feature vectors, but they are not the same feature vector.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (13)
1. A method of controlling an automatic walking device, the method comprising:
acquiring a plurality of images corresponding to the current action of a user, and extracting a plurality of first characteristic vectors corresponding to the plurality of images respectively, wherein the number of the plurality of images is N, and the plurality of images comprise the current Nth image and the previous N-1 images;
combining the first feature vectors corresponding to the Nth image with the N-1 first feature vectors corresponding to the previous N-1 images respectively to generate second feature vectors;
and controlling the state of the automatic walking equipment according to the second feature vector.
2. The method of claim 1, further comprising, prior to capturing a plurality of images corresponding to a current action of a user, the capturing a plurality of images corresponding to a current action of a user:
and entering an activated state according to preset voice in the voice information of the current scene.
3. The method of claim 2, wherein after the automated walking device is in the activated state, further comprising:
detecting whether the image information of the current scene comprises face image information or not;
if yes, executing face recognition and authenticating the control authority of the user;
if not, carrying out sound source positioning according to the voice signal and determining the position information of the sound source;
and controlling the automatic walking equipment to move towards the sound source according to the position information of the sound source so as to detect the face image information of the sound source and execute face recognition and authenticate the control authority of the user.
4. The method of claim 3, wherein the performing face recognition and authenticating the control authority of the user comprises:
authenticating the control authority of the user according to the face image information;
the collecting of the plurality of images corresponding to the current action of the user comprises the following steps:
and acquiring a plurality of images corresponding to the current action of the user if the authentication is passed.
5. The method according to claim 4, wherein the face image information includes a plurality of pieces of face image information; the authenticating the control authority of the user according to the face image information comprises the following steps:
according to the multiple pieces of face image information, control authorities of multiple users corresponding to the multiple pieces of face image information are authenticated;
determining control priorities corresponding to the multiple users;
determining a user corresponding to the highest control priority according to the control priorities corresponding to the multiple users;
and if the authentication is passed, acquiring a plurality of images corresponding to the current action of the user, wherein the steps comprise:
and the authentication is realized by acquiring a plurality of images corresponding to the current action of the user corresponding to the highest control priority.
6. The method of claims 1-5, wherein said controlling the state of the automated walking device according to the second eigenvector comprises:
acquiring a corresponding instruction according to the second feature vector;
and controlling the state of the automatic walking equipment according to the instruction.
7. The method according to any one of claims 1 to 5, wherein before the capturing of the plurality of images corresponding to the current action of the user, the method further comprises:
collecting an image sequence sample corresponding to a preset action;
and training a corresponding machine learning model in a data augmentation mode according to the image sequence sample, wherein the machine learning model is a deep convolution neural network model.
8. A control device for an automatic walking apparatus, characterized in that the device comprises:
the image acquisition module is used for acquiring a plurality of images corresponding to the current action of the user;
the extraction module is used for extracting a plurality of first characteristic vectors corresponding to the plurality of images respectively; the number of the multiple images is N, and the multiple images comprise the current Nth image and the previous N-1 images;
a merging module, configured to merge the first feature vector corresponding to the nth image with N-1 first feature vectors corresponding to the previous N-1 images, respectively, to generate a second feature vector;
and the control module is used for controlling the state of the automatic walking equipment or outputting corresponding interactive contents according to the second feature vector.
9. The apparatus of claim 8, further comprising:
and the activation module is used for enabling the automatic walking equipment to enter an activation state according to preset voice in the voice information of the current scene.
10. The apparatus of claim 9, further comprising an image processing module, a sound source localization module, and a face recognition module:
the image processing module is used for detecting whether the image information of the current scene comprises face image information or not;
the face recognition module is used for executing face recognition to authenticate the control authority of the user when the image information of the current scene comprises face image information;
the sound source positioning module is used for positioning a sound source according to the voice signal and determining the position information of the sound source when the image information of the current scene does not include face image information;
the control module is further used for controlling the automatic walking equipment to move towards the sound source according to the position information of the sound source so as to detect the face image information of the sound source and execute face recognition and authenticate the control authority of the user.
11. An automatic walking device is characterized by comprising a driving part, an image acquisition part, a voice acquisition part and a controller;
the driving component is used for driving the automatic walking equipment to move;
the voice acquisition component is arranged on the automatic walking equipment and is used for acquiring voice signals in the current scene;
the image acquisition component is arranged on the automatic walking equipment and is used for acquiring a plurality of images corresponding to the current action of the user;
the controller is used for extracting a plurality of first feature vectors corresponding to the plurality of images respectively, the number of the plurality of images is N, and the plurality of images comprise the current Nth image and the previous N-1 images; combining the first feature vectors corresponding to the Nth image with the N-1 first feature vectors corresponding to the previous N-1 images respectively to generate second feature vectors; and controlling the state of the automatic walking equipment according to the second feature vector.
12. The automatic walking device of claim 11, wherein the controller is further configured to detect whether the image information of the current scene includes face image information; if yes, executing face recognition to authenticate the control authority of the user; if not, carrying out sound source positioning according to the voice signal and determining the position information of the sound source; and controlling the automatic walking equipment to move towards the sound source according to the position information of the sound source so as to enable the image acquisition component to acquire the face image information of the sound source, and then detecting the face image information and executing face recognition to authenticate the control authority of the user.
13. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810840595.XA CN110852133A (en) | 2018-07-27 | 2018-07-27 | Automatic walking equipment, control method and control device thereof, and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810840595.XA CN110852133A (en) | 2018-07-27 | 2018-07-27 | Automatic walking equipment, control method and control device thereof, and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110852133A true CN110852133A (en) | 2020-02-28 |
Family
ID=69594738
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810840595.XA Pending CN110852133A (en) | 2018-07-27 | 2018-07-27 | Automatic walking equipment, control method and control device thereof, and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110852133A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104985599A (en) * | 2015-07-20 | 2015-10-21 | 百度在线网络技术（北京）有限公司 | Intelligent robot control method and system based on artificial intelligence and intelligent robot |
| CN105235615A (en) * | 2015-10-27 | 2016-01-13 | 浙江吉利控股集团有限公司 | Vehicle control system based on face recognition |
| CN107480586A (en) * | 2017-07-06 | 2017-12-15 | 天津科技大学 | Bio-identification photo bogus attack detection method based on human face characteristic point displacement |
| CN107511832A (en) * | 2016-06-15 | 2017-12-26 | 深圳光启合众科技有限公司 | High in the clouds interaction systems and its more sensing type intelligent robots and perception interdynamic method |
| CN107703931A (en) * | 2016-08-09 | 2018-02-16 | 北京百度网讯科技有限公司 | Method and apparatus for controlling automatic driving vehicle |
| US20180143635A1 (en) * | 2010-06-07 | 2018-05-24 | Affectiva, Inc. | Vehicle manipulation using occupant image analysis |
-
2018
- 2018-07-27 CN CN201810840595.XA patent/CN110852133A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180143635A1 (en) * | 2010-06-07 | 2018-05-24 | Affectiva, Inc. | Vehicle manipulation using occupant image analysis |
| CN104985599A (en) * | 2015-07-20 | 2015-10-21 | 百度在线网络技术（北京）有限公司 | Intelligent robot control method and system based on artificial intelligence and intelligent robot |
| CN105235615A (en) * | 2015-10-27 | 2016-01-13 | 浙江吉利控股集团有限公司 | Vehicle control system based on face recognition |
| CN107511832A (en) * | 2016-06-15 | 2017-12-26 | 深圳光启合众科技有限公司 | High in the clouds interaction systems and its more sensing type intelligent robots and perception interdynamic method |
| CN107703931A (en) * | 2016-08-09 | 2018-02-16 | 北京百度网讯科技有限公司 | Method and apparatus for controlling automatic driving vehicle |
| CN107480586A (en) * | 2017-07-06 | 2017-12-15 | 天津科技大学 | Bio-identification photo bogus attack detection method based on human face characteristic point displacement |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11257223B2 (en) | Systems and methods for user detection, identification, and localization within a defined space | |
| KR102465532B1 (en) | Method for recognizing an object and apparatus thereof | |
| US9977954B2 (en) | Robot cleaner and method for controlling a robot cleaner | |
| KR101603017B1 (en) | Gesture recognition device and gesture recognition device control method | |
| CN111857356B (en) | Method, device, equipment and storage medium for recognizing interaction gesture | |
| US10733274B2 (en) | Biometric authentication using gesture | |
| TWI416366B (en) | Method, electronic apparatus and computer program product for creating biologic feature data | |
| EP3528093B1 (en) | Non-contact gesture control method, and electronic terminal device | |
| US20110158476A1 (en) | Robot and method for recognizing human faces and gestures thereof | |
| US10710244B2 (en) | Robot control using gestures | |
| WO2016127437A1 (en) | Live body face verification method and system, and computer program product | |
| JP5662670B2 (en) | Image processing apparatus, image processing method, and program | |
| CN108983979B (en) | Gesture tracking recognition method and device and intelligent equipment | |
| CN110705357A (en) | Face recognition method and face recognition device | |
| US20110273551A1 (en) | Method to control media with face detection and hot spot motion | |
| JP6265592B2 (en) | Facial feature extraction apparatus and face authentication system | |
| CN107273869B (en) | Gesture recognition control method and electronic equipment | |
| US9355641B2 (en) | Monitoring device using selective attention model and method for monitoring same | |
| JP5012589B2 (en) | Human tracking system using image information | |
| CN113696849B (en) | Gesture-based vehicle control method, device and storage medium | |
| JP2007249298A (en) | Face authentication apparatus and face authentication method | |
| JP2010086403A (en) | Facial recognition device, facial recognition method, and passage controller | |
| CN110852133A (en) | Automatic walking equipment, control method and control device thereof, and computer equipment | |
| JP2015187769A (en) | Object detection device, object detection method, and program | |
| KR20190132885A (en) | Apparatus, method and computer program for detecting hand from video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200228 |
|
| RJ01 | Rejection of invention patent application after publication |