CN113095268B - Robot gait learning method, system and storage medium based on video stream - Google Patents
Robot gait learning method, system and storage medium based on video stream Download PDFInfo
- Publication number
- CN113095268B CN113095268B CN202110438406.8A CN202110438406A CN113095268B CN 113095268 B CN113095268 B CN 113095268B CN 202110438406 A CN202110438406 A CN 202110438406A CN 113095268 B CN113095268 B CN 113095268B
- Authority
- CN
- China
- Prior art keywords
- gait
- foot
- node
- coordinates
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005021 gait Effects 0.000 title claims abstract description 113
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000013528 artificial neural network Methods 0.000 claims abstract description 27
- 238000013135 deep learning Methods 0.000 claims abstract description 22
- 239000011664 nicotinic acid Substances 0.000 claims abstract description 13
- 230000004044 response Effects 0.000 claims abstract description 13
- 230000000007 visual effect Effects 0.000 claims abstract description 8
- 210000000988 bone and bone Anatomy 0.000 claims description 14
- 238000000605 extraction Methods 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 12
- 238000001914 filtration Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 claims description 8
- 206010034701 Peroneal nerve palsy Diseases 0.000 claims description 7
- 238000004422 calculation algorithm Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 5
- 210000003414 extremity Anatomy 0.000 claims description 5
- 238000007781 pre-processing Methods 0.000 claims description 5
- 238000012549 training Methods 0.000 claims description 5
- 238000010606 normalization Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000005260 corrosion Methods 0.000 claims description 2
- 230000007797 corrosion Effects 0.000 claims description 2
- 230000003447 ipsilateral effect Effects 0.000 claims description 2
- 210000003141 lower extremity Anatomy 0.000 claims description 2
- 210000001364 upper extremity Anatomy 0.000 claims description 2
- 230000009471 action Effects 0.000 abstract description 4
- 238000010801 machine learning Methods 0.000 abstract description 4
- 230000007613 environmental effect Effects 0.000 description 5
- 210000002414 leg Anatomy 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000003708 edge detection Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 210000001519 tissue Anatomy 0.000 description 2
- 206010000117 Abnormal behaviour Diseases 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 210000004394 hip joint Anatomy 0.000 description 1
- 210000000629 knee joint Anatomy 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 238000012847 principal component analysis method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/66—Analysis of geometric attributes of image moments or centre of gravity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Geometry (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Manipulator (AREA)
Abstract
The application discloses a robot gait learning method, a system and a storage medium based on video streams, which are characterized in that firstly, motion video streams of quadruped creatures moving under different environments are respectively collected, key pixel points of quadruped creatures in key frame images are extracted in response to the key frame images, the neural network is utilized to carry out deep learning on asynchronous state information corresponding to different environments, the environment information of a current frame image in visual equipment is identified, a discretized gait model is established according to foot point coordinates in the gait information, and foot point coordinates of a foot falling point when a quadruped bionic robot moves are obtained in the discretized gait model. According to the application, the characteristic points are extracted through massive videos to acquire skeleton actions, so that the video characteristic point data is utilized to perform stable machine learning of the bionic gait.
Description
Technical Field
The application relates to the field of robot gait learning, in particular to a robot gait learning method, a system and a storage medium based on video streaming.
Background
In the research work of the multi-legged robot, the learning ability, the evolution ability and the control decision ability of a person or organism are simulated, the robot is endowed with intelligence, the necessary conditions of the robot for adapting to a complex environment, completing complex work tasks and implementing self evolution are provided, and the method is the core content and the important direction of the current and future robot research fields.
The application of the current image recognition field is becoming mature, and the streaming media itself is composed of a plurality of data frames, so that the streaming media is applied to the streaming media field on the basis of mature image recognition technology, and has a certain positive value in terms of technology and value, and from the application point of view, the streaming media field has remarkable demands.
In the prior art, the patent with the publication number of CN112220650A discloses an online gait generation control system for contralateral training of an exoskeleton robot, which comprises a healthy side pre-motion data acquisition module, a sensor signal acquisition and processing module, a healthy side patient side data conversion module and a patient side step correction module; the method comprises the steps of establishing a gait phase specimen set through data acquired by a healthy side pre-motion data acquisition module, acquiring healthy side motion data through a sensor signal acquisition and processing module, predicting healthy side motion data by a healthy side affected side data conversion module through combining the healthy side motion data and the gait phase specimen set, and correcting a prediction result through a healthy side step correction module to finally obtain the output positions and interpolation time of a healthy side hip joint direct current servo motor and a healthy side knee joint direct current servo motor. Therefore, the traditional robot training method is low in intelligence by utilizing the sensors to collect data, a large number of sensors are needed, the engineering quantity is large, the training efficiency is low, and the stable gait of the robot during movement cannot be adjusted.
Therefore, a robot gait learning method based on video streaming is provided, which extracts feature points through massive videos to acquire skeletal actions, so that the robot gait learning method based on video streaming performs stable bionic gait by utilizing video feature point data is needed to be solved.
Disclosure of Invention
The embodiment of the application aims to provide a robot gait learning method, a system and a storage medium based on video streaming, which are used for solving the technical problems mentioned in the background art section.
In a first aspect, the present application provides a robot gait learning method based on video streaming, comprising the steps of:
s1, respectively acquiring motion video streams of quadruped creatures moving in different environments, and extracting key frame images in the video streams in different environments;
s2, responding to the extraction of a key frame image, and extracting key pixel points of a quadruped biological skeleton in the key frame image, wherein the quadruped biological skeleton comprises a basal node, a femoral node and a tibial node, and the key pixel points comprise a first connecting point for connecting the basal node and the femoral node, a second connecting point for connecting the femoral node and the tibial node and a foot point connected to the tibial node;
s3, responding to extraction of key pixel points, and performing deep learning on asynchronous state information corresponding to different environments by utilizing a neural network, wherein gait information comprises foot point coordinates X1, first node coordinates X2, second node coordinates X3, a pitch R representing the distance between first connecting points, a step distance M representing the distance of foot points moving in a period, a position coordinate X4 of the first connecting points when the foot is lifted to step, and a gait period T, and the gait period T is the time interval between two adjacent touchdowns of the same foot;
s4, identifying environment information of a current frame image in the visual equipment, judging corresponding gait information based on a neural network of deep learning, establishing a discretized gait model according to foot point coordinates in the gait information, and solving foot point coordinates of a next foot drop point when the four-foot bionic robot moves in the discretized gait model.
Further, the step S2 specifically includes: preprocessing the key frame image, and extracting the quadruped biological bones in the preprocessed foreground image by using a mathematical morphology method.
Further, the key frame image preprocessing specifically includes:
s11, calculating the pixel average value of the key frame image as the pixel of the background image, and identifying the background image of the current frame of the video stream according to the pixel of the background image;
s12, responding to a background image of a current frame of the video stream, performing gray-level subtraction operation on the current frame and the background image of the video stream to obtain a foreground image, and taking an absolute value of an operation result of the gray-level subtraction operation;
s13, responding to the obtained foreground image, and performing filtering processing on the foreground image by using a median filtering algorithm;
s14, performing binarization post-expansion corrosion treatment on the foreground image in response to the filtering treatment of the foreground image, and performing normalization treatment on the foreground image.
Further, the method for obtaining the coordinates of the footpoints and the coordinates of the nodes in the step S3 comprises the following steps: responding to four-foot biological skeletons of a foreground image of an extracted key frame image, solving the barycenter coordinates of the foreground image, taking a skeleton point closest to the barycenter coordinates in the four-foot biological skeletons as a coordinate origin, establishing a space rectangular coordinate system, and respectively solving the coordinates of a first node, a second node and a foot point in different environments in the space direct coordinate system.
Further, training of the neural network is as follows: and acquiring environment information and gait information in a video stream, wherein the environment information forms a data set Y, the gait information forms a data set W (X1, X2, X3, R, M and T), the data set Y and the data set W are in corresponding mapping relation, and the neural network receives the data set Y and the data set W to perform deep learning to obtain a function F, and Y=F (W).
Further, in step S4, the discretized gait model is established as follows: inserting N equidistant step discrete points in the step distance, wherein N is greater than or equal to 2, and the N step discrete points are marked as a set N1 (N1N, N1N-1, N1N-2 … … N1 in a robot space rectangular coordinate system 2 ) Coordinate points symmetric to the discrete step points along the vertical axis are denoted as a set N2 (N1N ', N1N-1', N1N-2' … … N1) 2 '), the direction of the straight line where X1N and X1N' are located is marked as a supporting phase, and the direction of the coordinates X4 and the coordinates of any point of the set N2 is marked as a swinging phase.
Further, in the step S4, the step of obtaining the coordinates of the foot point of the next foot falling point in the motion of the quadruped bionic robot in the discretized gait model specifically includes:
the coordinates X1 are respectively recorded asAnd->Wherein,is->The coordinates of the next foot point of the ipsilateral leg movement, wherein +.>For the left anterior extremity foot point coordinate, +.>For the coordinates of the foot points of the right anterior limb +.>For the left hind limb foot point coordinates +.>The coordinates of the foot points of the right hind limb;
calculating a foot point stability margin corresponding to the coordinate X1 using an sgn function
Wherein N is the number of discrete steps, and the value range of m is 0<m<2, V is the motion speed of the quadruped robot when the gait cycle is T.
Further, the equation of V is V= (N-2) (U/T), wherein U is the distance between any two adjacent discrete points of the single-foot step distance M,and->The relation of (2) is:
wherein random (1, 2 … … n) is as defined in S 1i When the value is less than 0, searching is continued within the range of 1-nUp to the calculated S 1i ≥0。
In a second aspect, the present application provides a robot gait learning system based on video streaming,
comprising the following units:
the image processing unit is configured to acquire motion video streams of quadruped creatures moving in different environments respectively and extract key frame images in the video streams in different environments;
an image skeleton information extraction unit configured to extract key pixel points of a quadruped biological skeleton in a key frame image in response to the key frame image, wherein the quadruped biological skeleton comprises a basal node, a femoral node and a tibial node, wherein the key pixel points comprise a first connection point at which the basal node and the femoral node are connected, a second connection point at which the femoral node and the tibial node are connected, and a foot point connected to the tibial node;
the neural network unit is configured to respond to the extraction of key pixel points, and deep learning is performed on asynchronous state information corresponding to different environments by utilizing the neural network, wherein the gait information comprises foot point coordinates X1, first node coordinates X2, second node coordinates X3, a pitch R representing the distance between first connecting points, a step distance M representing the distance of foot points moving in a period, a position coordinate X4 of the first connecting points during leg lifting and stepping and a gait period T, and the gait period T is the time interval between two adjacent touchdowns of the same foot;
the stabilized gait computing unit is configured to recognize the environment information of the current frame image in the visual equipment, judge corresponding gait information based on the neural network of deep learning, establish a discretized gait model according to the foot point coordinates in the gait information, and calculate the foot point coordinates of the next foot drop point in the motion of the four-foot bionic robot in the discretized gait model.
In a third aspect, embodiments of the present application provide a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method as described in any of the implementations of the first aspect.
The method and the system respectively acquire motion video streams of quadruped creatures moving in different environments, respond to extraction of key frame images, extract key pixel points of quadruped creature bones in the key frame images, utilize a neural network to carry out deep learning on asynchronous state information corresponding to different environments, identify environment information of a current frame image in visual equipment, establish a discretized gait model according to foot point coordinates in the gait information, and calculate foot point coordinates of a foot drop point when the quadruped bionic robot moves in the discretized gait model. According to the application, the feature points are extracted through massive videos to acquire skeleton actions, and the neural network deep learning method is utilized, so that the capability of searching for an optimal solution is high-speed, and further, the video feature point data is utilized to perform stable machine learning of bionic gait.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the accompanying drawings in which:
FIG. 1 is an overall flow diagram of one embodiment of a video streaming based robot gait learning method;
FIG. 2 is a schematic diagram of one embodiment of a video streaming based robotic gait learning system;
FIG. 3 is a schematic diagram of a discretized gait model of a robot gait learning method based on video streaming;
fig. 4 is a schematic diagram of a video stream based robotic gait learning system.
Detailed Description
The application is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be noted that, for convenience of description, only the portions related to the present application are shown in the drawings.
It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.
Referring to fig. 1, an embodiment of the present application provides a robot gait learning method based on video streaming, including the following steps:
step S101, respectively acquiring motion video streams of quadruped creatures moving in different environments, and extracting key frame images in the video streams in different environments, wherein the different environments comprise a flat horizontal plane, a flat slope, a rugged horizontal plane and a rugged slope;
in this embodiment, an image of one cycle of motion of the quadruped creature can be extracted as a key frame image, a motion analysis method can be adopted to extract a key frame, a video stream is obtained from video equipment, the light flow of object motion is analyzed, and a video frame with the minimum number of times of light flow movement in a video lens is selected as the extracted key frame each time. Therefore, a proper amount of key frames can be extracted from most video shots, and the extracted key frames can also effectively express the characteristics of video motion.
Step S102, in response to the extraction of the key frame image, extracting key pixel points of a quadruped biological skeleton in the key frame image, wherein the quadruped biological skeleton comprises a basal node, a femoral node and a tibial node, and the key pixel points comprise a first connecting point for connecting the basal node and the femoral node, a second connecting point for connecting the femoral node and the tibial node and a foot point connected to the tibial node;
in this embodiment, bio-skeletal keypoints are critical to describing bio-attitudes, predicting bio-behavior. Therefore, the detection of the key points of the biological skeleton is the basis of various computer vision tasks, such as action classification and abnormal behavior detection, so that the acquisition of the key pixel points of the biological skeleton of the quadruped is extremely important for researching the gait of the quadruped.
Step S103, in response to extracting key pixel points, deep learning is performed on asynchronous state information corresponding to different environments by using a neural network, wherein gait information comprises foot point coordinates X1, first node coordinates X2, second node coordinates X3, a pitch R representing the distance between first connecting points, a step distance M representing the distance of foot points moving in a period, a position coordinate X4 of the first connecting points when the foot is lifted and walked, and a gait period T, wherein the gait period T is the time interval between two adjacent touchdowns of the same foot;
in this embodiment, the convolutional neural network may be used to perform deep learning on the gait of the living beings in different environments, where deep learning is one of machine learning, and machine learning is a necessary path for implementing artificial intelligence. The deep learning concept is derived from the research of an artificial neural network, and a multi-layer sensor with a plurality of hidden layers is a deep learning structure. Deep learning forms more abstract high-level representation attribute categories or features by combining low-level features to discover distributed feature representations of data.
Step S104, identifying the environment information of the current frame image in the visual equipment, judging corresponding gait information based on a neural network of deep learning, establishing a discretized gait model according to foot point coordinates in the gait information, and solving foot point coordinates of the next foot falling point when the four-foot bionic robot moves in the discretized gait model so as to obtain more stable foot point coordinates.
In this embodiment, when the quadruped robot actually moves, the current frame image can be acquired through the vision equipment of the quadruped robot, the current environment is identified, the neural network is used for calculating the relevant information of the corresponding gait to be made, the control unit in the quadruped robot can control the position of the limb movement according to the gait information in the neural network, the distance between the four limbs in the gait information is discretized in the movement, whether the falling feet meet the stable state is judged, and if the falling feet do not meet the stable state, the falling feet are adjusted through the discretization model.
Further, the specific steps of the key frame image preprocessing are as follows:
step S201, the average value of pixels of the key frame image is calculated as pixels of the background image, and the background image of the current frame of the video stream is identified according to the pixels of the background image;
step S202, in response to a background image of a current frame of a video stream, performing gray-level subtraction operation on the current frame and the background image of the video stream to obtain a foreground image, and taking an absolute value of an operation result of the gray-level subtraction operation; namely, the result of subtracting the gray value of the background image from the gray value of the current frame of the video stream takes an absolute value;
step S203, in response to obtaining the foreground image, filtering the foreground image by using a median filtering algorithm;
step S204, in response to the filtering processing of the foreground image, binarizing the foreground image and then expanding and corroding the foreground image, and normalizing the foreground image.
In this embodiment, the currently acquired image frame and the background image are subtracted to obtain a gray level image of the target motion area, the gray level image is thresholded to extract the motion area, and in order to avoid the influence of ambient light variation, the background image is updated according to the currently acquired image frame. The algorithm is simpler, and the influence of ambient light is overcome.
Responding to the normalization processing of the foreground image, extracting the quadruped biological skeleton in the normalized foreground image by using a mathematical morphology method, and obtaining key pixel points of the quadruped biological skeleton, wherein the quadruped biological skeleton comprises a basal node, a femoral node and a tibial node. The normalization process involves capturing a small circumscribing frame of the image, removing too much background information that is not related to the foreground graphics.
In this embodiment, the mathematical morphology method is specifically: and carrying out three-dimensional reconstruction on the skeleton image of the multi-foot organism to obtain key pixel points of the skeleton. The bone two-dimensional image edge extraction of the multi-foot organism is the first step of three-dimensional reconstruction, two-dimensional boundary coordinates are obtained, and a reconstructed kenili image is obtained through three-dimensional interpolation.
First, a bone contour composed of continuous, closed edge lines is obtained from a two-dimensional image edge extraction of a bone of a multi-foot creature, and a three-dimensional reconstruction of the bone image is performed based on the bone contour. The bones of the multi-foot organisms are coated in tissues such as skin, muscle and the like, and the bones, other biological tissues and the background show different gray values on a bone image due to different components, and the edges in the image are local extreme points of gray or the collection of points where the gray changes sharply. Nonlinear wavelet edge detection or neural network edge detection algorithms may be used.
Referring to fig. 2, in response to extracting a quadruped biological skeleton of a foreground image, a centroid coordinate of the foreground image is obtained, a skeleton point closest to the centroid coordinate in the quadruped biological skeleton is taken as a coordinate origin, a space rectangular coordinate system is established, and coordinates of a first node, a second node and a foot point in different environments are respectively obtained in the space direct coordinate system.
Further solving the gait of the quadruped creature in the video stream under different environments according to the coordinates of the first connecting point, the second connecting point and the foot point, wherein the gait comprises a foot point coordinate X1, a first node coordinate X2, a second node coordinate X3, a pitch R, a step distance M, a second joint position coordinate X4 during leg lifting and a gait period T, and the gait period T is the time interval between two adjacent touchdowns of the same foot;
collecting environment information and gait information in a video stream, wherein the environment information forms a data set Y, the gait information forms a data set W (X1, X2, X3, R, M and T), and the data set Y and the data set W are in a corresponding mapping relation, and the relation between the two is expressed by a function F: y=f (W).
In this embodiment, the recognition of the environmental information uses the appearance information of the scene image, such as the global features of color, texture, etc., and after the environmental image is obtained, the robot first roams in the working environment by a person or automatically for learning, and acquires a large number of image sequences of the environment. Then, manual or automatic environmental classification and learning is performed, and classification marking is typically performed manually. The technology mainly used in the aspects of environment classification and learning is a multi-dimensional histogram method, a principal component analysis method and the like.
In this embodiment, the quadruped robot acquires environmental state information in a motion state through a vision device, acquires corresponding gait by using neural network learning to adjust the positions of the foot points of the quadruped, and converts the spatial direct coordinates under the skeleton map into a robot spatial rectangular coordinate system.
With continued reference to fig. 3, a discretized gait model is established, N equidistant stride discrete points are inserted into the stride M, where N is greater than or equal to 2, and the N stride discrete points are recorded as a set N1 (N1N, N1N-1, N1N-2 … … N1) in the robot space rectangular coordinate system 2 ) Coordinate points symmetric to the discrete step points along the vertical axis are denoted as a set N2 (N1N ', N1N-1', N1N-2' … … N1) 2 '), the directions of X1N to X1N' are marked as supporting phases, and the directions of the coordinates X4 and any point coordinates of the set N2 are marked as swinging phases;
calculating the motion speed V= (N-2) (U/T) of the quadruped robot based on the discretized gait pattern and the gait period T, wherein U is the distance between any two adjacent discrete points of the single-foot step distance M;
the coordinates X1 are respectively recorded asAnd-> Is thatIs the next foot point coordinate of the leg, wherein +.>For the left anterior branch foot point coordinate, +.>For the coordinates of the right anterior branch foot point +.>For the left posterior branch foot point coordinate, +.>The coordinates of the right rear foot point;
by sgnFunction calculation of the foot stability margin corresponding to coordinate X1The Sgn function is a library function in the software OpenAI.
S 1i ＝[sgn(N+m-X 1i -V)+1]Wherein N is the number of discrete steps, and the value range of m is 0<m<2, V is the movement speed of the quadruped robot when the gait cycle is T;
and->The relation of (2) is:
when the value of the stable margin S is greater than or equal to zero, one position coordinate of the foot point is the next position coordinate obtained by adding V to the current coordinates of x, y and Z, and if the value of the stable margin S is smaller than zero, the values of 1-n are searched until the value of the stable margin S is greater than zero.
Referring to fig. 4, fig. 4 is a robot gait learning system based on video streaming, including:
an image processing unit 501, configured to collect motion video streams of quadruped creatures moving under different environments, and extract key frame images under different environments in the video streams, where the different environments include a flat horizontal plane, a flat slope, a rugged horizontal plane, and a rugged slope;
an image skeleton information extraction unit 502 configured to extract key pixel points of a quadruped biological skeleton in a key frame image in response to the key frame image, wherein the quadruped biological skeleton includes a basal node, a femoral node, and a tibial node, wherein the key pixel points include a first connection point at which the basal node and the femoral node are connected, a second connection point at which the femoral node and the tibial node are connected, and a foot point connected to the tibial node;
the neural network unit 503 is configured to perform deep learning on asynchronous state information corresponding to different environments by using a neural network in response to extracting key pixel points, where the gait information includes a foot point coordinate X1, a first node coordinate X2, a second node coordinate X3, a pitch R representing a distance between the first connection points, a step distance M representing a distance of moving the foot point in a period, a position coordinate X4 of the first connection point when lifting the leg and taking a step, and a gait cycle T, and the gait cycle T is a time interval between two adjacent touchdowns of the same foot;
the stabilized gait computing unit 504 is configured to identify the environmental information of the current frame image in the visual device, judge corresponding gait information based on the neural network of deep learning, establish a discretized gait model according to the foot point coordinates in the gait information, and calculate the foot point coordinates of the next foot drop point in the motion of the four-foot bionic robot in the discretized gait model so as to obtain more stabilized foot point coordinates.
As another aspect, the present application also provides a computer-readable medium, which may be an electronic device integrated thereon; or may exist alone without being incorporated into the electronic device. The computer-readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform the steps associated with the video stream-based robot gait learning method described in this embodiment.
It should be noted that, the above-mentioned robot gait learning method based on video stream, robot gait learning system based on video stream and computer readable storage medium belong to one general inventive concept, and the contents thereof are mutually applicable.
Those skilled in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, where the program may be stored in a non-volatile computer readable storage medium, and where the program, when executed, may include processes in the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above description is only illustrative of the preferred embodiments of the present application and of the principles of the technology employed. It will be appreciated by persons skilled in the art that the scope of the application referred to in the present application is not limited to the specific combinations of the technical features described above, but also covers other technical features formed by any combination of the technical features described above or their equivalents without departing from the inventive concept described above. Such as the above-mentioned features and the technical features disclosed in the present application (but not limited to) having similar functions are replaced with each other.
Claims (10)
1. The robot gait learning method based on the video stream is characterized by comprising the following steps:
s1, respectively acquiring motion video streams of quadruped creatures moving in different environments, and extracting key frame images in the video streams in the different environments;
s2, responding to the extraction of the key frame image, and extracting key pixel points of a quadruped biological skeleton in the key frame image, wherein the quadruped biological skeleton comprises a basal node, a femoral node and a tibial node, and the key pixel points comprise a first connecting point for connecting the basal node and the femoral node, a second connecting point for connecting the femoral node and the tibial node and a foot point connected to the tibial node;
s3, responding to the extraction of the key pixel points, and performing deep learning on asynchronous state information corresponding to different environments by utilizing a neural network, wherein the gait information comprises foot point coordinates X1, first node coordinates X2, second node coordinates X3, a pitch R representing the distance between the first connecting points, a step distance M representing the distance of the foot points moving in a period, a position coordinate X4 of the first connecting points when the foot is lifted to step, and a gait period T, and the gait period T is the time interval between two adjacent touchdowns of the same foot;
s4, identifying environment information of a current frame image in the visual equipment, judging corresponding gait information based on the neural network of deep learning, establishing a discretized gait model according to foot point coordinates in the gait information, and solving foot point coordinates of a next foot drop point when the four-foot bionic robot moves in the discretized gait model.
2. The robot gait learning method based on the video stream according to claim 1, wherein step S2 specifically comprises: preprocessing the key frame image, and extracting quadruped biological bones in the preprocessed foreground image by using a mathematical morphology method.
3. The robot gait learning method based on the video stream according to claim 2, wherein the key frame image preprocessing specifically includes:
s11, obtaining the pixel average value of the key frame image as the pixel of the background image, and identifying the background image of the current frame of the video stream according to the pixel of the background image;
s12, responding to a background image of a current frame of the video stream, performing gray-level subtraction operation on the current frame of the video stream and the background image to obtain a foreground image, and taking an absolute value of an operation result of the gray-level subtraction operation;
s13, responding to the obtained foreground image, and performing filtering processing on the foreground image by using a median filtering algorithm;
s14, performing binarization post-expansion corrosion processing on the foreground image in response to the filtering processing of the foreground image, and performing normalization processing on the foreground image.
4. The robot gait learning method based on the video stream according to claim 1, wherein the method for obtaining the coordinates of the footpoints and the coordinates of the nodes in S3 is as follows: responding to four-foot biological bones of a foreground image of the key frame image, solving the barycenter coordinates of the foreground image, taking a bone point closest to the barycenter coordinates in the four-foot biological bones as a coordinate origin, establishing a space rectangular coordinate system, and respectively solving the coordinates of the first node, the second node and the foot point in different environments in the space direct coordinate system.
5. The robot gait learning method based on video streaming of claim 1, wherein the training of the neural network is as follows: and acquiring environment information and gait information in the video stream, wherein the environment information forms a data set Y, the gait information forms a data set W (X1, X2, X3, R, M and T), the data set Y and the data set W are in corresponding mapping relation, and the neural network receives the data set Y and the data set W and performs deep learning to obtain a function F, and Y=F (W).
6. The robot gait learning method based on the video stream according to claim 1, wherein the discrete gait model in S4 is established as follows: inserting N equidistant step discrete points in the step distance, wherein N is more than or equal to 2, and the N step discrete points are marked as a set N1 (N1N, N1N-1, N1N-2 … … N1 in the robot space rectangular coordinate system 2 ) The coordinate points symmetrical to the discrete step points along the vertical axis are marked as a set N2 (N1N ', N1N-1', N1N)-2’……N1 2 'the direction of the straight line where X1N and X1N' are located is denoted as limb support phase, and the direction of the coordinates X4 and the coordinates of any point of the set N2 is denoted as swing phase.
7. The robot gait learning method based on the video stream according to claim 1, wherein the step of obtaining the foot point coordinates of the next foot drop point in the motion of the four-foot bionic robot in the discretized gait model in S4 is specifically:
the coordinates X1 are respectively recorded asAnd->Wherein (1)>Is->The coordinates of the next foot point of the ipsilateral leg movement, wherein +.>For the left anterior extremity foot point coordinate, +.>Is the coordinates of the foot point of the right anterior limb,for the left hind limb foot point coordinates +.>The coordinates of the foot points of the right hind limb;
calculating a foot point stability margin corresponding to the coordinate X1 by utilizing sgn function
S 1i ＝[sgn(N+m-X 1i -V)+1]Wherein N is the number of discrete steps, and the value range of m is 0<m<2, V is the motion speed of the quadruped robot when the gait cycle is T.
8. The robot gait learning method based on video streaming of claim 7, wherein the equation of V is v= (N-2) (U/T), wherein U is the distance between any two adjacent discrete points in a single-foot step M, theAnd->The relation of (2) is: />
Wherein, random (1, 2.). N) is when the S 1i When the value is less than 0, searching is continued within the range of 1-nUp to the calculated S 1i ≥0。
9. A robot gait learning system based on video streaming, comprising the following units:
the image processing unit is configured to acquire motion video streams of quadruped creatures moving in different environments respectively and extract key frame images in the video streams in the different environments;
an image skeleton information extraction unit configured to extract key pixel points of a quadruped biological skeleton in the key frame image in response to the key frame image, wherein the quadruped biological skeleton comprises a basal node, a femoral node and a tibial node, wherein the key pixel points comprise a first connection point at which the basal node and the femoral node are connected, a second connection point at which the femoral node and the tibial node are connected, and a foot point connected to the tibial node;
the neural network unit is configured to respond to the extraction of the key pixel points, and performs deep learning on asynchronous state information corresponding to different environments by utilizing the neural network, wherein the gait information comprises foot point coordinates X1, first node coordinates X2, second node coordinates X3, a pitch R representing the distance between the first connecting points, a step distance M representing the distance of the foot points moving in a period, a position coordinate X4 of the first connecting points when the foot is lifted to step, and a gait period T, and the gait period T is the time interval between two adjacent touchdowns of the same foot;
the stabilized gait computing unit is configured to recognize the environment information of the current frame image in the visual equipment, judge corresponding gait information based on the neural network of the deep learning, establish a discretized gait model according to the foot point coordinates in the gait information, and calculate the foot point coordinates of the next foot drop point when the four-foot bionic robot moves in the discretized gait model.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110438406.8A CN113095268B (en) | 2021-04-22 | 2021-04-22 | Robot gait learning method, system and storage medium based on video stream |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110438406.8A CN113095268B (en) | 2021-04-22 | 2021-04-22 | Robot gait learning method, system and storage medium based on video stream |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113095268A CN113095268A (en) | 2021-07-09 |
| CN113095268B true CN113095268B (en) | 2023-11-21 |
Family
ID=76679780
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110438406.8A Active CN113095268B (en) | 2021-04-22 | 2021-04-22 | Robot gait learning method, system and storage medium based on video stream |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113095268B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113688701B (en) * | 2021-08-10 | 2022-04-22 | 江苏仁和医疗器械有限公司 | Facial paralysis detection method and system based on computer vision |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017133009A1 (en) * | 2016-02-04 | 2017-08-10 | 广州新节奏智能科技有限公司 | Method for positioning human joint using depth image of convolutional neural network |
| WO2019032622A1 (en) * | 2017-08-07 | 2019-02-14 | The Jackson Laboratory | Long-term and continuous animal behavioral monitoring |
| CN110801233A (en) * | 2019-11-05 | 2020-02-18 | 上海电气集团股份有限公司 | Human body gait monitoring method and device |
| CN111027432A (en) * | 2019-12-02 | 2020-04-17 | 大连理工大学 | Gait feature-based visual following robot method |
-
2021
- 2021-04-22 CN CN202110438406.8A patent/CN113095268B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017133009A1 (en) * | 2016-02-04 | 2017-08-10 | 广州新节奏智能科技有限公司 | Method for positioning human joint using depth image of convolutional neural network |
| WO2019032622A1 (en) * | 2017-08-07 | 2019-02-14 | The Jackson Laboratory | Long-term and continuous animal behavioral monitoring |
| CN110801233A (en) * | 2019-11-05 | 2020-02-18 | 上海电气集团股份有限公司 | Human body gait monitoring method and device |
| CN111027432A (en) * | 2019-12-02 | 2020-04-17 | 大连理工大学 | Gait feature-based visual following robot method |
Non-Patent Citations (2)
| Title |
|---|
| 基于神经网络的四足机器人运动目标检测;吴宝江;张怡;;数字技术与应用(第11期);全文 * |
| 基于视频的步态特征提取技术及其应用;郭建林;高原;;电脑知识与技术(第S2期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113095268A (en) | 2021-07-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111126272B (en) | Posture acquisition method, and training method and device of key point coordinate positioning model | |
| Rodríguez et al. | Agents and computer vision for processing stereoscopic images | |
| CN111160294B (en) | Gait recognition method based on graph convolution network | |
| CN113658211B (en) | User gesture evaluation method and device and processing equipment | |
| CN110427900A (en) | A kind of method, apparatus and equipment of intelligent guidance body-building | |
| CN113095268B (en) | Robot gait learning method, system and storage medium based on video stream | |
| Seo et al. | A yolo-based separation of touching-pigs for smart pig farm applications | |
| CN110543817A (en) | Pedestrian re-identification method based on posture guidance feature learning | |
| Zhang et al. | Directional PointNet: 3D environmental classification for wearable robotics | |
| CN108664942A (en) | The extracting method and video classification methods of mouse video multidimensional characteristic value | |
| CN115346272A (en) | Real-time tumble detection method based on depth image sequence | |
| CN113177564B (en) | Computer vision pig key point identification method | |
| CN113033501A (en) | Human body classification method and device based on joint quaternion | |
| CN112001240B (en) | Living body detection method, living body detection device, computer equipment and storage medium | |
| CN113920020B (en) | Human body point cloud real-time restoration method based on depth generation model | |
| Kuang et al. | An effective skeleton extraction method based on Kinect depth image | |
| CN114596632A (en) | Medium-large quadruped animal behavior identification method based on architecture search graph convolution network | |
| Ramasamy et al. | Object detection and tracking in video using deep learning techniques: A review | |
| WO2005125210A1 (en) | Methods and apparatus for motion capture | |
| CN113255514A (en) | Behavior identification method based on local scene perception graph convolutional network | |
| Hazra et al. | A pilot study for investigating gait signatures in multi-scenario applications | |
| Zhang et al. | Recent development in human motion and gait prediction | |
| Zelman et al. | Nearly automatic motion capture system for tracking octopus arm movements in 3D space | |
| Marfil et al. | A novel hierarchical framework for object-based visual attention | |
| Xiong et al. | Enhancing Human Motion Prediction through Joint-based Analysis and AVI Video Conversion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |