CN112315456B - Human body action prediction method based on jump attention mechanism - Google Patents

Human body action prediction method based on jump attention mechanism Download PDF

Info

Publication number
CN112315456B
CN112315456B CN202011067849.2A CN202011067849A CN112315456B CN 112315456 B CN112315456 B CN 112315456B CN 202011067849 A CN202011067849 A CN 202011067849A CN 112315456 B CN112315456 B CN 112315456B
Authority
CN
China
Prior art keywords
human body
human
skeleton point
decoder
features
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011067849.2A
Other languages
Chinese (zh)
Other versions
CN112315456A (en
Inventor
舒祥波
张瑞鹏
宋砚
唐金辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing University of Science and Technology
Original Assignee
Nanjing University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University of Science and Technology filed Critical Nanjing University of Science and Technology
Priority to CN202011067849.2A priority Critical patent/CN112315456B/en
Publication of CN112315456A publication Critical patent/CN112315456A/en
Application granted granted Critical
Publication of CN112315456B publication Critical patent/CN112315456B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1118—Determining activity level
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1121—Determining geometric values, e.g. centre of rotation or angular range of movement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7271—Specific aspects of physiological measurement analysis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7271—Specific aspects of physiological measurement analysis
    • A61B5/7275—Determining trends in physiological measurement data; Predicting development of a medical condition based on physiological measurements, e.g. determining a risk factor

Abstract

The invention discloses a human body action prediction method based on a jump connection attention mechanism, which comprises the following steps: inputting the normalized human skeleton point coordinates of each frame into an encoder consisting of a multilayer self-updating convolution gating circulation network, and extracting the layering characteristics of the human skeleton point sequence; aggregating the layered features extracted at the last time step of the encoder to obtain a long-term semantic vector of the human body action; taking hidden variables of each layer of the last time step of the encoder as initial hidden variables of the decoder, and calculating the weight of each layer of characteristics of the decoder according to human body action long-term semantic vectors of the encoder by using a jump attention mechanism; connecting the adjusted hierarchical features and the human motion long-term semantic vector into new features, generating the variable quantity of a human skeleton point frame to be predicted and the previous frame through a convolutional neural network, adding the variable quantity to the input skeleton points, and then performing normalized data reduction to obtain the predicted value of the human motion skeleton points. The method can effectively predict the human body action and has high accuracy.

Description

Human body action prediction method based on jump attention mechanism
Technical Field
The invention relates to a human body motion prediction technology, in particular to a human body motion prediction method based on a jump-over attention mechanism.
Background
The human body action prediction is to automatically predict the future action sequence according to the observed human body action sequence, and can be applied to most human-computer interaction systems. The human body action prediction has wide application scenes: the system has supplementary effect on industrial automation and automatic driving technology and has important effect on interaction of intelligent robots and human beings.
Human motion prediction has attracted more and more researchers' attention, with the main challenge that early human motion prediction tasks rarely target human behavior; the long-term human body action prediction result tends to be constant action; the prediction effect is poor for aperiodic actions.
Disclosure of Invention
The invention aims to provide a human body motion prediction method based on a jump connection attention mechanism, which has a good long-term prediction effect on periodic and aperiodic human body motions.
The technical scheme for realizing the purpose of the invention is as follows: a human body action prediction method based on a jump-over attention mechanism comprises the following steps:
step 1, inputting a section of human skeleton point coordinate sequence, and carrying out normalization processing on the human skeleton point coordinate sequence to obtain a processed section of human skeleton point coordinate sequence;
step 2, inputting the human skeleton point coordinates of each frame into an encoder consisting of a multilayer self-updating convolution gating circulation network, and extracting the layering characteristics of the human skeleton point sequence;
step 3, aggregating the hierarchical features extracted by the gated loop network of the last time step to obtain a long-term semantic vector of the human body action;
step 4, inputting the layered features extracted by the encoder and the previous frame of human skeleton points into a decoder to obtain layered features;
step 5, calculating the weight of each layer of features of the decoder according to the human body action long-term semantic vector of the encoder by using a jump attention mechanism so as to adjust the importance of the layered features;
step 6, connecting the adjusted hierarchical features and the human body action long-term semantic vectors into new features, and generating the variation of the human body skeleton point frame to be predicted and the previous frame by the new features through a convolutional neural network;
and 7, adding the variable quantity of the skeleton points and the input frame of the decoder, and then restoring the normalized data to obtain the final predicted value of the human action skeleton points.
Compared with the prior art, the invention has the remarkable advantages that; (1) the method is based on the jump attention mechanism, can effectively predict the human body action, and has high prediction accuracy; (2) the method has good long-term prediction effect on periodic and aperiodic human body actions.
Drawings
Fig. 1 is a flowchart of a human body motion prediction method based on a jump attention mechanism according to the present invention.
Fig. 2 is an effect diagram of a human body motion prediction method based on a jump attention mechanism.
Detailed Description
The invention is described in further detail below with reference to the accompanying drawings:
as shown in FIG. 1, a human body motion prediction method based on a jump attention mechanism comprises four processes of extracting human body skeleton point sequence hierarchical features, extracting long-term semantic vectors, calculating weights of decoder hierarchical features and generating skeleton point variable quantities.
Extracting human skeleton point sequence layering characteristics: inputting the normalized human skeleton point coordinates of each frame into an encoder consisting of a multilayer self-updating convolution gating circulation network, and extracting the hierarchical characteristics of the human skeleton point sequence;
extracting a long-term semantic vector: aggregating the layered features extracted at the last time step of the encoder to obtain a long-term semantic vector of the human body action;
calculating weights of decoder hierarchical features: taking hidden variables of each layer of the last time step of the encoder as initial hidden variables of the decoder, and calculating the weight of each layer of features of the decoder according to human body action long-term semantic vectors of the encoder by using a jump attention mechanism so as to adjust the importance of the hierarchical features;
generating the bone point variation: and connecting the adjusted hierarchical features and the human motion long-term semantic vector into new features, generating the variation of a human skeleton point frame to be predicted and a previous frame by the new features through a convolutional neural network, adding the variation of the skeleton point to the input skeleton point of a decoder, and restoring normalized data to finally obtain the predicted value of the human motion skeleton point. The method can effectively predict the human body action and has high prediction accuracy.
The steps of the above method will be described in detail below.
The method for extracting the human skeleton point sequence layering characteristics comprises the following steps:
step 1), inputting a section of human body skeleton point coordinate sequence, and carrying out normalization processing on the human body skeleton point coordinate sequence to obtain a processed section of human body skeleton point coordinate sequence X ═ X1,…,xs,…,xS]Which describes the activity of a person, of which there is xs∈RNAnd the skeleton key point coordinates of the human body at the time step s are shown, and N is the number of joints of the human body.
Step 2), the sequence X obtained in step 1) ═ X1,…,xs,…,xs]Is input toIn an encoder composed of a multi-layer Self-updating convolution gating circulation network (Self-updating ConvGRU), the calculation formula of one layer of the multi-layer Self-updating convolution gating circulation network (Self-updating ConvGRU) at the time step t is as follows:
updating the door ztCalculated from the formula:
zt＝σ(Wzh*ht-1+Wzx*xt+bz),
reset gate rtCalculated from the formula:
rt＝σ(Wrh*ht-1+Wrx*xt+br),
candidate hidden variables
Figure BDA0002714340890000031
Calculated from the formula:
Figure BDA0002714340890000032
self-updating door hhtCalculated from the formula:
Figure BDA0002714340890000033
then candidate hidden variables
Figure BDA0002714340890000034
Is updated again:
Figure BDA0002714340890000035
finally, the current hidden variable h is obtainedt：
Figure BDA0002714340890000036
Wherein h ist-1Is the last time step tHidden variable of-1, xtIs a hidden variable of the previous layer at time step t, σ () is a sigmoid activation function, tanh () is a tanh activation function, W*Is a learnable transformation parameter, b*Is a bias term, is a convolution operation,
Figure BDA0002714340890000037
are multiplied by element.
Step 3) obtaining hidden variables of each layer of the input sequence at the last time step of the encoder through the step 2)
Figure BDA0002714340890000038
Where K denotes the number of layers of the encoder, these hidden variables are the hierarchical characteristics of the input sequence.
The extraction of the long-term semantic vector comprises the following steps:
step 4) the hierarchical characteristics of the input sequence obtained in the step 3)
Figure BDA0002714340890000039
And (3) aggregating to form a human body action long-term semantic vector:
Figure BDA00027143408900000310
where g () is an aggregation function, a convolution operation is used as the aggregation function.
The method for calculating the weight of the decoder hierarchical characteristics mainly comprises the following steps:
step 5) constructing a decoder by adopting the same calculation formula in the step 2), taking hidden variables of each layer of the encoder obtained in the step 3) as initial hidden variables of the decoder, taking human skeleton points obtained by predicting at the time step t-1 as input of the decoder at the time step t, and finally obtaining hidden variables of the self-updating convolution gated cyclic network of each layer
Figure BDA0002714340890000041
Step 6), hidden variables from decoder by convolution operation
Figure BDA0002714340890000042
Extracting the characteristics of each layer at time step t
Figure BDA0002714340890000043
Step 7), calculating the weight of each layer of features of the decoder according to the long-term semantic vector of the human body action, and setting the features of the nth layer of the time step t
Figure BDA0002714340890000044
The score calculation formula is as follows:
Figure BDA0002714340890000045
wherein WfIs a weight matrix, bfIs a bias vector;
the score represents the importance of the features of the decoder layers, and is normalized using the Softmax function:
Figure BDA0002714340890000046
finally, the
Figure BDA0002714340890000047
Representing the adjusted characteristics of the decoder layers.
Generating the bone point variation comprises the following steps:
step 8), connecting the characteristics of the time step t adjusted by the decoder in the step 7) with the long-term semantic vector of the human action in the step 4), and generating the variation of the human skeleton points through convolution operation:
Figure BDA0002714340890000048
the predicted human body bone point coordinate at time step t +1 can be obtained by adding the human body bone point variation at time step t and the input human body bone point coordinate of the decoder.
Fig. 2 is a diagram of the effect of the human body motion prediction method based on the jump-joint attention mechanism, in which the first four rows in the first row are real bone points, the first four rows are predicted bone points from the fifth row, and the second row is real bone points for comparison.

Claims (3)

1. A human body action prediction method based on a jump-over attention mechanism is characterized by comprising the following steps:
step 1, inputting a section of human skeleton point coordinate sequence, and carrying out normalization processing on the human skeleton point coordinate sequence to obtain a processed section of human skeleton point coordinate sequence;
step 2, inputting the human skeleton point coordinates of each frame into an encoder consisting of a multilayer self-updating convolution gating circulation network, and extracting the layering characteristics of the human skeleton point sequence;
step 3, aggregating the hierarchical features extracted by the gated loop network of the last time step to obtain a long-term semantic vector of the human body action;
step 4, inputting the layered features extracted by the encoder and the previous frame of human skeleton points into a decoder to obtain layered features;
step 5, calculating the weight of each layer of features of the decoder according to the human body action long-term semantic vector of the encoder by using a jump attention mechanism so as to adjust the importance of the layered features;
step 6, connecting the adjusted hierarchical features and the human body action long-term semantic vectors into new features, and generating the variation of the human body skeleton point frame to be predicted and the previous frame by the new features through a convolutional neural network;
step 7, adding the variable quantity of the skeleton points and the input frame of the decoder, and then restoring the normalized data to obtain a final predicted value of the human body action skeleton points;
the step 7 specifically comprises the following steps:
the variation of the human skeleton point of the last time step t
Figure DEST_PATH_IMAGE001
And the input human skeleton point coordinate y of the last time step t decodertAdding to obtain the human skeleton point coordinate y of the predicted time step t +1t+1：
Figure DEST_PATH_IMAGE002
Finally, the predicted human skeleton point y of the time step t +1t+1And restoring the normalized data to finally obtain the predicted value of the human body action skeletal point at the time step t + 1.
2. The human motion prediction method based on the jump attention mechanism according to claim 1, wherein the step 5 specifically comprises the following steps:
step 501, calculating the weight of each layer of features of the decoder according to the long-term semantic vector of human body motion, and setting the features of the nth layer of a time step t
Figure DEST_PATH_IMAGE003
The score calculation formula is as follows:
Figure DEST_PATH_IMAGE004
wherein WfIs a weight matrix, bfIs a bias vector, C represents a long-term semantic vector of human body motion;
step 502, the score represents the importance of each layer feature of the decoder, and the score is normalized by using a Softmax function
Figure DEST_PATH_IMAGE005
：
Figure DEST_PATH_IMAGE006
Finally, the
Figure DEST_PATH_IMAGE007
Representing the adjusted characteristics of the decoder layers.
3. The human motion prediction method based on the jump attention mechanism according to claim 2, wherein the step 6 specifically comprises the following steps:
connecting the characteristics of the time step t after being adjusted by the decoder with the long-term semantic vector of the human body action, and generating the variable quantity of the human body skeleton point through convolution operation:
Figure DEST_PATH_IMAGE008
。
CN202011067849.2A 2020-10-07 2020-10-07 Human body action prediction method based on jump attention mechanism Active CN112315456B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011067849.2A CN112315456B (en) 2020-10-07 2020-10-07 Human body action prediction method based on jump attention mechanism

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011067849.2A CN112315456B (en) 2020-10-07 2020-10-07 Human body action prediction method based on jump attention mechanism

Publications (2)

Publication Number Publication Date
CN112315456A CN112315456A (en) 2021-02-05
CN112315456B true CN112315456B (en) 2022-02-11

Family

ID=74314478

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011067849.2A Active CN112315456B (en) 2020-10-07 2020-10-07 Human body action prediction method based on jump attention mechanism

Country Status (1)

Country Link
CN (1) CN112315456B (en)

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012008868A (en) * 2010-06-25 2012-01-12 Toshiba Corp Display processing device, display processing method, and display processing program
US9839392B2 (en) * 2011-07-20 2017-12-12 Elminda Ltd. Method and system for estimating brain concussion
US10405045B2 (en) * 2015-12-14 2019-09-03 Google Llc Systems and methods for estimating user attention
US11076766B2 (en) * 2017-04-19 2021-08-03 Beijing Shunyuan Kaihua Technology Limited Systems and methods for providing user insights based on heart rate variability
US10970527B2 (en) * 2017-09-01 2021-04-06 Digital Dream Labs, Llc Robot attention detection
CN109961444B (en) * 2019-03-01 2022-12-20 腾讯科技（深圳）有限公司 Image processing method and device and electronic equipment
CN110580456A (en) * 2019-08-22 2019-12-17 南京理工大学 Group activity identification method based on coherent constraint graph long-time memory network
CN111242048A (en) * 2020-01-15 2020-06-05 南京理工大学 Human skeleton action prediction method based on space-time cooperative attention circulation neural network

Also Published As

Publication number Publication date
CN112315456A (en) 2021-02-05

Similar Documents

Publication Publication Date Title
CN108448610B (en) Short-term wind power prediction method based on deep learning
Qiang et al. Reinforcement learning model, algorithms and its application
Gencay et al. Nonlinear modelling and prediction with feedforward and recurrent networks
Meireles et al. A comprehensive review for industrial applicability of artificial neural networks
Xu et al. Evolutionary extreme learning machine–based on particle swarm optimization
US20140032459A1 (en) Apparatus and methods for generalized state-dependent learning in spiking neuron networks
Zhu et al. Masked contrastive representation learning for reinforcement learning
Yesil et al. Fuzzy cognitive maps learning using artificial bee colony optimization
Yang et al. A novel self-constructing radial basis function neural-fuzzy system
CN114723065A (en) Optimal strategy obtaining method and device based on double-layer deep reinforcement learning model
CN113469891A (en) Neural network architecture searching method, training method and image completion method
Salimi et al. Extended mixture of MLP experts by hybrid of conjugate gradient method and modified cuckoo search
Zhang et al. Discrete time convolution for fast event-based stereo
Chen et al. Policy gradient from demonstration and curiosity
Schneider Causal cognitive architecture 1: Integration of connectionist elements into a navigation-based framework
Cao et al. An overview of deep reinforcement learning
CN112315456B (en) Human body action prediction method based on jump attention mechanism
Wang et al. Consciousness‐driven reinforcement learning: An online learning control framework
Cheng et al. Human motion prediction using adaptable neural networks
Kupwiwat et al. Deep deterministic policy gradient and graph convolutional network for bracing direction optimization of grid shells
CN115091467A (en) Intent prediction and disambiguation method and system based on fuzzy Petri net
Takano et al. Spatio-temporal structure of human motion primitives and its application to motion prediction
Hu et al. Time series prediction with a weighted bidirectional multi-stream extended Kalman filter
Terziyska A distributed adaptive neuro-fuzzy network for chaotic time series prediction
Xia et al. A conditional classification recurrent RBM for improved series mid-term forecasting

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant