CN111539289A - Method and device for identifying action in video, electronic equipment and storage medium - Google Patents
Method and device for identifying action in video, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111539289A CN111539289A CN202010301132.3A CN202010301132A CN111539289A CN 111539289 A CN111539289 A CN 111539289A CN 202010301132 A CN202010301132 A CN 202010301132A CN 111539289 A CN111539289 A CN 111539289A
- Authority
- CN
- China
- Prior art keywords
- video frame
- frame sequence
- action
- motion
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000009471 action Effects 0.000 title claims abstract description 234
- 238000000034 method Methods 0.000 title claims abstract description 58
- 230000033001 locomotion Effects 0.000 claims abstract description 167
- 238000000605 extraction Methods 0.000 claims description 79
- 238000009826 distribution Methods 0.000 claims description 38
- 238000012549 training Methods 0.000 claims description 29
- 230000004927 fusion Effects 0.000 claims description 20
- 230000002123 temporal effect Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 10
- 239000000126 substance Substances 0.000 claims description 7
- 230000000875 corresponding effect Effects 0.000 description 33
- 238000013507 mapping Methods 0.000 description 18
- 238000004891 communication Methods 0.000 description 10
- 238000013527 convolutional neural network Methods 0.000 description 9
- 230000009466 transformation Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000002372 labelling Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000007123 defense Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
- G06V20/42—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items of sport video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention provides a method and a device for identifying actions in a video, electronic equipment and a storage medium. The method comprises the following steps: determining a first sequence of video frames to be identified; inputting the first video frame sequence to be recognized into a motion recognition model to obtain a motion recognition result output by the motion recognition model; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence. According to the method, the device, the electronic equipment and the storage medium for recognizing the action in the video, provided by the embodiment of the invention, the video frame sequence to be recognized is input into the action recognition model which is trained in advance, and the action recognition model outputs one or more types of actions corresponding to each moment in the video frame sequence to be recognized, so that the recognition of the actions of multiple types which occur at the same time is realized, and the loss of information is effectively avoided.
Description
Technical Field
The present invention relates to the field of video technologies, and in particular, to a method and an apparatus for identifying an action in a video, an electronic device, and a storage medium.
Background
The video-based action recognition can be applied to multiple fields, such as behavior analysis, man-machine interaction, public safety, intelligent monitoring and the like.
There is also a wide demand for motion recognition of sports videos such as soccer, basketball, volleyball, and the like. For example, when a sports video is edited, the corresponding purpose can be quickly realized by means of the action recognition method when wonderful moments such as shooting and shooting need to be found out from the sports video.
In the motion recognition method in the prior art, a time interval with potential motion is selected from a video, then, features of the time interval are extracted and classified, and the motion type is recognized according to a classification result. However, the time intervals extracted by the method are independent, the correlation among the actions is not considered, only a single action can be recognized for each time interval, and a plurality of actions which occur simultaneously cannot be recognized.
In sports videos, multiple actions often occur simultaneously, such as dribbling-defense, dribbling-passing, shooting-capping, shooting-robbing board, etc. in videos of basketball games. Therefore, the motion recognition method in the prior art can be not applied to sports videos.
Disclosure of Invention
The embodiment of the invention provides a method and a device for identifying actions in a video, electronic equipment and a storage medium, which are used for solving the defect that a plurality of actions which occur simultaneously cannot be identified by an action identification method in the prior art.
An embodiment of a first aspect of the present invention provides a method for identifying an action in a video, including:
determining a first sequence of video frames to be identified;
inputting the first video frame sequence to be recognized into a motion recognition model to obtain a motion recognition result output by the motion recognition model; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence; wherein the content of the first and second substances,
the action recognition model is obtained by training based on a sample video frame sequence and a sample video frame sequence label; the sample video frame sequence label comprises category information of one or more actions corresponding to each time in the sample video frame sequence;
the motion recognition model is used for recognizing motion types in the video based on a first motion characteristic and a time domain space domain characteristic which are obtained by a first video frame sequence to be recognized; wherein the first motion characteristics are capable of characterizing a temporal distribution of the respective category motions in the first sequence of video frames and an association between the respective category motions.
In the above technical solution, the motion recognition model includes a first feature extraction layer, a second feature extraction layer, a feature fusion layer, and a motion classification layer; wherein the content of the first and second substances,
the first feature extraction layer is used for extracting a first action feature from the first video frame sequence;
the second feature extraction layer is used for extracting time domain and space domain features from the first video frame sequence;
the characteristic fusion layer is used for fusing the first action characteristic with the time domain and space domain characteristics to obtain a fused characteristic;
and the action classification layer is used for generating an action recognition result of the first video frame sequence according to the fused features.
In the above technical solution, the motion recognition model includes a first feature extraction layer, a second feature extraction layer, a feature fusion layer, and a motion classification layer;
correspondingly, the step of inputting the first video frame sequence to be recognized into the action recognition model to obtain an action recognition result output by the action recognition model specifically includes:
inputting a first video frame sequence to be identified into a first feature extraction layer of a motion identification model to obtain a first motion feature of the first video frame sequence to be identified;
inputting a first video frame sequence to be identified into a second feature extraction layer of the action identification model to obtain time domain and space domain features of the first video frame sequence to be identified;
inputting the first action characteristic and the time domain and space domain characteristic of the first video frame sequence to be identified into a characteristic fusion layer of an action identification model to obtain a fused characteristic of the first video frame sequence to be identified;
and inputting the fused features of the first video frame sequence to be recognized into the action classification layer of the action recognition model to obtain an action recognition result of the first video frame sequence.
In the above technical solution, the first feature extraction layer includes: an intra-frame feature extraction layer and an action feature extraction layer;
correspondingly, the inputting the first video frame sequence to be recognized into the first feature extraction layer of the motion recognition model to obtain the first motion feature of the first video frame sequence to be recognized specifically includes:
inputting a first video frame sequence to be identified into an intra-frame feature extraction layer of a first feature extraction layer to obtain intra-frame features of all video frames in the first video frame sequence to be identified;
and inputting the intraframe characteristics of each video frame in the first video frame sequence to be identified into the action characteristic extraction layer of the first characteristic extraction layer to obtain the first action characteristics of each category of actions contained in the first video frame sequence to be identified.
In the above technical solution, the action feature extraction layer includes a time structure information extraction layer and a weighting layer;
correspondingly, the step of inputting the intra-frame features of each video frame in the first video frame sequence to be recognized into the action feature extraction layer of the first feature extraction layer to obtain the first action features of each category of actions included in the first video frame sequence to be recognized specifically includes:
inputting the intraframe characteristics of each video frame in the first video frame sequence to be identified into a time structure information extraction layer, and acquiring the second action characteristics of each category of actions contained in the first video frame sequence; wherein the second motion characteristics are capable of characterizing a temporal distribution of respective category motions in the first sequence of video frames;
and inputting the second action characteristics into a weighting layer, wherein the weighting layer gives weighting parameters for describing the relevance among all the classes of actions to the second action characteristics to generate the first action characteristics.
In the above technical solution, the inputting the first video frame sequence to be recognized into the second feature extraction layer of the motion recognition model to obtain the time domain and space domain features of the first video frame sequence to be recognized includes:
and sequentially selecting one video frame from the first video frame sequence to be identified, and generating related time domain and space domain characteristics for the selected video frame according to the second characteristic extraction layer of the action identification model to obtain the time domain and space domain characteristics of the first video frame sequence.
In the above technical solution, the generating, according to the second feature extraction layer of the motion recognition model, a relevant time-domain spatial-domain feature for the selected video frame includes:
selecting a plurality of video frames adjacent to each other in front and back from a first video frame sequence to be identified by taking the selected video frame as a center to form a video frame group;
and inputting the video frame group into a second feature extraction layer to generate time domain and space domain features related to the selected video frame.
The embodiment of the second aspect of the present invention provides an apparatus for identifying actions in a video, including:
a determining module for determining a first sequence of video frames to be identified;
the action recognition module is used for inputting the first video frame sequence to be recognized into an action recognition model to obtain an action recognition result output by the action recognition model; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence; wherein the content of the first and second substances,
the action recognition model is obtained by training based on a sample video frame sequence and a sample video frame sequence label; the sample video frame sequence label comprises category information of one or more actions corresponding to each time in the sample video frame sequence;
the motion recognition model is used for recognizing motion types in the video based on a first motion characteristic and a time domain space domain characteristic which are obtained by a first video frame sequence to be recognized; wherein the first motion characteristics are capable of characterizing a temporal distribution of the respective category motions in the first sequence of video frames and an association between the respective category motions.
In a third embodiment of the present invention, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the method for identifying an action in a video according to the first embodiment of the present invention.
A fourth aspect of the present invention provides a non-transitory computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the method for identifying actions in a video according to the first aspect of the present invention.
According to the method, the device, the electronic equipment and the storage medium for recognizing the action in the video, provided by the embodiment of the invention, the video frame sequence to be recognized is input into the action recognition model which is trained in advance, and the action recognition model outputs one or more types of actions corresponding to each moment in the video frame sequence to be recognized, so that the recognition of the actions of multiple types which occur at the same time is realized, and the loss of information is effectively avoided.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a flowchart of a method for recognizing an action in a video according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating operation of a motion recognition model according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for recognizing actions in a video according to another embodiment of the present invention;
fig. 4 is a schematic diagram of an apparatus for recognizing actions in a video according to an embodiment of the present invention;
fig. 5 illustrates a physical structure diagram of an electronic device.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In some videos such as sports videos, a plurality of actions occur at the same time, and the action recognition method in the prior art can only recognize one of the plurality of actions occurring at the same time, which causes a loss of information.
In view of the above, the embodiment of the present invention provides a method for identifying an action in a video. Fig. 1 is a flowchart of a method for identifying an action in a video according to an embodiment of the present invention, as shown in fig. 1, the method includes:
In an embodiment of the invention, the first sequence of video frames to be identified may be extracted from a video of a sports game, such as a video of a basketball game, a soccer game or a volleyball game. In other embodiments of the invention, the sequence of video frames to be identified may also be derived from other types of video, such as movies, television shows, etc.
The extraction of the first sequence of video frames to be identified from the video may be performed by methods known in the art, such as ffmpeg or opencv.
102, inputting a first video frame sequence to be recognized into a motion recognition model to obtain a motion recognition result output by the motion recognition model; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence.
The motion recognition model is obtained by training based on a sample video frame sequence and a sample video frame sequence label; the sample video frame sequence label includes information of one or more action categories corresponding to respective times in the sample video frame sequence.
The motion recognition model is used for recognizing motion types in the video based on a first motion characteristic and a time-domain spatial-temporal (spatio-temporal) characteristic obtained by a first video frame sequence to be recognized; wherein the first motion characteristics are capable of characterizing a temporal distribution of the respective category motions in the first sequence of video frames and an association between the respective category motions.
Time-space-domain (spatial-Temporal) features are well known to those skilled in the art, and are described in detail in, for example, the article entitled "Learning spatial-Temporal reconstruction with Pseudo-3D spatial Networks," which is published by "ZHAOFAN Qiu y, Ting Yao z, and Tao Mei," et al. Related articles can be found in the following websites: https:// arxiv. org/abs/1711.10305.
The method for recognizing the action in the video provided by the embodiment of the invention realizes the recognition of the actions of various types which occur at the same time and effectively avoids the loss of information by inputting the video frame sequence to be recognized into the pre-trained action recognition model and outputting the action of one or more types corresponding to each moment in the video frame sequence to be recognized by the action recognition model.
Based on any one of the above embodiments, in an embodiment of the present invention, the motion recognition model includes: the device comprises a first feature extraction layer, a second feature extraction layer, a feature fusion layer and an action classification layer. Correspondingly, fig. 2 is a flow chart of the operation of the motion recognition model provided in the embodiment of the present invention, and as shown in fig. 2, the inputting the first video frame sequence to be recognized into the motion recognition model to obtain the motion recognition result output by the motion recognition model specifically includes:
Step 204, inputting the fused features of the first video frame sequence to be recognized into a motion classification layer of a motion recognition model to obtain a motion recognition result of the first video frame sequence; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence.
According to the method for recognizing the action in the video, provided by the embodiment of the invention, the sequence of the video frames to be recognized is input into the action recognition model which is trained in advance, and the operations such as feature extraction, feature fusion, action classification and the like are respectively realized by the first feature extraction layer, the second feature extraction layer, the feature fusion layer and the action classification layer in the action recognition model, so that one or more types of actions corresponding to all times in the sequence of the video frames to be recognized are obtained, the recognition of the actions of multiple types occurring at the same time is realized, and the loss of information is effectively avoided.
Based on any one of the above embodiments, in an embodiment of the present invention, the first feature extraction layer further includes: an intra feature extraction layer and an action feature extraction layer. Correspondingly, the step 201 further includes:
step 2011, the first video frame sequence to be identified is input into the intra-frame feature extraction layer of the first feature extraction layer, so as to obtain the intra-frame features of each video frame in the first video frame sequence to be identified.
Step 2012, the intra-frame features of each video frame in the first video frame sequence to be identified are input into the motion feature extraction layer of the first feature extraction layer, so as to obtain the first motion features of each category of motion included in the first video frame sequence to be identified.
Specifically, in the embodiment of the present invention, the intra-frame feature extraction layer referred to in step 2011 is a ResNet34 network, and in other embodiments of the present invention, the intra-frame feature extraction layer may also be another type of image convolution neural network.
Table 1 describes a network structure of the ResNet34 network employed in the embodiment of the present invention.
TABLE 1
The ResNet34 network uses the output of the average pooling layer unwrapped as an intra feature for video frames using vtIf the length of a video frame sequence is T, the dimension of the intra-frame feature of the video frame sequence output by the ResNet34 network is T × D after the video frame sequence is input into the ResNet34 network.
In a preferred implementation, the sequence of video frames to be recognized is prior to the input intra feature extraction layer, and the step of resizing the video frame images is further included.
For example, the picture size is directly changed to 256px × 240px by direct change and center cropping, and then center cropping is performed to 224px × 224 px.
In the embodiment of the present invention, the action feature extraction layer referred to in step 2012 includes a temporal structure information extraction layer and a weighting layer. Correspondingly, the step 2012 further includes:
step 2012-1, inputting the intra-frame features of each video frame in the first video frame sequence to be identified into the time structure information extraction layer, and obtaining the second motion features of each category of motion included in the first video frame sequence.
In the embodiment of the present invention, the time structure information extraction layer is implemented by using a cauchy distribution filter, and in other embodiments of the present invention, the time structure information extraction layer may also use other types of filters, such as a gaussian distribution filter.
The Cauchy distribution filter comprises N Cauchy distributions, and the expression of one Cauchy distribution is as follows:
wherein n represents the number of Cauchy distribution, 0<N is less than or equal to N, the size of N can be set according to actual conditions, T represents time T ∈ {1, 2nRepresents a normalized coefficient;
represents the center point of the nth Cauchy distribution;
representing the width of the nth cauchy distribution the cauchy distribution output has a dimension T × N.
The center point and the width of each Cauchy distribution in the Cauchy distribution filter are determined in the training stage, so that the intra-frame features of the video frame can be directly processed by the Cauchy distribution filter in the step.
Obtaining the related expression of the second action characteristic by utilizing the Cauchy distribution filter for the intraframe characteristic of each video frame in the first video frame sequence to be identified, wherein the related expression is as follows:
wherein C ∈ C, C is the number of categories of actions, and taking basketball action as an example, the categories of the basketball action specifically include: dribbling, defense, passing, dunking, shooting, capping, and cricket, so C ═ 7.
Sc[n]The characteristic obtained by the action of the class c under the action of the nth Cauchy distribution is reflected and is denoted as a second action characteristic in the embodiment of the present invention, and the second action characteristic can represent the time distribution of each category action in the first video frame sequence.
Step 2012-2, inputting the second motion characteristics into a weighting layer, and obtaining first motion characteristics capable of characterizing the time distribution of the motion of each category in the first video frame sequence and the relevance between the motion of each category.
The second motion feature mainly reflects the existence of motion at a certain time point in the video frame sequence to be recognized, but cannot reflect the relevance between the motions. For example, dribbling-defense in a video of a basketball game occurs simultaneously, and there is a correlation between the dribbling and defense actions. Therefore, it is also necessary to obtain the first motion characteristics capable of reflecting the degree of association between the different types of motions based on the second motion characteristics.
In the embodiment of the invention, the correlation degree between different types of actions is reflected by soft attention (soft attention).
The formula for soft attention is:
wherein M represents the number of action categories having an association relation with a certain category of actions, M needs to be less than the category number, that is, M<C；m∈M；Wc，mAnd representing a weight parameter, and determining the specific value of the weight parameter in a training stage.
Soft attention A if M is 2c，mSigmoid (W) may also be usedc，m) And 1-sigmoid (W)c，m) Instead, wherein sigmoid function is
According to the second action characteristic and the soft attention, a calculation formula for obtaining the first action characteristic is as follows:
wherein, SAcThe first action characteristic used for representing the action of the type c has the dimension of D × N.
The first action characteristic can reflect the degree of association between a certain class of action and other classes of action compared to the second action characteristic.
The method for identifying the action in the video provided by the embodiment of the invention can obtain the action time distribution information of each category in the video frame sequence to be identified and the association degree information between the action time distribution information and the action of other categories through the analysis of the video frame sequence to be identified, and is favorable for identifying multiple categories of actions occurring at the same time.
Based on any one of the above embodiments, in an embodiment of the present invention, the second feature extraction layer is a C3D convolutional neural network. Correspondingly, the step 202 further includes:
step 2021, selecting a video frame from the first video frame sequence to be identified, taking the video frame as a center, selecting a group of video frames from the first video frame sequence to be identified, inputting the group of video frames into a C3D convolutional neural network, and obtaining a time domain and space domain feature related to the selected video frame.
Step 2022, executing the operation of step 2021 on each video frame in the first video frame sequence to be identified, to obtain the time domain and space domain features of the first video frame sequence to be identified.
Specifically, in a specific embodiment, 16 pictures (i.e., L ═ 16) of the first 7 frames and the last 8 frames of the selected video frame can be used as a video frame group (if there are insufficient video frames before or after the selected video frame, the video frame group is filled up by filling up 0), and the video frame group is input into the C3D convolutional neural network. Table 2 describes the network structure of the C3D convolutional neural network employed in the embodiment of the present invention.
TABLE 2
The output of the fully-connected layer 2 in the C3D convolutional neural network was used as a time-domain-space-domain feature, v'tIf the length of a video frame sequence is T, the dimension of the time-domain and space-domain features of the video frame sequence output by the C3D convolutional neural network is T × D' after the video frame sequence is input into the C3D convolutional neural network.
As a preferred implementation, the video frame images to be identified need to be resized before being input to the C3D network. For example, the video frame image size is changed to 128px × 171px, and then center-clipped to 112px × 112px size.
In other embodiments of the present invention, the second feature extraction layer may also be other types of 3D convolutional neural networks, such as I3D convolutional neural networks.
The method for identifying the action in the video provided by the embodiment of the invention can obtain the time domain and space domain characteristics of the video frame sequence to be identified by analyzing the video frame sequence to be identified, and is beneficial to identifying various types of actions occurring at the same time.
In any of the above embodiments, in an embodiment of the present invention, the feature fusion layer includes a first mapping transformation layer, a second mapping transformation layer, and a fusion layer. Correspondingly, the step 203 further comprises:
step 2031, inputting the first motion feature in the first video frame sequence to be identified into the first mapping transformation layer in the feature fusion layer to obtain the first motion feature after mapping transformation.
Step 2032, inputting the time domain and space domain features in the first video frame sequence to be identified into the second mapping transformation layer in the feature fusion layer to obtain the time domain and space domain features after mapping transformation.
Step 2033, inputting the mapped first motion feature and the mapped time domain and space domain feature into a fusion layer of the feature fusion layer to obtain a fused feature of the first video frame sequence to be identified.
In the embodiment of the present invention, the first mapping conversion layer implements mapping conversion of the first motion characteristic by using the parameter W.
In the previous embodiment of the present invention, SA was employedcTo represent the first action characteristic of the class c action, with dimension D × N, dimension of parameter H and SAcAre consistent in dimension. When the mapping transformation of the first action characteristic is realized through the parameter H, the mapping transformation can be realized by adopting a matrix dot multiplication and then summing, and the correlation formula is as follows:
∑SAc*H (5)
wherein, the value of the parameter H is determined in the model training stage.
The dimension of the mapped first motion feature is 1 × 1.
In the embodiment of the invention, the second mapping conversion layer is realized by adopting a 3D convolutional layer, the input dimension of the 3D convolutional layer is D ', the output dimension is 1, the step size is 1 × 1 × 1, and the time-domain and space-domain characteristics (namely conv3D (v't) ) is T × 1.
In the embodiment of the invention, the fusion layer realizes the fusion of the characteristics by directly adding the mapping-transformed first action characteristics and the mapping-transformed time-domain and space-domain characteristics. The corresponding formula is as follows:
Vc＝(∑SAc*H)+conv3D(v’t) (6)
wherein conv3D represents a 3D convolution network, the adding part adopts a broadcasting mechanism, and the dimension of the fused feature is T multiplied by 1.
From the fused feature VcCan know that Vc,tRepresents the fused features corresponding to the action of the c-th category at time t.
The method for identifying the action in the video, provided by the embodiment of the invention, is beneficial to realizing the identification of multiple types of actions occurring at the same time by fusing the first action characteristic and the time domain and space domain characteristic of the video frame sequence to be identified.
Based on any one of the above embodiments, in the embodiment of the present invention, the action classification layer is implemented by using a sigmoid function. Specifically, the step 204 further includes:
2041, inputting the fused features of the first video frame sequence into a sigmoid function to obtain a mapping value.
For example, the fused feature V corresponding to the action of the c-th category at time tc,tInputting a sigmoid function to obtain a mapping value sigmoid (V) of the type c action at time tc,t) And the value range of the mapping value is between 0 and 1.
2042, comparing the mapping value with a preset threshold value, and obtaining a motion recognition result of the first video frame sequence according to the comparison result; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence.
For example, if the predetermined threshold value is 0.5, at time t, the map value of the action of the 1 st category (dribbling) is 0.6, the map value of the action of the 2 nd category (defensive) is 0.9, the map value of the 3 rd category (passing) is 0.2, the map value of the 4 th category (shooting) is 0.15, the map value of the 5 th category (shooting) is 0.01, the map value of the 6 th category (capping) is 0.32, and the map value of the 7 th category (basketball) is 0.36. By comparing the mapping values corresponding to different categories of actions with the threshold value, it can be known that there are two categories of actions, namely dribbling and defending, in the first video frame sequence at the time t.
In the embodiment of the invention, the action in the basketball game is taken as an example, and the identification method of the action in the video is exemplified. However, it should be understood by those skilled in the art that the method for identifying the motion in the video provided by the embodiment of the present invention is not limited to the motion in the basketball game, and further, is not limited to the motion of the human body. The method for recognizing the motion in the video provided by the embodiment of the invention can be used for recognizing the motion in the video as long as the motion in different types occurs at the same time in the video.
The method for identifying the action in the video, provided by the embodiment of the invention, realizes the identification of multiple types of actions occurring at the same time by classifying the fused features of the sequence of the video frames to be identified.
Based on any of the above embodiments, fig. 3 is a flowchart of a method for recognizing actions in a video according to another embodiment of the present invention, and as shown in fig. 3, the method for recognizing actions in a video according to another embodiment of the present invention includes:
In embodiments of the present invention, a sequence of sample video frames may be extracted from a sample video.
The sample video may be a sports-like video. Such as a basketball game video, a football game video, a volleyball game video, etc. The sample video may be a full sports game video or a partial video of a sports game. The sample video should be of a certain size, e.g., the sum of the sample video durations is greater than 200 hours.
The sample video frame sequence can be extracted from the sample video by the methods known in the art, such as ffmpeg or opencv.
And 302, labeling the sample video frame sequence to obtain a sample video frame sequence label.
In the embodiment of the present invention, labeling the sample video frame sequence requires labeling the time interval of the occurrence of the motion and the corresponding motion category. For example, the action categories of basketball actions include: dribbling, defense, passing, dunking, shooting, capping, cricket, etc.
From the time intervals of the occurrence of the actions in the sample video frame sequence and the corresponding action categories, category information of one or more actions corresponding to each time in the sample video frame sequence can be obtained.
For example, by labeling the sample video frame sequence, it can be known that the action occurring at the 5 th second in the sample video frame sequence appears in the shooting and capping categories of actions.
Specifically, in the embodiment of the present invention, the forward propagation process for training the motion recognition model includes:
when the intra-frame extraction layer is trained, the ResNet34 network is trained by using the sample video frame sequence as input data for training, and using the category information of one or more actions corresponding to each time in the sample video frame sequence as label data for training. Initial parameters of the ResNet34 network may employ ImageNet pre-trained parameters.
As a preferred implementation, the video frame images need to be resized before the sample video frame sequence is input to the ResNet34 network.
When training the motion feature extraction layer, taking a cauchy distribution filter as an example, the filter includes N cauchy distributions, and the training process includes:
wherein, the formula (7) is a Cauchy distribution center point iterative formula, xnFor the Cauchy distribution center point obtained in the previous iteration step,
n ∈ {1, 2.., N } is the result obtained after updating.
Equation (8) is an iterative equation of the Cauchy distribution width, γnFor the cauchy distribution width obtained in the previous iteration,
n ∈ {1, 2.., N } is the result obtained after updating.
During the training process, the center point and the width of the Cauchy distribution need to be continuously learned so as to better represent the action. At the beginning of training xnThe initialization values of (a) are: normal distribution random sampling with mean of 0 and labeling difference of 0.5, gammanThe initialization values of (a) are: and the normal distribution with the average value of 0 and the labeled difference of 0.0001 is randomly sampled.
Equation (9) is an expression of the cauchy distribution.
In addition, weights in the feature extraction layerWeight parameter Wc，mAdjustments are also required during training.
When the second feature extraction layer is trained, the C3D network is trained by using the sample video frame sequence as input data for training, and by using time interval information of occurrence of an action and category information of the action in the sample video frame sequence as label data for training. Initial parameters for the C3D network may employ pre-trained kinetics400 parameters.
As a preferred implementation, the video frame images need to be resized before the sample video frame sequence is input to the C3D network. For example, the video frame image size is changed to 128px × 171px, and then center-clipped to 112px × 112px size.
When training the first mapping transformation layer, the value of the parameter H needs to be adjusted during the training process.
In training the second mapping transform layer, the parameters in the 3D convolutional layer need to be adjusted during the training process.
And finally, performing binary classification loss function calculation and summation on the C-type actions in the forward propagation process to obtain final loss, wherein the loss function is as follows:
wherein ZcThe true label for each type of action is a vector of T × 1, the element Z of whichc，t∈ {0,1} indicates whether there is a type c action in the sequence of video frames at time t, 0 indicates none, and 1 indicates presence.
And after the loss is obtained, adjusting parameters in the action recognition model through a back propagation algorithm to reduce the value of the loss function until the value of the loss function accords with an expected value.
The method for recognizing the actions in the video, provided by the embodiment of the invention, trains the action recognition model through the sample video frame sequence and the sample video frame sequence label, and outputs one or more types of actions corresponding to all moments in the video frame sequence to be recognized by the action recognition model, so that the actions of multiple types occurring at the same time are recognized, and the information loss is effectively avoided.
Based on any of the above embodiments, fig. 4 is a schematic diagram of an apparatus for recognizing an action in a video according to an embodiment of the present invention, and as shown in fig. 4, the apparatus for recognizing an action in a video according to an embodiment of the present invention includes:
a determining module 401, configured to determine a sequence of video frames to be identified;
a motion recognition module 402, configured to input the first video frame sequence to be recognized into a motion recognition model, so as to obtain a motion recognition result output by the motion recognition model; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence; wherein the content of the first and second substances,
the action recognition model is obtained by training based on a sample video frame sequence and a sample video frame sequence label; the sample video frame sequence label comprises category information of one or more actions corresponding to each time in the sample video frame sequence;
the motion recognition model is used for recognizing motion in the video based on a first motion characteristic and a time domain and space domain characteristic which are obtained by a first video frame sequence to be recognized; wherein the first motion characteristics are capable of characterizing a temporal distribution of the respective category motions in the first sequence of video frames and an association between the respective category motions.
The device for recognizing the action in the video provided by the embodiment of the invention inputs the video frame sequence to be recognized into the pre-trained action recognition model, and the action recognition model outputs one or more types of actions corresponding to each moment in the video frame sequence to be recognized, so that the recognition of the actions of multiple types occurring at the same time is realized, and the loss of information is effectively avoided.
Fig. 5 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 5: a processor (processor)510, a communication Interface (Communications Interface)520, a memory (memory)530 and a communication bus 540, wherein the processor 510, the communication Interface 520 and the memory 530 communicate with each other via the communication bus 540. Processor 510 may call logic instructions in memory 530 to perform the following method: determining a first sequence of video frames to be identified; inputting the first video frame sequence to be recognized into a motion recognition model to obtain a motion recognition result output by the motion recognition model; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence; the motion recognition model is obtained by training based on a sample video frame sequence and a sample video frame sequence label; the sample video frame sequence label comprises category information of one or more actions corresponding to each time in the sample video frame sequence; the motion recognition model is used for recognizing motion types in the video based on a first motion characteristic and a time domain space domain characteristic which are obtained by a first video frame sequence to be recognized; wherein the first motion characteristics are capable of characterizing a temporal distribution of the respective category motions in the first sequence of video frames and an association between the respective category motions.
It should be noted that, when being implemented specifically, the electronic device in this embodiment may be a server, a PC, or other devices, as long as the structure includes the processor 510, the communication interface 520, the memory 530, and the communication bus 540 shown in fig. 5, where the processor 510, the communication interface 520, and the memory 530 complete mutual communication through the communication bus 540, and the processor 510 may call the logic instructions in the memory 530 to execute the above method. The embodiment does not limit the specific implementation form of the electronic device.
Furthermore, the logic instructions in the memory 530 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Further, embodiments of the present invention disclose a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which when executed by a computer, the computer is capable of performing the methods provided by the above-mentioned method embodiments, for example, comprising: determining a first sequence of video frames to be identified; inputting the first video frame sequence to be recognized into a motion recognition model to obtain a motion recognition result output by the motion recognition model; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence; the motion recognition model is obtained by training based on a sample video frame sequence and a sample video frame sequence label; the sample video frame sequence label comprises category information of one or more actions corresponding to each time in the sample video frame sequence; the motion recognition model is used for recognizing motion in the video based on a first motion characteristic and a time domain and space domain characteristic which are obtained by a first video frame sequence to be recognized; wherein the first motion characteristics are capable of characterizing a temporal distribution of the respective category motions in the first sequence of video frames and an association between the respective category motions.
In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented by a processor to perform the method provided by the foregoing embodiments, for example, including: determining a first sequence of video frames to be identified; inputting the first video frame sequence to be recognized into a motion recognition model to obtain a motion recognition result output by the motion recognition model; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence; the motion recognition model is obtained by training based on a sample video frame sequence and a sample video frame sequence label; the sample video frame sequence label comprises category information of one or more actions corresponding to each time in the sample video frame sequence; the motion recognition model is used for recognizing motion in the video based on a first motion characteristic and a time domain and space domain characteristic which are obtained by a first video frame sequence to be recognized; wherein the first motion characteristics are capable of characterizing a temporal distribution of the respective category motions in the first sequence of video frames and an association between the respective category motions.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for recognizing actions in a video is characterized by comprising the following steps:
determining a first sequence of video frames to be identified;
inputting the first video frame sequence to be recognized into a motion recognition model to obtain a motion recognition result output by the motion recognition model; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence; wherein the content of the first and second substances,
the action recognition model is obtained by training based on a sample video frame sequence and a sample video frame sequence label; the sample video frame sequence label comprises category information of one or more actions corresponding to each time in the sample video frame sequence;
the motion recognition model is used for recognizing motion types in the video based on a first motion characteristic and a time domain space domain characteristic which are obtained by a first video frame sequence to be recognized; wherein the first motion characteristics are capable of characterizing a temporal distribution of the respective category motions in the first sequence of video frames and an association between the respective category motions.
2. The method for recognizing the motion in the video according to claim 1, wherein the motion recognition model comprises a first feature extraction layer, a second feature extraction layer, a feature fusion layer and a motion classification layer; wherein the content of the first and second substances,
the first feature extraction layer is used for extracting a first action feature from the first video frame sequence;
the second feature extraction layer is used for extracting time domain and space domain features from the first video frame sequence;
the characteristic fusion layer is used for fusing the first action characteristic with the time domain and space domain characteristics to obtain a fused characteristic;
and the action classification layer is used for generating an action recognition result of the first video frame sequence according to the fused features.
3. The method for recognizing the motion in the video according to claim 1, wherein the motion recognition model comprises a first feature extraction layer, a second feature extraction layer, a feature fusion layer and a motion classification layer;
correspondingly, the step of inputting the first video frame sequence to be recognized into the action recognition model to obtain an action recognition result output by the action recognition model specifically includes:
inputting a first video frame sequence to be identified into a first feature extraction layer of a motion identification model to obtain a first motion feature of the first video frame sequence to be identified;
inputting a first video frame sequence to be identified into a second feature extraction layer of the action identification model to obtain time domain and space domain features of the first video frame sequence to be identified;
inputting the first action characteristic and the time domain and space domain characteristic of the first video frame sequence to be identified into a characteristic fusion layer of an action identification model to obtain a fused characteristic of the first video frame sequence to be identified;
and inputting the fused features of the first video frame sequence to be recognized into the action classification layer of the action recognition model to obtain an action recognition result of the first video frame sequence.
4. The method according to claim 3, wherein the first feature extraction layer comprises: an intra-frame feature extraction layer and an action feature extraction layer;
correspondingly, the inputting the first video frame sequence to be recognized into the first feature extraction layer of the motion recognition model to obtain the first motion feature of the first video frame sequence to be recognized specifically includes:
inputting a first video frame sequence to be identified into an intra-frame feature extraction layer of a first feature extraction layer to obtain intra-frame features of all video frames in the first video frame sequence to be identified;
and inputting the intraframe characteristics of each video frame in the first video frame sequence to be identified into the action characteristic extraction layer of the first characteristic extraction layer to obtain the first action characteristics of each category of actions contained in the first video frame sequence to be identified.
5. The method according to claim 4, wherein the motion feature extraction layer comprises a temporal structure information extraction layer and a weighting layer;
correspondingly, the step of inputting the intra-frame features of each video frame in the first video frame sequence to be recognized into the action feature extraction layer of the first feature extraction layer to obtain the first action features of each category of actions included in the first video frame sequence to be recognized specifically includes:
inputting the intraframe characteristics of each video frame in the first video frame sequence to be identified into a time structure information extraction layer, and acquiring the second action characteristics of each category action contained in the first video frame sequence; wherein the second motion characteristics are capable of characterizing a temporal distribution of respective category motions in the first sequence of video frames;
and inputting the second action characteristics into a weighting layer, wherein the weighting layer gives weighting parameters for describing the relevance among all the classes of actions to the second action characteristics to generate the first action characteristics.
6. The method according to claim 3, wherein the inputting the first video frame sequence to be recognized into the second feature extraction layer of the motion recognition model to obtain the temporal-spatial features of the first video frame sequence to be recognized comprises:
and sequentially selecting one video frame from the first video frame sequence to be identified, and generating related time domain and space domain characteristics for the selected video frame according to the second characteristic extraction layer of the action identification model to obtain the time domain and space domain characteristics of the first video frame sequence.
7. The method according to claim 6, wherein the generating the relevant temporal-spatial features for the selected video frame according to the second feature extraction layer of the motion recognition model comprises:
selecting a plurality of video frames adjacent to each other in front and back from a first video frame sequence to be identified by taking the selected video frame as a center to form a video frame group;
and inputting the video frame group into a second feature extraction layer to generate time domain and space domain features related to the selected video frame.
8. An apparatus for recognizing a motion in a video, comprising:
a determining module for determining a first sequence of video frames to be identified;
the action recognition module is used for inputting the first video frame sequence to be recognized into an action recognition model to obtain an action recognition result output by the action recognition model; the action recognition result comprises one or more action categories corresponding to each time in the first video frame sequence; wherein the content of the first and second substances,
the action recognition model is obtained by training based on a sample video frame sequence and a sample video frame sequence label; the sample video frame sequence label comprises category information of one or more actions corresponding to each time in the sample video frame sequence;
the motion recognition model is used for recognizing motion types in the video based on a first motion characteristic and a time domain space domain characteristic which are obtained by a first video frame sequence to be recognized; wherein the first motion characteristics are capable of characterizing a temporal distribution of the respective category motions in the first sequence of video frames and an association between the respective category motions.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method for identifying an action in a video according to any one of claims 1 to 7 when executing the program.
10. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for identifying an action in a video according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010301132.3A CN111539289A (en) | 2020-04-16 | 2020-04-16 | Method and device for identifying action in video, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010301132.3A CN111539289A (en) | 2020-04-16 | 2020-04-16 | Method and device for identifying action in video, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111539289A true CN111539289A (en) | 2020-08-14 |
Family
ID=71974958
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010301132.3A Pending CN111539289A (en) | 2020-04-16 | 2020-04-16 | Method and device for identifying action in video, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111539289A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112580523A (en) * | 2020-12-22 | 2021-03-30 | 平安国际智慧城市科技股份有限公司 | Behavior recognition method, behavior recognition device, behavior recognition equipment and storage medium |
| CN112818846A (en) * | 2021-01-29 | 2021-05-18 | 湖南科技学院 | Video frame feature extraction method and device and electronic equipment |
| CN112966551A (en) * | 2021-01-29 | 2021-06-15 | 湖南科技学院 | Method and device for acquiring video frame description information and electronic equipment |
| CN113516030A (en) * | 2021-04-28 | 2021-10-19 | 上海科技大学 | Action sequence verification method and device, storage medium and terminal |
| CN113569805A (en) * | 2021-08-13 | 2021-10-29 | 北京建筑大学 | Action recognition method and device, electronic equipment and storage medium |
| CN114241270A (en) * | 2022-02-25 | 2022-03-25 | 动联(山东)电子科技有限公司 | Intelligent monitoring method, system and device for home care |
| WO2022193865A1 (en) * | 2021-03-16 | 2022-09-22 | Huawei Technologies Co.,Ltd. | Systems, methods and computer media for joint attention video processing |
| CN116824275A (en) * | 2023-08-29 | 2023-09-29 | 青岛美迪康数字工程有限公司 | Method, device and computer equipment for realizing intelligent model optimization |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107609460A (en) * | 2017-05-24 | 2018-01-19 | 南京邮电大学 | A kind of Human bodys' response method for merging space-time dual-network stream and attention mechanism |
| CN110084228A (en) * | 2019-06-25 | 2019-08-02 | 江苏德劭信息科技有限公司 | A kind of hazardous act automatic identifying method based on double-current convolutional neural networks |
| CN110222574A (en) * | 2019-05-07 | 2019-09-10 | 杭州智尚云科信息技术有限公司 | Production operation Activity recognition method, apparatus, equipment, system and storage medium based on structuring double fluid convolutional neural networks |
| US20190354835A1 (en) * | 2018-05-17 | 2019-11-21 | International Business Machines Corporation | Action detection by exploiting motion in receptive fields |
| CN110569814A (en) * | 2019-09-12 | 2019-12-13 | 广州酷狗计算机科技有限公司 | Video category identification method and device, computer equipment and computer storage medium |
-
2020
- 2020-04-16 CN CN202010301132.3A patent/CN111539289A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107609460A (en) * | 2017-05-24 | 2018-01-19 | 南京邮电大学 | A kind of Human bodys' response method for merging space-time dual-network stream and attention mechanism |
| US20190354835A1 (en) * | 2018-05-17 | 2019-11-21 | International Business Machines Corporation | Action detection by exploiting motion in receptive fields |
| CN110222574A (en) * | 2019-05-07 | 2019-09-10 | 杭州智尚云科信息技术有限公司 | Production operation Activity recognition method, apparatus, equipment, system and storage medium based on structuring double fluid convolutional neural networks |
| CN110084228A (en) * | 2019-06-25 | 2019-08-02 | 江苏德劭信息科技有限公司 | A kind of hazardous act automatic identifying method based on double-current convolutional neural networks |
| CN110569814A (en) * | 2019-09-12 | 2019-12-13 | 广州酷狗计算机科技有限公司 | Video category identification method and device, computer equipment and computer storage medium |
Non-Patent Citations (3)
| Title |
|---|
| AJ PIERGIOVANNI 等: "Learning Latent Super-Events to Detect Multiple Activities in Videos" * |
| JIANAN LI 等: "SGM-Net: Skeleton-guided multimodal network for action recognition" * |
| 张聪聪;何宁;: "基于关键帧的双流卷积网络的人体动作识别方法" * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112580523A (en) * | 2020-12-22 | 2021-03-30 | 平安国际智慧城市科技股份有限公司 | Behavior recognition method, behavior recognition device, behavior recognition equipment and storage medium |
| CN112818846A (en) * | 2021-01-29 | 2021-05-18 | 湖南科技学院 | Video frame feature extraction method and device and electronic equipment |
| CN112966551A (en) * | 2021-01-29 | 2021-06-15 | 湖南科技学院 | Method and device for acquiring video frame description information and electronic equipment |
| WO2022193865A1 (en) * | 2021-03-16 | 2022-09-22 | Huawei Technologies Co.,Ltd. | Systems, methods and computer media for joint attention video processing |
| US11902548B2 (en) | 2021-03-16 | 2024-02-13 | Huawei Technologies Co., Ltd. | Systems, methods and computer media for joint attention video processing |
| CN113516030A (en) * | 2021-04-28 | 2021-10-19 | 上海科技大学 | Action sequence verification method and device, storage medium and terminal |
| CN113516030B (en) * | 2021-04-28 | 2024-03-26 | 上海科技大学 | Action sequence verification method and device, storage medium and terminal |
| CN113569805A (en) * | 2021-08-13 | 2021-10-29 | 北京建筑大学 | Action recognition method and device, electronic equipment and storage medium |
| CN114241270A (en) * | 2022-02-25 | 2022-03-25 | 动联(山东)电子科技有限公司 | Intelligent monitoring method, system and device for home care |
| CN116824275A (en) * | 2023-08-29 | 2023-09-29 | 青岛美迪康数字工程有限公司 | Method, device and computer equipment for realizing intelligent model optimization |
| CN116824275B (en) * | 2023-08-29 | 2023-11-17 | 青岛美迪康数字工程有限公司 | Method, device and computer equipment for realizing intelligent model optimization |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111539289A (en) | Method and device for identifying action in video, electronic equipment and storage medium | |
| Zhang et al. | Deep image deblurring: A survey | |
| CN108986050B (en) | Image and video enhancement method based on multi-branch convolutional neural network | |
| CN109145784B (en) | Method and apparatus for processing video | |
| Li et al. | No-reference image quality assessment with deep convolutional neural networks | |
| CN111444878A (en) | Video classification method and device and computer readable storage medium | |
| CN111539290B (en) | Video motion recognition method and device, electronic equipment and storage medium | |
| CN112950581A (en) | Quality evaluation method and device and electronic equipment | |
| KR20180065889A (en) | Method and apparatus for detecting target | |
| CN109472193A (en) | Method for detecting human face and device | |
| CN111784624B (en) | Target detection method, device, equipment and computer readable storage medium | |
| CN110889824A (en) | Sample generation method and device, electronic equipment and computer readable storage medium | |
| CN114463218B (en) | Video deblurring method based on event data driving | |
| CN113065645A (en) | Twin attention network, image processing method and device | |
| CN112699786A (en) | Video behavior identification method and system based on space enhancement module | |
| CN110958469A (en) | Video processing method and device, electronic equipment and storage medium | |
| CN112052808A (en) | Human face living body detection method, device and equipment for refining depth map and storage medium | |
| Yang et al. | Deep feature importance awareness based no-reference image quality prediction | |
| CN111079864A (en) | Short video classification method and system based on optimized video key frame extraction | |
| Choudhary et al. | Image dehazing using deep learning techniques | |
| CN112101091B (en) | Video classification method, electronic device and storage medium | |
| CN110110651B (en) | Method for identifying behaviors in video based on space-time importance and 3D CNN | |
| Huang et al. | Learning channel-wise spatio-temporal representations for video salient object detection | |
| CN112084371B (en) | Movie multi-label classification method and device, electronic equipment and storage medium | |
| Ouyang | Total variation constraint GAN for dynamic scene deblurring |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200814 |
|
| RJ01 | Rejection of invention patent application after publication |