CN110427834A - A kind of Activity recognition system and method based on skeleton data - Google Patents
A kind of Activity recognition system and method based on skeleton data Download PDFInfo
- Publication number
- CN110427834A CN110427834A CN201910620715.XA CN201910620715A CN110427834A CN 110427834 A CN110427834 A CN 110427834A CN 201910620715 A CN201910620715 A CN 201910620715A CN 110427834 A CN110427834 A CN 110427834A
- Authority
- CN
- China
- Prior art keywords
- data
- skeleton
- activity recognition
- real time
- serializing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000000694 effects Effects 0.000 title claims abstract description 55
- 238000000034 method Methods 0.000 title claims abstract description 28
- 230000006399 behavior Effects 0.000 claims abstract description 30
- 238000010586 diagram Methods 0.000 claims abstract description 23
- 230000005540 biological transmission Effects 0.000 claims abstract description 22
- 230000007246 mechanism Effects 0.000 claims abstract description 10
- 238000013527 convolutional neural network Methods 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 10
- 239000000284 extract Substances 0.000 claims description 8
- 230000002123 temporal effect Effects 0.000 claims description 6
- 230000002708 enhancing effect Effects 0.000 claims description 2
- 238000012544 monitoring process Methods 0.000 abstract description 10
- 239000011159 matrix material Substances 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 230000004913 activation Effects 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 5
- 238000011160 research Methods 0.000 description 4
- 210000000988 bone and bone Anatomy 0.000 description 3
- 230000033001 locomotion Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
- G06F18/24133—Distances to prototypes
- G06F18/24137—Distances to cluster centroïds
- G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The present invention relates to a kind of Activity recognition system and method based on skeleton data, wherein system includes data acquisition unit, server, data transmission unit and display prewarning unit, and data acquisition unit is for acquiring real time video data；Server is for handling real time video data, output Activity recognition data and output alarm command；Data transmission unit is used for transmission real time video data, Activity recognition data and alarm command；Display prewarning unit is for showing Activity recognition data, alarm command and being sounded an alarm according to alarm command.Compared with prior art, the present invention uses space-time diagram convolution combination long and short cycle memory network, and introduce attention mechanism, network is set preferably to learn the space-time characteristic of field of skeleton data, and when there is unlawful practice, it is sounded an alarm in time by display prewarning unit, present invention combination monitoring camera is able to achieve the identification to specified region human body behavior, and recognition speed is fast, recognition accuracy is high.
Description
Technical field
The present invention relates to the Activity recognition technical field in computer vision, more particularly, to a kind of based on skeleton data
Activity recognition system and method.
Background technique
Human bodys' response is the research hotspot in computer vision research field, is carried out to the movement posture of human body
Automatic identification will provide completely new interactive mode.In crowded place or need the place of key monitoring security protection,
Real-time pictures shooting is usually carried out by camera, and mainly the behavior in video pictures is identified by artificial mode
Judgement, this Activity recognition method inefficiency, not can guarantee yet Activity recognition judgement accuracy, if by computer into
The automatic Activity recognition of row, is identified in real time with substituting manual type to certain specifies behaviors, can save a large amount of manpowers, thus
Effectively improve recognition efficiency and identification accuracy.
Currently both at home and abroad about the technology of Activity recognition mainly include the following types: the neural network mould based on binary-flow network
Type；Neural network model based on Three dimensional convolution；Convolutional neural networks based on skeleton；Network mould based on Recognition with Recurrent Neural Network
Type；Based on conventional method, manual characteristic matching is carried out.
And the technology of above-mentioned Activity recognition has the disadvantage in that in a particular application
1. being handled based on RGB image, computationally intensive, speed is slow, can not handle non-European structured data, cannot reach
The effect identified in real time；
2. pair illumination variation is sensitive, once occurring blocking or complicated weather, the accuracy rate to Activity recognition will be reduced；
3. the irrelevant information in pair pictured scene is more sensitive, such as the variation of clothing, background.
Summary of the invention
It is an object of the present invention to overcome the above-mentioned drawbacks of the prior art and provide one kind to be based on skeleton data
Activity recognition system and method.
The purpose of the present invention can be achieved through the following technical solutions: a kind of Activity recognition system based on skeleton data
System, including data acquisition unit, server, data transmission unit and display prewarning unit, the data acquisition unit pass through number
It is connect according to transmission unit with server, the server is connect by data transmission unit with display prewarning unit, the data
Acquisition unit is for acquiring real time video data；
The server is for handling real time video data, output Activity recognition data and output alarm command；
The data transmission unit is used for transmission real time video data, Activity recognition data and alarm command；
The display prewarning unit is warned for showing Activity recognition data, alarm command and being issued according to alarm command
Report.
Preferably, the server includes sequentially connected Attitude estimation module, skeleton data processing module, Activity recognition
Module and early warning control module, the Attitude estimation module are used to extract human skeleton data from real time video data；
The skeleton data processing module is used to human skeleton data being processed into the non-European knot that network can be directly read
Structure data；
The Activity recognition module is for identifying behavior category result from non-European structured data；
The early warning control module is used to export corresponding alarm command according to behavior category result.
Preferably, the data acquisition unit is camera, and the data transmission unit is wireless network or network data
Line, the display prewarning unit includes display and alarm.
A kind of Activity recognition method based on skeleton data, comprising the following steps:
S1, acquisition real time video data；
S2, according to Openpose Attitude estimation algorithm, the human skeleton data of abstraction sequence from real time video data；
S3, the non-European structured data that the human skeleton data of serializing are processed into serializing；
S4, it is based on space-time diagram convolutional neural networks and long and short cycle memory network, from the non-European structured data of serializing
In identify corresponding behavior category result.
Preferably, the step S2 specifically includes the following steps:
S21, the video frame that real time video data is processed into serializing；
S22, Attitude estimation is carried out to the image of video frame, extracts the human skeleton data of serializing, wherein human body bone
Rack data includes human skeleton key point information；
S23, by two dimension or three-dimensional data in the form of, the human skeleton data of serializing are saved as into JSON formatted file.
Preferably, the step S4 specifically includes the following steps:
S41, space-time diagram convolutional neural networks are based on, extract skeleton space respectively from the non-European structured data of serializing
Information and skeleton temporal information, the skeleton initial characteristic values serialized；
S42, the skeleton initial characteristic values of serializing are inputted into long and short cycle memory network, obtains the final characteristic value of skeleton；
The final characteristic value of S43, skeleton enters Softmax classifier, exports behavior category result.
Preferably, the convolution kernel of the space-time diagram convolutional neural networks include the first dimension and the second dimension, described first
Dimension is for extracting skeleton spatial information, and second dimension is for extracting skeleton temporal information.
It preferably, include attention mechanism in the long and short cycle memory network, the attention mechanism is for enhancing bone
Frame key point information.
Compared with prior art, the invention has the following advantages that
One, the present invention use Openpose Attitude estimation algorithm, using video data as serialize video frame at
Reason extracts skeleton key point information from each frame, and light can effectively be avoided by carrying out Activity recognition using skeleton key point information
According to the influence of the, factor based on pixel value such as background, weather, background, clothing, the unfavorable shadow of illumination condition difference bring are alleviated
It rings, the post-processing data and raising identification accuracy facilitated.
Two, the present invention is based on space-time diagram convolutional neural networks directly to handle human skeleton data, rather than to original
Beginning video data is handled, and using end-to-end algorithm, greatly reduces calculation amount, accelerates network operation speed and instruction
Practice speed, solves the problems, such as that traditional Activity recognition technical speed based on RGB image is slow.
Three, the present invention is by long and short cycle memory network, the skeleton initial characteristics for exporting space-time diagram convolutional neural networks
Value can further learning time feature, and attention mechanism is added, to enhance skeleton key point information, enables network more preferably
The space-time characteristic of field of ground study movement effectively improves identification behavior to further increase the accuracy of the final characteristic value of skeleton
The accuracy of classification results.
Detailed description of the invention
Fig. 1 is system structure diagram of the invention；
Fig. 2 is method flow schematic diagram of the invention；
Fig. 3 is human skeleton schematic diagram data in embodiment；
Fig. 4 is the schematic diagram of figure convolution operator in figure convolutional network；
Fig. 5 is method flow block diagram of the invention；
Fig. 6 is Activity recognition schematic diagram data in embodiment；
Description of symbols in figure: 1, data acquisition unit, 2, server, 21, Attitude estimation module, 22, skeleton data processing
Module, 23, Activity recognition module, 24, early warning control module, 3, data transmission unit, 4, display prewarning unit, 41, display,
42, alarm.
Specific embodiment
The present invention is described in detail with specific embodiment below in conjunction with the accompanying drawings.
As shown in Figure 1, a kind of Activity recognition system based on skeleton data, including data acquisition unit 1, server 2, number
According to transmission unit 3 and display prewarning unit 4, data acquisition unit 1 is connect by data transmission unit 3 with server 2, server
2 are connect by data transmission unit 3 with display prewarning unit 4, wherein and data acquisition unit 1 is used to acquire real time video data,
Camera specially in monitoring range；
Server 2 is for handling real time video data, output Activity recognition data and output alarm command, including posture
Estimation module 21, skeleton data processing module 22, Activity recognition module 23 and early warning control module 24；
Data transmission unit 3 is used for transmission real time video data, Activity recognition data and alarm command, specially wirelessly
Local area network or network data line；
Display prewarning unit 4 is wrapped for showing Activity recognition data, alarm command and being sounded an alarm according to alarm command
Include display 41 and alarm 42.
After installing whole system, entire Activity recognition process be from input terminal (video that monitor camera takes) to
Output end (monitor video comprising skeleton and the classification of motion) settles at one go, the behavior proposed by the present invention based on skeleton data
Identifying system, whole process be it is end-to-end and visual, in a particular application, data acquisition unit 1 is taken the photograph for will monitor
Video in camera is transmitted to server 2 by data transmission unit 3, and server 2 is used to handle video data, including posture is estimated
Count module 21, skeleton data processing module 22, Activity recognition mould 23 and early warning control module 24:
Attitude estimation mould 21 is for obtaining human skeleton data；
Skeleton data processing module 22 is used to skeleton data being processed into the data format that network can acquire, i.e., non-European
Structured data；
The classification of the current behavior for identification of Activity recognition module 23；
Early warning control module 24 is for judging that whether in violation of rules and regulations behavior classification, it is pre- to display exports alarm command if in violation of rules and regulations
Alert unit 4, triggering alarm 42 issue warning information in time.
As shown in Fig. 2, the Activity recognition method based on above system, comprising the following steps:
S1, acquisition real time video data；
S2, according to Openpose Attitude estimation algorithm, the human skeleton data of abstraction sequence from real time video data:
S21, the video frame that real time video data is processed into serializing；
S22, Attitude estimation is carried out to the image of video frame, extracts the human skeleton data of serializing, wherein human body bone
Rack data includes human skeleton key point information；
S23, by two dimension or three-dimensional data in the form of, the human skeleton data of serializing are saved as into JSON formatted file；
S3, the non-European structured data that the human skeleton data of serializing are processed into serializing；
S4, it is based on space-time diagram convolutional neural networks and long and short cycle memory network, from the non-European structured data of serializing
In identify corresponding behavior category result:
S41, space-time diagram convolutional neural networks are based on, extract skeleton space respectively from the non-European structured data of serializing
Information and skeleton temporal information, the skeleton initial characteristic values serialized；
S42, the skeleton initial characteristic values of serializing are inputted into long and short cycle memory network, obtains the final characteristic value of skeleton；
The final characteristic value of S43, skeleton enters Softmax classifier, exports behavior category result.
Wherein, the advantage of space-time diagram convolutional neural networks is to can handle the data of non-European structure, such as social network
Network, molecular structure, skeleton data etc., these data have the characteristics that similar: can regard the data class by putting and side forms as
In type, such as social networks, artificial point, human relationship is as side；In molecular structure, atom is as point, chemical combination key conduct
Side；In organization of human body, for artis as point, arm, leg, cervical vertebra etc. are used as side, and for the data of this non-European structure, it passes
The convolutional neural networks of system can not be handled as image, and the present invention can be very good to locate using figure convolutional neural networks
Such data are managed, enable these skeleton key points to be easier to be read by network, and classify end to end.
In the present embodiment, human skeleton data as shown in Figure 3, node mark are obtained using Openpose Attitude estimation algorithm
In number, 0 is nose key point, and 1 is neck key point, and 2,3,4,5,6,7 be arm key point, and 15,16,17,18 be face
Portion's key point, 9,10,11,12,13,14 be leg key point, and 19,20,21,22,23,24 be sole key point,
People in the image of each frame as the frame of serializing, is carried out Attitude estimation by video by Openpose, and by every picture
Posture kept records of with JSON file.
Fig. 4 is the schematic diagram of figure convolution operator, the convolution class in the definition procedure of picture scroll product and traditional convolutional neural networks
Seemingly, convolution is substantially a process of weighted sum: the process of the convolution nuclear convolution of a 3*3 can actually regard one as
Characteristic value weighted sum obtains the process of new deeper sub-eigenvalue in the region of a 3*3, for the convolution of point 1 in Fig. 4, if
Distance is taken to carry out convolution for 1 neighborhood, then the point got is the point 2,3,4,5 being connected and point 1 itself, this five point weightings are flat
, next layer of characteristic value is obtained.Its formulation is expressed as follows:
In formula,Indicate feature representation of the node j at l layers；
CijTo there is nine points in normalization factor, such as the field of 3*3, then its value is 9；
NiIt is the neighborhood of node j, including node j itself；
Wj lIndicate weight matrix, the value of this weight matrix is learnt automatically in neural network training process；
σ is activation primitive, and the effect of activation primitive is that linear transformation before is converted to nonlinear transformation；
Characteristic value is obtained after carrying out convolution for each skeleton node, later similar to image convolution, after activation primitive,
Obtain next layer of characteristic value
The present invention of convolution in to(for) skeleton is identical as process shown in Fig. 4, i.e., skeleton is considered as figure, by the current root of convolution
Node is multiplied by corresponding weight matrix with the adjacent node of surrounding, after activation primitive activates, obtains next layer node
Characteristic value.
The flow diagram of Activity recognition method of the present invention as shown in figure 5, input data is the video of monitoring collection, by
After Openpose method estimates human body attitude, the skeleton data of acquisition is processed into spatial skeleton data；
When carrying out space diagram convolution, there are two dimensions for the convolution kernel of space diagram convolution: first dimension of convolution kernel is mentioned
The spatial information of skeleton is taken, first dimension extracts the information in the time series of skeleton, obtains the tool of a serializing later
There is the skeleton of advanced features, has had been provided with the spatial structural form and temporal information of original video；
Feature graphic sequence is sent into LSTM (long and short cycle memory) network learning time feature again, and in LSTM network
Attention mechanism (Attention) is added to enhance the information in important joint and characteristic pattern in inside, and can be further improved this is
The accuracy of identification of system；
Finally obtained characteristic value is sent into Softmax classifier and is classified, final behavior classification knot is obtained
Fruit, wherein behavior is divided into unlawful practice and normal behaviour by Soft classifier.
Wherein, in LSTM network shown in Fig. 5, Att is the attention mechanism being added,Representing matrix
Kronecker product,Representing matrix addition, specific formula for calculation are as follows:
ii=σ (Wxi*gXt+Whi*gHt-1+bi) (2)
ft=σ (Wxf*gXt+Whf*gHt-1+bf) (3)
ot=σ (Wxo*gXt+Who*gHt-1+bo) (4)
ut=tanh (Wxc*gXt+Whc*gHt-1+bc) (5)
Ct=ft⊙Ct-1+ii⊙ui (6)
In formula, itIndicate input gate output as a result, ftIndicate to forget door output as a result, otIndicate out gate output as a result, ut
It indicates adjusted to be originally inputted, wherein it、ft、ot、utIt is to be obtained by picture scroll product, we use Wxi*gXtIndicate WxiAnd Xt
Picture scroll product, other W***gX*With W*** gH indicates the picture scroll product of both front and backs, bi、bf、bo、bcIt is neural network for deviation
Automatically the parameter learnt, equally, Wxi、Wxf、Wxo、Wxc、Whi、Whf、Who、WhcFor the weight matrix that neural network learns automatically, Ct
And Ct-1The intermediate variable after matrix operation is done for t and t-1 moment input gate in LSTM network and forgetting door,Do not add for LSTM
Output when attention mechanism is as a result, HtAnd Ht-1Respectively indicate the shape of the t and t-1 moment hidden layer after attention mechanism is added
State, σ and tanh are activation primitive, fattTo pay attention to force function, ⊙ representing matrix dot product.
Fig. 6 is Activity recognition schematic diagram data in embodiment, is identified as a skateboarding behavior, belongs to normal behaviour,
In practical applications, display is shown comprising a main window triggers the picture that the camera of alarm takes, in picture
Epideictic behaviour identifies data simultaneously, the information such as classification of the skeleton of the people including unlawful practice, unlawful practice or normal behaviour, when
When there is no unlawful practice, can manually select and check any monitoring camera, everyone framework information of interface display and
Action classification, the processing result of final server can show over the display, if behavior is in violation of rules and regulations, alarm can bright warning light simultaneously
And it blows a whistle.
In conclusion the present invention contains complete prewarning unit, visualization interface is write based on QT, C Plus Plus, i.e.,
Early warning is visualized as a result, more practical, can be generalized in more actual scenes, for example applies this system in prison,
When the specified behavior such as fight in prison occurs, alarm is flashed and is blown a whistle, meanwhile, display can jump to generation
Under the video monitoring fought, and the framework information and behavior type of current people is marked out, plays the role of notifying in time,
If daily want to check a certain monitored picture, the monitoring is selected from user interface；
Activity recognition method proposed in the present invention is from monitoring scene using algorithm, the input of network end to end
The video data of middle acquisition, the output of network are the video with human skeleton information and the real-time action of personage, general optimizing
While open network deficiency, reduce calculation amount, accelerate the speed of service and training speed of network, solves current many bases
Directly skeleton data is carried out in the slow-footed problem of Activity recognition method of RGB image, and by figure convolutional neural networks
It manages to original video frame processing, considerably reduces calculation amount, enable the network to real time execution, solve the network operation
Speed is slow, cannot analyze the problem of result in real time.And the case where setting can be judged in time, dramatically reduce artificial labor
Fatigue resistance, and can adapt to the Activity recognition of people in scene under various weather, widen application range.
The emergence of artificial intelligence is the progress of the mankind, even more social development, and intelligent monitor system also increasingly becomes research
Hot spot.Behavior analysis system belongs to the scope of intelligent video monitoring, simply and effectively realizes row using end-to-end algorithm
For identification, it can be quickly obtained human body attitude and behavioural information in scene, provide a new direction to social security.This
System cost is lower, can save a large amount of manpower and time, is applicable not only to identify pedestrian behavior in monitoring area, to calculation
Method, which is slightly improved, can be completed all behaviors of anyone in monitor video, while can calculate the number in scene.The system can
For any places equipped with camera head monitor device such as in finance and Business building, office and transport hub, prisons, have
Practical application value and research significance.
Claims (8)
1. a kind of Activity recognition system based on skeleton data, which is characterized in that including data acquisition unit, server, data
Transmission unit and display prewarning unit, the data acquisition unit are connect by data transmission unit with server, the service
Device is connect by data transmission unit with display prewarning unit, and the data acquisition unit is for acquiring real time video data；
The server is for handling real time video data, output Activity recognition data and output alarm command；
The data transmission unit is used for transmission real time video data, Activity recognition data and alarm command；
The display prewarning unit is for showing Activity recognition data, alarm command and being sounded an alarm according to alarm command.
2. a kind of Activity recognition system based on skeleton data according to claim 1, which is characterized in that the server
Including sequentially connected Attitude estimation module, skeleton data processing module, Activity recognition module and early warning control module, the appearance
State estimation module is used to extract human skeleton data from real time video data；
The skeleton data processing module is used to human skeleton data being processed into the non-European structure number that network can be directly read
According to；
The Activity recognition module is for identifying behavior category result from non-European structured data；
The early warning control module is used to export corresponding alarm command according to behavior category result.
3. a kind of Activity recognition system based on skeleton data according to claim 1, which is characterized in that the data are adopted
Integrate unit as camera, the data transmission unit is wireless network or network data line, and the display prewarning unit includes aobvious
Show device and alarm.
4. a kind of Activity recognition method based on skeleton data, which comprises the following steps:
S1, acquisition real time video data；
S2, according to Openpose Attitude estimation algorithm, the human skeleton data of abstraction sequence from real time video data；
S3, the non-European structured data that the human skeleton data of serializing are processed into serializing；
S4, it is based on space-time diagram convolutional neural networks and long and short cycle memory network, known from the non-European structured data of serializing
It Chu not corresponding behavior category result.
5. a kind of Activity recognition method based on skeleton data according to claim 4, which is characterized in that the step S2
Specifically includes the following steps:
S21, the video frame that real time video data is processed into serializing；
S22, Attitude estimation is carried out to the image of video frame, extracts the human skeleton data of serializing, wherein human skeleton number
According to including human skeleton key point information；
S23, by two dimension or three-dimensional data in the form of, the human skeleton data of serializing are saved as into JSON formatted file.
6. a kind of Activity recognition method based on skeleton data according to claim 5, which is characterized in that the step S4
Specifically includes the following steps:
S41, space-time diagram convolutional neural networks are based on, extract skeleton spatial information respectively from the non-European structured data of serializing
With skeleton temporal information, the skeleton initial characteristic values serialized；
S42, the skeleton initial characteristic values of serializing are inputted into long and short cycle memory network, obtains the final characteristic value of skeleton；
The final characteristic value of S43, skeleton enters Softmax classifier, exports behavior category result.
7. a kind of Activity recognition method based on skeleton data according to claim 6, which is characterized in that the space-time diagram
The convolution kernel of convolutional neural networks includes the first dimension and the second dimension, and first dimension is used to extract skeleton spatial information,
Second dimension is for extracting skeleton temporal information.
8. a kind of Activity recognition method based on skeleton data according to claim 6, which is characterized in that the length week
It include attention mechanism in phase memory network, the attention mechanism is for enhancing skeleton key point information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910620715.XA CN110427834A (en) | 2019-07-10 | 2019-07-10 | A kind of Activity recognition system and method based on skeleton data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910620715.XA CN110427834A (en) | 2019-07-10 | 2019-07-10 | A kind of Activity recognition system and method based on skeleton data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110427834A true CN110427834A (en) | 2019-11-08 |
Family
ID=68409198
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910620715.XA Pending CN110427834A (en) | 2019-07-10 | 2019-07-10 | A kind of Activity recognition system and method based on skeleton data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110427834A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110929637A (en) * | 2019-11-20 | 2020-03-27 | 中国科学院上海微***与信息技术研究所 | Image identification method and device, electronic equipment and storage medium |
| CN111347438A (en) * | 2020-02-24 | 2020-06-30 | 五邑大学 | Learning type robot and learning correction method based on same |
| CN112270289A (en) * | 2020-07-31 | 2021-01-26 | 广西科学院 | Intelligent monitoring method based on graph convolution attention network |
| CN112329689A (en) * | 2020-11-16 | 2021-02-05 | 北京科技大学 | Abnormal driving behavior identification method based on graph convolution neural network under vehicle-mounted environment |
| CN112487926A (en) * | 2020-11-26 | 2021-03-12 | 电子科技大学 | Scenic spot feeding behavior identification method based on space-time diagram convolutional network |
| CN112528811A (en) * | 2020-12-02 | 2021-03-19 | 建信金融科技有限责任公司 | Behavior recognition method and device |
| CN112560582A (en) * | 2020-11-24 | 2021-03-26 | 超越科技股份有限公司 | Real-time abnormal behavior monitoring method based on LSTM |
| CN113361417A (en) * | 2021-06-09 | 2021-09-07 | 陕西理工大学 | Human behavior identification method based on variable time sequence |
| CN113408455A (en) * | 2021-06-29 | 2021-09-17 | 山东大学 | Action identification method, system and storage medium based on multi-stream information enhanced graph convolution network |
| CN113963304A (en) * | 2021-12-20 | 2022-01-21 | 山东建筑大学 | Cross-modal video time sequence action positioning method and system based on time sequence-space diagram |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108229355A (en) * | 2017-12-22 | 2018-06-29 | 北京市商汤科技开发有限公司 | Activity recognition method and apparatus, electronic equipment, computer storage media, program |
| CN108549841A (en) * | 2018-03-21 | 2018-09-18 | 南京邮电大学 | A kind of recognition methods of the Falls Among Old People behavior based on deep learning |
| CN108875708A (en) * | 2018-07-18 | 2018-11-23 | 广东工业大学 | Behavior analysis method, device, equipment, system and storage medium based on video |
| CN109101896A (en) * | 2018-07-19 | 2018-12-28 | 电子科技大学 | A kind of video behavior recognition methods based on temporal-spatial fusion feature and attention mechanism |
| CN109101876A (en) * | 2018-06-28 | 2018-12-28 | 东北电力大学 | Human bodys' response method based on long memory network in short-term |
| CN109448291A (en) * | 2018-11-22 | 2019-03-08 | 广东点讯智能科技有限公司 | A kind of behavior video monitoring system applied to construction site |
| CN109784280A (en) * | 2019-01-18 | 2019-05-21 | 江南大学 | Human bodys' response method based on Bi-LSTM-Attention model |
-
2019
- 2019-07-10 CN CN201910620715.XA patent/CN110427834A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108229355A (en) * | 2017-12-22 | 2018-06-29 | 北京市商汤科技开发有限公司 | Activity recognition method and apparatus, electronic equipment, computer storage media, program |
| CN108549841A (en) * | 2018-03-21 | 2018-09-18 | 南京邮电大学 | A kind of recognition methods of the Falls Among Old People behavior based on deep learning |
| CN109101876A (en) * | 2018-06-28 | 2018-12-28 | 东北电力大学 | Human bodys' response method based on long memory network in short-term |
| CN108875708A (en) * | 2018-07-18 | 2018-11-23 | 广东工业大学 | Behavior analysis method, device, equipment, system and storage medium based on video |
| CN109101896A (en) * | 2018-07-19 | 2018-12-28 | 电子科技大学 | A kind of video behavior recognition methods based on temporal-spatial fusion feature and attention mechanism |
| CN109448291A (en) * | 2018-11-22 | 2019-03-08 | 广东点讯智能科技有限公司 | A kind of behavior video monitoring system applied to construction site |
| CN109784280A (en) * | 2019-01-18 | 2019-05-21 | 江南大学 | Human bodys' response method based on Bi-LSTM-Attention model |
Non-Patent Citations (2)
| Title |
|---|
| CHENYANG SI ET AL.: "《An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition》", 《ARXIV:1902.09130V2 [CS.CV]》 * |
| SIJIE YAN ET AL.: "《Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition》", 《ARXIV:1801.07455V2 [CS.CV]》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110929637A (en) * | 2019-11-20 | 2020-03-27 | 中国科学院上海微***与信息技术研究所 | Image identification method and device, electronic equipment and storage medium |
| CN110929637B (en) * | 2019-11-20 | 2023-05-16 | 中国科学院上海微***与信息技术研究所 | Image recognition method and device, electronic equipment and storage medium |
| CN111347438A (en) * | 2020-02-24 | 2020-06-30 | 五邑大学 | Learning type robot and learning correction method based on same |
| CN112270289A (en) * | 2020-07-31 | 2021-01-26 | 广西科学院 | Intelligent monitoring method based on graph convolution attention network |
| CN112329689A (en) * | 2020-11-16 | 2021-02-05 | 北京科技大学 | Abnormal driving behavior identification method based on graph convolution neural network under vehicle-mounted environment |
| CN112560582A (en) * | 2020-11-24 | 2021-03-26 | 超越科技股份有限公司 | Real-time abnormal behavior monitoring method based on LSTM |
| CN112487926A (en) * | 2020-11-26 | 2021-03-12 | 电子科技大学 | Scenic spot feeding behavior identification method based on space-time diagram convolutional network |
| CN112528811A (en) * | 2020-12-02 | 2021-03-19 | 建信金融科技有限责任公司 | Behavior recognition method and device |
| CN113361417A (en) * | 2021-06-09 | 2021-09-07 | 陕西理工大学 | Human behavior identification method based on variable time sequence |
| CN113361417B (en) * | 2021-06-09 | 2023-10-31 | 陕西理工大学 | Human behavior recognition method based on variable time sequence |
| CN113408455A (en) * | 2021-06-29 | 2021-09-17 | 山东大学 | Action identification method, system and storage medium based on multi-stream information enhanced graph convolution network |
| CN113963304A (en) * | 2021-12-20 | 2022-01-21 | 山东建筑大学 | Cross-modal video time sequence action positioning method and system based on time sequence-space diagram |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110427834A (en) | A kind of Activity recognition system and method based on skeleton data | |
| WO2022022368A1 (en) | Deep-learning-based apparatus and method for monitoring behavioral norms in jail | |
| CN110942009B (en) | Fall detection method and system based on space-time hybrid convolutional network | |
| CN108830145B (en) | People counting method based on deep neural network and storage medium | |
| Zhou et al. | Activity analysis, summarization, and visualization for indoor human activity monitoring | |
| CN110428522A (en) | A kind of intelligent safety and defence system of wisdom new city | |
| CN105027550B (en) | For handling visual information with the system and method for detecting event | |
| CN106845351A (en) | It is a kind of for Activity recognition method of the video based on two-way length mnemon in short-term | |
| CN108875708A (en) | Behavior analysis method, device, equipment, system and storage medium based on video | |
| CN110222611A (en) | Human skeleton Activity recognition method, system, device based on figure convolutional network | |
| CN107832708A (en) | A kind of human motion recognition method and device | |
| CN109492612A (en) | Fall detection method and its falling detection device based on skeleton point | |
| CN104010168B (en) | A kind of non-overlapping visual field multiple-camera monitors network topology adaptation learning method | |
| CN109886153B (en) | Real-time face detection method based on deep convolutional neural network | |
| CN110827505A (en) | Smoke segmentation method based on deep learning | |
| CN111707375B (en) | Electronic class card with intelligent temperature measurement attendance and abnormal behavior detection functions | |
| CN110399908B (en) | Event-based camera classification method and apparatus, storage medium, and electronic apparatus | |
| WO2021073311A1 (en) | Image recognition method and apparatus, computer-readable storage medium and chip | |
| CN111860390A (en) | Elevator waiting number detection and statistics method, device, equipment and medium | |
| CN111832465A (en) | Real-time head classification detection method based on MobileNet V3 | |
| CN113642361A (en) | Method and equipment for detecting falling behavior | |
| CN114463837A (en) | Human behavior recognition method and system based on self-adaptive space-time convolution network | |
| CN115841651A (en) | Constructor intelligent monitoring system based on computer vision and deep learning | |
| CN102867214B (en) | Counting management method for people within area range | |
| CN110348395B (en) | Skeleton behavior identification method based on space-time relationship |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191108 |
|
| RJ01 | Rejection of invention patent application after publication |